Successful decommissioning of an MSDP - addendum to TECH150431
Working as a consultant I have recently come across quite a few cases where my customers have tried to decommission
an MSDP for one reason or another (usually because of serious disk failures).
Sometimes the MSDP disks and services are still available (this is always the easiest case).
Other times the MSDP disks are gone or the media server is not working at all.
Most of the steps from this recipe will still work but with some few deviations.
The customers had usually tried to follow TECH150431 (http://www.symantec.com/business/support/index?page=content&id=TECH150431) with various luck. The technote has some obvious errors but is also missing some important information.
I'm here trying to describe the procedure that works for me (in most cases). It is founded on TECH150431, but has five additional steps.
Apologies if other and better technotes already exists. Then please comment!
I have mostly used the procedure on NBU 7.5.0.x but it should also work with earlier versions as mentioned in the technote.
However some steps will be a little different in pre-7.5.0.x versions.
The procedure should also work for recommissioning of the MSDP server (after it has been decommissioned).
Prerequisites: The same as listed in TECH150431
1) Ensure a recent and successful hot catalog backup exists before proceeding.
In the next steps we are going to delete SLPs, Storage Units and Image references in the Catalog.
In case something goes wrong a catalog backup can come in handy.
2) Make sure no policies point directly or indirectly (through SLP) to the MSDP we are going to decommission.
This is to make sure no new backup images are being written to the MSDP (if it is still possible) while we are removing it in the next steps.
Furthermore it will make it possible to remove the SLPs pointing to the MSDP.
3) Make sure that the MSDP doesn't hold any images still under SLP control.
This is an important step missing in the technote.
Use nbstlutil to cancel SLP processing for any images that are being read from or written to the MSDP being decommissioned.
4) Carefully expire only the images from the catalog that pertain to the Disk Pool to be removed. (NetBackup administration console -> NetBackup Management -> Catalog, 'Action' dropdown set to 'Verify', change 'Disk type' dropdown to PureDisk, and 'Disk Pool' dropdown to the MSDP Disk Pool name in question, then change the start date to past date/time to accommodate the oldest images, then click 'Search Now'. Select all the images found, right-click and choose 'Expire'.) Afterwards, wait for the NetBackup 'Image Cleanup' jobs to complete.
Alternative (or supplemental approach)
Obtain the Disk Media ID and Disk Pool from the command 'nbdevquery -listdv -stype PureDisk -U'
Use bpexdate to expire the images on the disk by using the command 'bpexpdate -stype PureDisk -dp <disk pool> -dv <disk volume> -force_not_complete
Eventually use -notimmediate if you have many of images to be deleted. Refer to http://www.symantec.com/business/support/index?pag...
Only then proceed to the next steps.
Not that the ordering of next steps is wrong in the technote which suggest to delete storage units before they are deleted from any lifecycle.
But it is not possible to delete a storage unit if it is referenced by an SLP. Instead:
5) Delete any SLPs that point to the MSDP.
If you instead choose to remove the MSDP from the SLPs (if you plan to later re-add it), then make sure that no SLP versions refer to the MSPP.
This can be verified with the command 'nbstl -L -all_versions'.
Note that if 'CLEANUP_SESSION_INTERVAL_HOURS' has not been changed in the LIFECYCLE_PARAMETERS config file, then old SLP versions default exists 24 hours after a new SLP version has been created.
6) Delete any storage units that refer to the MSDP
7) Clean up images using nbdelete and bpimage
'nbdelete -allvolumes -force'
'bpimage -cleanup -allclients'
8) Set the Disk Pool to a down state using nbdevconfig:
nbdevconfig -changestate -stype PureDisk -dp your_MSDP_disk_pool_name -dv PureDiskVolume -state DOWN
(the technote tells to do this in an earlier state - but then the image cleanup will hang/fail!)
9) Run command nbemmcmd -listhosts on the master server to list all the servers, take note of the NDMP machine type entry for the storage server in question.
10) Delete the Disk Pool from the GUI. administration console -> Devices -> Disk Pools
11) Delete the Storage Server from the GUI.
12) Delete the storage server credentials, storage server and the storage server NDMP machine type from the EMM database (nbdevconfig/tpconfig can be used to remove storage_server and creds:
tpconfig -delete -storage_server your_MSDP_storage_server_name -stype PureDisk -sts_user_id root
nbdevconfig -deletests -storage_server your_MSDP_storage_server_name -stype PureDisk
nbemmcmd -deletehost -machinename your_MSDP_storage_server_name -machinetype ndmp
The two first commands may fail if the GUI has been able to delete the storage server and credentials.
If the error is that the storage server doesn't exist then it is other words OK (given that you didn't make a typo)
If you have solved all the errors so far then proceed to
13) Stop NetBackup services on the MSDP server (requirement for next step)
The step is not listed in the technote!
14) Execute the PDDE_deleteConfig script to remove the MSDP configurations
15) Remove any cfg file referring to the MSDP server under /usr/openv/lib/ost-plugins (unix) or \Netbackup\bin\ost-plugins (windows).
This is not listed in the technote must be done on the MSDP server itself and on all other servers having the server referred to as additional server.
On the MSDP server also remove the cfg files for other MSDP servers. (These will be recreated when you add additional servers to your MSDP if you are going to recreate the MSDP).
Failure to do so will potentially lead to problems when trying to recreate the MSDP and when trying to add additional media servers for the storage server and to add the media server to the other storage servers. In other words note down all relations between storage servers and then remove all relations which include the MSDP we are decommissioning. Then recreate the relations if the MSDP is to be added later on.
16) Delete the deduplication storage directory (and db path, if it was specified at installation time)
17) Start the NetBackup services
18) Create New Storage Server and Disk Pool
Admin Console -> Storage Servers -> New Storage Server
Admin Console -> Devices -> Disk Pools -> New Disk Pool