Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Successful decommissioning of an MSDP - addendum to TECH150431

Created: 16 Jun 2013 • Updated: 19 Jun 2013 | 8 comments
Language Translations
jandersen's picture
+1 1 Vote
Login to vote

Working as a consultant I have recently come across quite a few cases where my customers have tried to decommission

an MSDP for one reason or another (usually because of serious disk failures).

Sometimes the MSDP disks and services are still available (this is always the easiest case).

 

Other times the MSDP disks are gone or the media server is not working at all.

Most of the steps from this recipe will still work but with some few deviations.

The customers had usually tried to follow TECH150431 (http://www.symantec.com/business/support/index?page=content&id=TECH150431) with various luck. The technote has some obvious errors but is also missing some important information.

I'm here trying to describe the procedure that works for me (in most cases). It is founded on TECH150431, but has five additional steps.

Apologies if other and better technotes already exists. Then please comment!

 

I have mostly used the procedure on NBU 7.5.0.x but it should also work with earlier versions as mentioned in the technote.

However some steps will be a little different in pre-7.5.0.x versions.

 

The procedure should also work for recommissioning of the MSDP server (after it has been decommissioned).

 

Prerequisites: The same as listed in TECH150431

 

1) Ensure a recent and successful hot catalog backup exists before proceeding.

In the next steps we are going to delete SLPs, Storage Units and Image references in the Catalog.

In case something goes wrong a catalog backup can come in handy.

 

2) Make sure no policies point directly or indirectly (through SLP) to the MSDP we are going to decommission.

This is to make sure no new backup images are being written to the MSDP (if it is still possible) while we are removing it in the next steps.

Furthermore it will make it possible to remove the SLPs pointing to the MSDP.

 

3) Make sure that the MSDP doesn't hold any images still under SLP control.

This is an important step missing in the technote.

Use nbstlutil to cancel SLP processing for any images that are being read from or written to the MSDP being decommissioned.

 

4) Carefully expire only the images from the catalog that pertain to the Disk Pool to be removed. (NetBackup administration console -> NetBackup Management -> Catalog, 'Action' dropdown set to 'Verify', change 'Disk type' dropdown to PureDisk, and 'Disk Pool' dropdown to the MSDP Disk Pool name in question, then change the start date to past date/time to accommodate the oldest images, then click 'Search Now'. Select all the images found, right-click and choose 'Expire'.) Afterwards, wait for the NetBackup 'Image Cleanup' jobs to complete.

 

Alternative (or supplemental approach)

Obtain the Disk Media ID and Disk Pool from the command 'nbdevquery -listdv -stype PureDisk -U'

Use bpexdate to expire the images on the disk by using the command 'bpexpdate -stype PureDisk -dp <disk pool> -dv <disk volume> -force_not_complete

Eventually use -notimmediate if you have many of images to be deleted. Refer to http://www.symantec.com/business/support/index?pag...

 

Only then proceed to the next steps.

 

Not that the ordering of next steps is wrong in the technote which suggest to delete storage units before they are deleted from any lifecycle.

But it is not possible to delete a storage unit if it is referenced by an SLP. Instead:

 

5) Delete any SLPs that point to the MSDP.

 

If you instead choose to remove the MSDP from the SLPs (if you plan to later re-add it), then make sure that no SLP versions refer to the MSPP.

This can be verified with the command 'nbstl  -L -all_versions'.

Note that if 'CLEANUP_SESSION_INTERVAL_HOURS' has not been changed in the LIFECYCLE_PARAMETERS config file, then old SLP versions default exists 24 hours after a new SLP version has been created.

 

6) Delete any storage units that refer to the MSDP

 

7) Clean up images using nbdelete and bpimage

 

'nbdelete -allvolumes -force'

'bpimage -cleanup -allclients'

 

8) Set the Disk Pool to a down state using nbdevconfig:

nbdevconfig -changestate -stype PureDisk -dp your_MSDP_disk_pool_name -dv PureDiskVolume -state DOWN

(the technote tells to do this in an earlier state - but then the image cleanup will hang/fail!)

 

9)  Run command nbemmcmd -listhosts on the master server to list all the servers, take note of the NDMP machine type entry for the storage server in question.

 

10) Delete the Disk Pool from the GUI. administration console -> Devices -> Disk Pools

 

11) Delete the Storage Server from the GUI.

 

12) Delete the storage server credentials, storage server and the storage server NDMP machine type from the EMM database (nbdevconfig/tpconfig can be used to remove storage_server and creds:

 

tpconfig -delete -storage_server your_MSDP_storage_server_name -stype PureDisk -sts_user_id root

nbdevconfig -deletests -storage_server your_MSDP_storage_server_name -stype PureDisk

nbemmcmd -deletehost -machinename your_MSDP_storage_server_name -machinetype ndmp

 

The two first commands may fail if the GUI has been able to delete the storage server and credentials.

If the error is that the storage server doesn't exist then it is other words OK (given that you didn't make a typo)

 

If you have solved all the errors so far then proceed to

 

13) Stop NetBackup services on the MSDP server (requirement for next step)

The step is not listed in the technote!

 

14) Execute the PDDE_deleteConfig script to remove the MSDP configurations

Windows: <install_path>\Veritas\pdde\PDDE_deleteConfig.bat

UNIX/Linux: /usr/openv/pdde/pdconfigure/scripts/installers/PDDE_deleteConfig.sh

 

15) Remove any cfg file referring to the MSDP server under /usr/openv/lib/ost-plugins (unix) or \Netbackup\bin\ost-plugins (windows).

This is not listed in the technote  must be done on the MSDP server itself and on all other servers having the server referred to as additional server.

On the MSDP server also remove the cfg files for other MSDP servers. (These will be recreated when you add additional servers to your MSDP if you are going to recreate the MSDP).

 

Failure to do so will potentially lead to problems when trying to recreate the MSDP and when trying to add additional media servers for the storage server and to add the media server to the other storage servers. In other words note down all relations between storage servers and then remove all relations which include the MSDP we are decommissioning. Then recreate the relations if the MSDP is to be added later on.

 

16) Delete the deduplication storage directory (and db path, if it was specified at installation time)

 

17) Start the NetBackup services

 

18) Create New Storage Server and Disk Pool

Admin Console -> Storage Servers -> New Storage Server

Admin Console -> Devices -> Disk Pools -> New Disk Pool

 

Done!

 

--jakob;

 

Comments 8 CommentsJump to latest comment

smurphy's picture

Good info, janderson - thanks.

The great majority of Tech Articles that come out of Support are based on actual field experience in cases we worked with customers.
I can update TECH150431 or we can create a new one just for 7.5 and above.

Question for you:  Besides step 10 (Execute the PDDE_deleteConfig script to remove the MSDP configurations), what else would be different about pre-7.5 in your procedure?

Steve Murphy
NetBackup Technical Support
_________________________________
http://go.symantec.com/nb

0
Login to vote
jandersen's picture

Hi,

I can't really remember and I find nothing in my notes regarding issues on 7.1.x.x which are not adressed in my procedure.

But I crosschecked my notes and realize I haven't tried to decommission a 7.0 MSDP. So i can't really say this is well tested on 7.0 smiley

Today - however - I came across a situation on a 7.5.0.5 where deletion of a diskpool was not possible because it still contained images. The images where not visible in the catalog, but they showed up in a "Images on Disk" GUI report.

I expired the images with bpexpdate -backupid xxxx -copy y -d 0 -nodelete -force

After that the pool could be deleted.

This procedure could maybe added as a tip for step 4. There is without doubt other small catches - like this one - that I have run into and 'just solved' and therefore forgotten to write down.
But maybe you can add some of the common problems you see to the technote.

You can choose to create a special technote for 7.5 and up. That is fine with me.
Are you suggesting this because you think 7.1 is less restrictive regarding the order in which SLP, STU etc are deleted?

Or is there any other reason not to keep it in one technote?

--jakob;

 

 

0
Login to vote
smurphy's picture

No, I was just concerned that the procedure would be different enough in 7.5 to warrant a new article.
I believe I'll just update the current one.

Steve Murphy
NetBackup Technical Support
_________________________________
http://go.symantec.com/nb

0
Login to vote
jandersen's picture

Sounds to me like a fine idea

--jakob;!

0
Login to vote
smurphy's picture

jakob, can you clarify this statement in Step 5, please:

 

 then old SLP versions default exists 24 hours after a new SLP version has been created

 

Thanks.

Steve Murphy
NetBackup Technical Support
_________________________________
http://go.symantec.com/nb

0
Login to vote
jandersen's picture

Arh. I see I made a mistake - it should be 14 days - and not 24 hours!

Take a look at http://www.symantec.com/business/support/index?page=content&id=HOWTO68315

and the parameters VERSION_CLEANUP_DELAY_HOURS and CLEANUP_SESSION_INTERVAL_HOURS

The first is further explaned in http://www.symantec.com/business/support/index?page=content&id=HOWTO68320#v42387360

VERSION_CLEANUP_DELAY_HOURS defaults to 14 days!

Old SLP versions that points to STU with references to the MSDP you are decomissioning need to be deleted before you can proceed.

I actually never understood why the default value is 14 days. What is the point with such a long retention?

Maybe you can explain?

--jakob;

 

 

 
0
Login to vote
smurphy's picture

I do not know why, it does seem like a long time.  Err on the side of caution, perhaps.

At any rate, the article has been updated. I have conferred with Subject Matter Experts here on the content and this is what we came up with:

http://www.symantec.com/docs/TECH150431

Steve Murphy
NetBackup Technical Support
_________________________________
http://go.symantec.com/nb

0
Login to vote
jandersen's picture

Cool,

I will try to strictly follow the updated technote the next time I have to decomission an MSDP.

Hopefully most issues are covered now :-)

--jakob; 

0
Login to vote