Video Screencast Help

Image Cleanup status 37

Created: 03 Aug 2012 • Updated: 28 Nov 2012 | 17 comments
Marianne's picture
This issue has been solved. See solution.

Master: Linux RHEL 
Media servers: Mix of various OS's
NBU version: 7.5.0.3 on master and all media servers

This is an 'inherited' environment and nobody knows anything about media09_bck.

How media server was removed with valid images in the catalog is a mystery.

Image cleanup fails with following details:

 

08/02/2012 15:03:42 - Info nbdelete (pid=6628) deleting expired images. Media Server: media09_bck Media: /dd/09
08/02/2012 15:03:42 - Error nbdelete (pid=6628) Fragments were not removed. Server media09_bck is invalid (37 )
08/02/2012 15:03:43 - Info nbdelete (pid=6628) deleting expired images. Media Server: media09_bck Media: /dd_co/09_dd
08/02/2012 15:03:43 - Error nbdelete (pid=6628) Fragments were not removed. Server media09_bck is invalid (37 )

 

nbemmcmd -listhosts
NBEMMCMD, Version: 7.5.0.3
The following hosts were found:
server           master
master           master
media            media02
media            media01
media            media04
foreign_media    media09_bck
media            media03

Command completed successfully.

 

 

nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media
NBEMMCMD, Version: 7.5.0.3
The function returned the following failure status:
the media is allocated for use (199)

Command did not complete successfully.

Unable to find evidence of allocated media:

nbrbutil -dump

 

MDS allocations in EMM:

        MdsAllocation: allocationKey=25267 jobType=1 mediaKey=0 mediaId= driveKey=0 driveName= drivePath= stuName=media01_disk_backup2 masterServerName=master mediaServerName=media01 ndmpTapeServerName= diskVolumeKey=60 mountKey=109 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0

        MdsAllocation: allocationKey=25268 jobType=1 mediaKey=0 mediaId= driveKey=0 driveName= drivePath= stuName=media01_disk_backup3 masterServerName=master mediaServerName=media01 ndmpTapeServerName= diskVolumeKey=71 mountKey=120 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0

 

nbdecommission fails with :

unable to obtain machine record for media09_bck, machine type media server

 

Please help!

How do we get rid of this 'ghost' media server?

Comments 17 CommentsJump to latest comment

mph999's picture

Ouch,

"nobody knows anything about media09_bck" - neither dows NBU it would appear ...

Manual SQL script, or NBCC I suspect - log call time.

BUT ..  it seems the media server is gone, which is why it is "foregin"

Images may include the names of media servers that do not exist in the current
domain. These media servers are known as foreign media servers.
5 Chapter
The following describes when a foreign media server may appear in a backup
image:
■ When a backup image is migrated from one domain to another.
■ When a media server is decommissioned.
Starting with NetBackup 7.5, foreign media servers also appear in the output for
the following command:
nbemmcmd -listhosts
The output displays the foreign media server as follows:
foreign_media server
NetBackup 7.5 operational behavior changes
About foreign media servers
36
 
So the question may be more ' how to get rid of the images in the DB .. presuming the images were on a disk STU that is now gone " ?
 

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mph999's picture

Hmm,

Did you try nbdelete -allvolumes -force

M

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
Marianne's picture

Yes, we have tried nbdelete -allvolumes -force.

Not sure if nbdelete shows up as Image Cleanup in Activity Monitor or if it was coincidence that we saw failed Image Cleanup job shortly after running nbdelete.
No idea if PID 6628  (nbdelete (pid=6628) in Activity Monitor) was the same as our nbdelete command or another one called by 'normal' Image Cleanup.
I guess admin log (if enabled) will tell us but I think that will be purely academical and not really help...

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

mph999's picture

OK, can't get to the GUI at the sec, but nbdelete shows in the trylogs, so should be in the gui.

I'm wondering if NBCC can pick this up, it does some disk stuff now, and runs some sql I believe if necessary.  Could be worth trying that.

M

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
Marianne's picture

Thanks Martin

Will suggest that.  This means that we need to log a Support call, right?? wink

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

mph999's picture

Yep, looking that way I think ...  Might want to include a nbdb_unload output, bpimagelist and nbstlutil list + nbsu -c -t ...

M

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
Marianne's picture

Will update here when we have response from Support.

Since nothing is 'broken', customer and colleagues will probably wait till Monday to log a call.

Thanks for all your support, Martin!

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Praveen_Sundriyal's picture

Hi Marianne ,

You can try below steps to fix this issue,

Step 1 : Check what all media still assigned to this host i.e. media09_bck

bpmedialist -l -h media09_bck

Step 2 : Move any tapes assigned to this host to any working media server ,

bpmedia -movedb -m <tape_number> -newserver <working_media_server_name> -oldserver <decommission_media_server_name>

Step 3 : If there are any image referrring to disk volume in bpmedialist output in Step1, run below command against each volume,

nbdelete -purge_deletion_list -media_id  'Disk_Path' -force

Step 4 : Run below command ,

nbdelete -allvolumes -force

Step 5 : Remove problematic media server from EMM by using below command

nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media

Let me know if it fixed this issue .

Regards,
Praveen

Marianne's picture

Thanks for your suggestions Praveen.

Sincere apologies for not updating this post...

There have been other issues at this particular customer that received priority.

We still need to get rid of this 'ghost media server' - hopefully next week some time.

I will discuss suggestions in this thread with the customer and report back.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Radhakrishnan's picture

This happens because the images you have imported EMM has been inherited from this server.

Try # bpimage - newserver <> -oldserver <>

then try to delete the server

nbemmcmd -deletehost -machinename media09_bck -machinetype foreign_media

SOLUTION
Radhakrishnan's picture

I encountered this issue when we did catalgo manipulation for 7.5 and after an hour of research I could resolve it .

Marianne's picture

Thanks for all the ideas and advice.

There are unfortunaly issues with failed backups that is taking up everyone's time and energy (logs uploaded to Symantec support, tests, more logs, etc...)

We will get back to this non-critical issue once the 'dust has settled'....

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Mark_Solutions's picture

Hi Marianne - when you get chance to look at this again and if the above do not help (most should do unless the images for that server were on disk) then run an All Log Entries report - especially either side of midnight when the cleanup runs as it will actually list the image IDs of the problem images

From that, if they are on disk, they can be forcibly deleted against each image id after which you should be able to get rid of the rogue media server

the oldserver newserver usually sorts most things out and then just remove all reference to that server from the installation

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Genericus's picture

Marianne - instead of specifying media use allvolumes.

 

bpmedia -movedb -allvolumes -newserver newservername -oldserver oldservername

 

It also runs much quicker than individual media id...

 

NetBackup 7.5.0.3 on Solaris 10
EMC 4206 VTL
acsls 7.3.1
SL8500 LTO5

Genericus's picture

Once all volumes are off the old server, you should be able to remove or decommission it,

 

NetBackup 7.5.0.3 on Solaris 10
EMC 4206 VTL
acsls 7.3.1
SL8500 LTO5

Gnur's picture

Hi Marianne,

Thanks to Genericus, I was able to remove the "Ghost" media server I had already decomissioned but could not rid it from residing within the EMM DB tablespace. 

$ bpmedia -movedb -allvolumes -newserver <valid server> -oldserver <ghost media server>

$ nbemmcmd -deletehost -machinename <ghost media server> -machinetype <media or foreign_media>

Thanks again to all in this post that fixed my headache...

Gnur

Marianne's picture

I need to get back to the customer about this one... have not had any feedback since August...

Apologies to all for leaving this thread unsolved.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links