Video Screencast Help

Vault Job Gets Hung

Created: 31 Dec 2012 • Updated: 31 Dec 2012 | 11 comments
This issue has been solved. See solution.

Hello All,

Facing an issue with automatic vault job.It runs fine to certain point but at 76% it hungs.I have tried to cancel and refire it but its again stopping at 76%

OS-Solaris 10,NBU-6.5

Dec 31, 2012 5:36:30 PM - vault waiting for session ID lock
Dec 31, 2012 5:37:03 PM - vault session ID lock acquired
Dec 31, 2012 5:37:03 PM - vault session ID lock released
Dec 31, 2012 5:36:03 PM - requesting resource asprd222.aldc.att.com.NBVAULT.MAXJOBS
Dec 31, 2012 5:36:03 PM - requesting resource asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Eject_Production
Dec 31, 2012 5:36:26 PM - granted resource  asprd222.aldc.att.com.NBVAULT.MAXJOBS
Dec 31, 2012 5:36:26 PM - granted resource  asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Eject_Production
Dec 31, 2012 5:36:26 PM - estimated 0 kbytes needed
Dec 31, 2012 5:36:26 PM - begin Parent Job
Dec 31, 2012 5:36:26 PM - begin Vault: Start Notify Script
Dec 31, 2012 5:36:26 PM - started process RUNCMD (pid=22600)
Dec 31, 2012 5:36:26 PM - ended process 0 (pid=22600)
Operation Status: 0
Dec 31, 2012 5:36:26 PM - end Vault: Start Notify Script; elapsed time 0:00:00
Dec 31, 2012 5:36:26 PM - begin Vault: Execute Script
Dec 31, 2012 5:36:27 PM - started process bpbrm (pid=22614)
Dec 31, 2012 5:36:31 PM - requesting resource asprd222.aldc.att.com.VAULT_CREATE_SESSION_ID.LOCK_TLD(0)_Production
Dec 31, 2012 5:37:02 PM - granted resource  asprd222.aldc.att.com.VAULT_CREATE_SESSION_ID.LOCK_TLD(0)_Production
Dec 31, 2012 5:40:58 PM - Duplication skipped
Dec 31, 2012 5:40:59 PM - vault waiting for assign slot lock
Dec 31, 2012 5:40:59 PM - requesting resource asprd222.aldc.att.com.VAULT_ASSIGN_SLOT.LOCK_TLD(0)_Production
Dec 31, 2012 5:41:07 PM - vault assign slot lock acquired
Dec 31, 2012 5:41:05 PM - granted resource  asprd222.aldc.att.com.VAULT_ASSIGN_SLOT.LOCK_TLD(0)_Production
Dec 31, 2012 5:41:42 PM - vault assign slot lock released
Dec 31, 2012 5:41:42 PM - Catalog Backup skipped
Dec 31, 2012 5:41:42 PM - vault waiting for assign slot lock
Dec 31, 2012 5:41:43 PM - requesting resource asprd222.aldc.att.com.VAULT_ASSIGN_SLOT.LOCK_TLD(0)_Production
Dec 31, 2012 5:42:15 PM - vault assign slot lock acquired
Dec 31, 2012 5:42:17 PM - vault assign slot lock released
Dec 31, 2012 5:42:20 PM - before eject, waiting for media to be unmounted; sleeping for 380 seconds
Dec 31, 2012 5:42:14 PM - granted resource  asprd222.aldc.att.com.VAULT_ASSIGN_SLOT.LOCK_TLD(0)_Production
Dec 31, 2012 5:48:40 PM - starting eject operation
Dec 31, 2012 5:48:40 PM - begin Eject and Report
Dec 31, 2012 5:48:40 PM - connecting
Dec 31, 2012 5:48:40 PM - connected; connect time: 0:00:00
Dec 31, 2012 5:48:42 PM - vault waiting for eject lock
Dec 31, 2012 5:48:42 PM - requesting resource asprd222.aldc.att.com.VAULT_EJECT.LOCK_0
Dec 31, 2012 5:49:08 PM - Info nbrb (pid=7943) Limit has been reached for the logical resource asprd222.aldc.att.com.VAULT_EJECT.LOCK_0
 

Tried to delete the lock files but seems nothing changed.

Please suggest.

Comments 11 CommentsJump to latest comment

Nagalla's picture

hi,

it looks like  you have another Vault job running...

For each vault name only one vault profile can be run at a  time. This is by design in Netbackup vault feature and cannot be changed.

if you are not seeing another vault job for the same profile.. 

could you expline from where/how did you try to delte the lock files?

 

Manjunath Rajanna's picture

Hi,

Just want to know are you configured any other vault profile under each vault name. If that is the case follow the below steps.

•Create new vault along with existing one
•Move the other profile to the new vault, or create a new profile under the new vault.

Then try to run concurrently both profiles, will allow the two vault profiles.

If you are not configured another valut profiles...send the nbrb logs will helps to know more on this issue

Shekaib's picture

@ Nagalla

I could see files in misc folder under volmgr.As of now i have these files

lmfcd.lock  robotic_db  vmd.lock

No other job for the same profile is running.

Nagalla's picture

I just want to make sure you are seeing misc folder in Robot control host.. it may be your master server or may not be.. .. but you are seeing in Robot control host.

could you check if you are seeing old jobs stuck in EMM.

nbrbuitl -dump | grep -i Eject

nbrbutil -dump | grep -i vault

and also  provide the output of below command.

vxlogview -p 51216 -o 118 -b "12/31/12 05:00:00 PM" -e "12/31/12 6:32:00 PM" -d p

 

Mark_Solutions's picture

do you have any vlt* processes running?
It may be an orphaned process causing the lock and it meeds clearing down to release it

If nothing else is running maybe shut down netbackup and see if any processes have hung and clear them down, or reboot the server for a complete cleanup and then try again

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Shekaib's picture

@Nagalla

   index=0 (Request provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0  userSequence=0 (CountedResourceRequest resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 max=1)))
         index=0 (Request provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0  userSequence=0 (CountedResourceRequest resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 max=1)))
         index=47 (Allocation: id={8027C028-1DD2-11B2-A310-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Eject_Production masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2425658" named resource allocation)
         index=170 (Allocation: id={CF1DCF76-1DD1-11B2-844F-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=0 userid="jobid=2403050" named resource allocation)

 

index=0 (Request provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0  userSequence=0 (CountedResourceRequest resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 max=1)))
         index=0 (Request provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0  userSequence=0 (CountedResourceRequest resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 max=1)))
         index=40 (Allocation: id={95B04D10-1DD1-11B2-9742-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_DUPLICATION.LOCK_TLD(0)_Vault_orca_pisces_toll masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=0 userid="jobid=2378694" named resource allocation)
         index=46 (Allocation: id={5035E592-1DD2-11B2-AA16-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Vault_LTO3_orca_pisces_toll masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2378694" named resource allocation)
         index=77 (Allocation: id={5035E51A-1DD2-11B2-A3BB-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBVAULT.MAXJOBS masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2378694" named resource allocation)
         index=78 (Allocation: id={76938746-1DD2-11B2-9476-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBVAULT.MAXJOBS masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2425161" named resource allocation)
         index=79 (Allocation: id={76D6E252-1DD2-11B2-B5C2-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBVAULT.MAXJOBS masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2425159" named resource allocation)
         index=81 (Allocation: id={7950D684-1DD2-11B2-87F5-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBVAULT.MAXJOBS masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2403050" named resource allocation)
         index=86 (Allocation: id={7950D710-1DD2-11B2-89C9-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Vault_LTO3_orca_pisces masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2403050" named resource allocation)
         index=125 (Allocation: id={00C7E7B8-1DD2-11B2-9E64-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_DUPLICATION.LOCK_TLD(0)_Production masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=0 userid="jobid=2425159" named resource allocation)
         index=128 (Allocation: id={769387BE-1DD2-11B2-9904-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Vault_LTO3_orca_2_7_10 masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2425161" named resource allocation)
         index=129 (Allocation: id={76D6E2D4-1DD2-11B2-A112-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Vault_Production masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2425159" named resource allocation)
         index=153 (Allocation: id={8027BFB0-1DD2-11B2-8BA9-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBVAULT.MAXJOBS masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2425658" named resource allocation)
         index=166 (Allocation: id={CF1DCF76-1DD1-11B2-844F-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=0 userid="jobid=2403050" named resource allocation)

 

vxlogview -p 51216 -o 118 -b "12/31/12 05:00:00 PM" -e "12/31/12 6:32:00 PM" -d p

V-1-45 No log files found.

@Mark

We have already bounced the services on this server.But issue seems to be there.

 

Nagalla's picture

 

         index=47 (Allocation: id={8027C028-1DD2-11B2-A310-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Eject_Production masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2425658" named resource allocation)
         index=170 (Allocation: id={CF1DCF76-1DD1-11B2-844F-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=0 userid="jobid=2403050" named resource allocation)
 
hi ,
what is the status of the jobs, and which one is the current one?
jobid=2425658
jobid=2403050
 
 
 
 
Shekaib's picture

 

2425658 Vault Active Limit has been reached for requested resource (asprd222.aldc.att.com.VAULT_EJECT.LOCK_0)   Eject_Production Production_Eject asprd222.aldc.att.com Mon Dec 31 17:36:03 IST 2012     8956     1 Eject and Report       76 22617 root   2425658   Mon Dec 31 17:36:26 IST 2012 8933 TLD(0) Production Production_Eject 540   Standard   asprd222.aldc.att.com 0

 

2403050 Vault Active     Vault_LTO3_orca_pisces Vault_LTO2_rsprd319 asprd222.aldc.att.com Tue Dec 25 14:30:00 IST 2012     537318     1 Eject and Report       76 24677 root   2403050   Tue Dec 25 14:30:05 IST 2012 537313 TLD(0) Vault_LTO3_orca_pisces Vault_LTO3_orca_pisces 55 2 Standard   asprd222.aldc.att.com 0

current one which is stuck at 76%

                                    STATE DETAILS                                                POLICY               SCHEDULE

2425658 Vault Active Limit has been reached for requested resource (asprd222.aldc.att.com.VAULT_EJECT.LOCK_0)   Eject_Production Production_Eject asprd222.aldc.att.com Mon Dec 31 17:36:03 IST 2012     8956     1 Eject and Report       76 22617 root   2425658   Mon Dec 31 17:36:26 IST 2012 8933 TLD(0) Production Production_Eject 540   Standard   asprd222.aldc.att.com 0

 

Nagalla's picture

both these jobs are for the same robot?

Shekaib's picture

I think some profile config issue has been encontered..coz both these jobs were for same robot but now only
profile for  Vault_LTO2_rsprod319 exists and i cant find Eject_Production one..:(

Nagalla's picture

so you got  to know what you need to do...

just disable/delete the policy which is triggering this vault job. and also you need to check with your team, about the chages on this vault profiles to make sure everything is fine..

Good luck.. 

SOLUTION