Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Netbackup 7.0 Linux - TL4000 only a working drive

Created: 08 Feb 2013 • Updated: 11 Feb 2013 | 20 comments
Npalma's picture
This issue has been solved. See solution.

Netbackup 7.0 Linux - TL4000 only a working drive:

Greetings!, I have a problem, (which appeared returned from my vacation), I have set up a media manager TLD - hcart2 TL4000, the problem is I can not do more than one job simultaneously to this medium, which earlier if possible, I give details:

Conf Media Manager.-

Only 001 mounts tapes.-

[root@backup6m-bkp bin]# /usr/openv/volmgr/bin/vmoprcmd -d

 
                                PENDING REQUESTS
 
                                     <NONE>
 
                                  DRIVE STATUS
 
Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart2   TLD                -                     No       -         0  
  0 hcart2   TLD                -                     No       -         0  
  1 hcart2   TLD               Yes   000011  000011   Yes     Yes        0  
  1 hcart2   TLD               Yes   000011  000011   Yes     Yes        0  
  2 hcart2   TLD                -                     No       -         0  
  2 hcart2   TLD                -                     No       -         0  
  3 hcart2   TLD                -                     No       -         0  
  3 hcart2   TLD                -                     No       -         0  
 
                             ADDITIONAL DRIVE STATUS
 
Drv DriveName            Shared    Assigned        Comment                   
  0 IBM.ULT3580-HH5.000   No       -                                         
  0 IBM.ULT3580-HH5.000   No       -                                         
  1 IBM.ULT3580-HH5.001   No       backup6m-bkp                              
  1 IBM.ULT3580-HH5.001   No       backup6m-bkp                              
  2 IBM.ULT3580-HH5.002   No       -                                         
  2 IBM.ULT3580-HH5.002   No       -                                         
  3 IBM.ULT3580-HH5.003   No       -                                         
  3 IBM.ULT3580-HH5.003   No       - 

SCSI devices.-

[root@backup6m-bkp bin]# lsscsi

[0:0:0:0]    disk    VMware   Virtual disk     1.0   /dev/sda
[1:0:0:0]    disk    EQLOGIC  100E-00          6.0   /dev/sdb
[1:0:1:0]    disk    EQLOGIC  100E-00          6.0   /dev/sdc
[1:0:2:0]    disk    EQLOGIC  100E-00          6.0   /dev/sdd
[1:0:3:0]    disk    EQLOGIC  100E-00          6.0   /dev/sde
[2:0:0:0]    tape    IBM      ULT3580-HH5      C7RD  /dev/st0
[3:0:0:0]    tape    IBM      ULT3580-HH5      C7RD  /dev/st1
[8:0:0:0]    tape    IBM      ULT3580-HH5      C7RD  /dev/st2
[9:0:0:0]    tape    IBM      ULT3580-HH5      C7RD  /dev/st3
[10:0:0:0]   mediumx IBM      3573-TL          B.60  -       
[11:0:0:0]   mediumx IBM      3573-TL          B.60  -       
[12:0:0:0]   tape    IBM      ULT3580-HH5      C7RD  /dev/st6
[13:0:0:0]   tape    IBM      ULT3580-HH5      C7RD  /dev/st7
[16:0:0:0]   tape    IBM      ULT3580-HH5      C7RD  /dev/st4
[17:0:0:0]   tape    IBM      ULT3580-HH5      C7RD  /dev/st5
 
Job Queued.-

Status Library.-

I hope I can help, greetings.

Comments 20 CommentsJump to latest comment

sazz.'s picture

what is policy schedule Media Multiplexing is set ? Is it set right? Make sure this is set correctly

Are they in same pool and retention. Also what about max jobs per client setting in master server properties.

Npalma's picture

Thanks:

Max jobs per client = 4

Media Multiplexing is set = Yes

regards.

sazz.'s picture

What is the number for media multiplexing ? 1 is the default which means you will able to run only 1 job but you have 4 drives which makes it 4 jobs.

Edit:I just checked the screenshot and you have Max concurrent drives set to 4 and multiplexing set to 4. Actually you should be able to run 4*4 =16 jobs if everything is set correctly. The other jobs that are getting queued are pointing to the same storage unit? Can you post the detailed status of the queued job

sazz.'s picture

lsscsi shows 8 tapes coming from different path but NBU shows 4 of them and that is also repetitive. Please check if you have the configuration correctly setup or run the device configuration again if you have recently changed something.

Marianne's picture

I know that you have said yes, but please double-check MPX level in all schedules.
Insufficient media can also cause this.

Also check Job details for queued to job to see the reason for job being queued.

Sometimes orphaned device allocations can cause this.
You can troubleshoot this with 'nbrbutil -dump', examine MDS allocations and release media and/or drive allocations that you know are not currently in use.
nbrbutil -releaseMDS <mdsAlocationKey>
Quick fix is to reset all allocations when no backups are running:
nbrbutil -resetAll

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

RamNagalla's picture

I Would like to see what is showing in Queud jobs detail status. could you post it?

do you say any Drives in Restart status when you check them for Device moniter?

what is scan and tpautoconf -t shoiwng?

Npalma's picture
Dear:
 
the job queue means that no samples available:
 
 
 
 
Regards.
Marianne's picture

The job queue says 'no drives are available'.

Please see last part of my previous post:

Sometimes orphaned device allocations can cause this.
You can troubleshoot this with 'nbrbutil -dump', examine MDS allocations and release media and/or drive allocations that you know are not currently in use.
nbrbutil -releaseMDS <mdsAlocationKey>

Quick fix is to reset all allocations when no backups are running:
nbrbutil -resetAll

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Npalma's picture

Marianne:

tpautoconf -t

[root@backup6m-bkp bin]# ./tpautoconf -t

TPAC60 IBM     ULT3580-HH5     C7RD 1068017095 -1 -1 -1 -1 /dev/nst0 - - 
TPAC60 IBM     ULT3580-HH5     C7RD 1068017282 -1 -1 -1 -1 /dev/nst7 - - 
TPAC60 IBM     ULT3580-HH5     C7RD 1068017271 -1 -1 -1 -1 /dev/nst5 - - 
TPAC60 IBM     ULT3580-HH5     C7RD 1068016944 -1 -1 -1 -1 /dev/nst3 - - 
TPAC60 IBM     ULT3580-HH5     C7RD 1068017095 -1 -1 -1 -1 /dev/nst1 - - 
TPAC60 IBM     ULT3580-HH5     C7RD 1068017282 -1 -1 -1 -1 /dev/nst6 - - 
TPAC60 IBM     ULT3580-HH5     C7RD 1068017271 -1 -1 -1 -1 /dev/nst4 - - 
TPAC60 IBM     ULT3580-HH5     C7RD 1068016944 -1 -1 -1 -1 /dev/nst2 - - 
[root@backup6m-bkp bin]# 
 
Drive 4 is the only drive that works:
 
 
Regards.
Npalma's picture

Marienne:

[root@backup6m-bkp bin]# ./nborautil -resetAll
 
nborautil [-eol [value]]
[-noindent]
[-bprd]
[-client <client>]
[-master <master>]
[-oradumpfile <filename>]
Oracle:<path>
 
 
everything remains the same.
RamNagalla's picture

did you check the resource allocation as suggested by Marianne?

did you check if the Drives are in RESTART state when you check from Device moniter ?

Npalma's picture

Nagalla:

They are in state "NO", so that this is right:

Regards.

Marianne's picture

Command is:

nbrbutil NOT nborautil and is located /usr/openv/netbackup/bin/admincmd

Best to first check resource allocation with 'nbrbutil -dump'.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Npalma's picture

Marienne:

[root@backup6m-bkp admincmd]# ./nbrbutil -dump
 
Allocation Requests
(AllocationRequestSeq 
index=0 (AllocationRequest: id={E3DCCC36-7443-11E2-8D6C-8185E32C2D5C} priority=0 secondPriority=26273 userid=jobid=27811 description=THE_BACKUP_JOB-27811-{E3DCCC36-7443-11E2-8D6C-8185E32C2D5C} (RequestSeq 
index=0 (Request provider=MPXProvider resourcename=MpxRequest-27811  userSequence=-1 (MPXGroupRequest maxmpx=2 media=(RequestSeq 
index=0 (Request provider=DriveOperationProvider resourcename=backup6m-bkp-hcart2-robot-tld-0  userSequence=0 (StorageUnitRequest: storageUnit=(StorageUnitRequest: storageUnit=backup6m-bkp-hcart2-robot-tld-0 mediaPool=Esign retentionLevel=1 mustUseLocalMediaServer=no failOnError=no mpxRequired=no mustBeNdmp=no getMaxFreeSpace=no minFreeSpaceKBytes=1605838 usageType=1 client=MP-Oracle-Prod stuSubType=-1 diskGroupName= storageServerType= shareGroup=*ANY* isNdmp=false isTirRestore=false isFlashbackupRestore=false isBlockMapRead=false isCatalogBackup=false isGcsCatalogBackup=false isVMWare=false isLifeCycle=false preferVtlToDirectAttachedTape=false backupCopy=-1 isGranularExchange=false REQ_IS_HYPER_V=false REQ_IS_VXVI=false REQ_IS_EXCHANGE14=false) preferredMediaServer= requiredMediaServer= previousStuName= previousStuType=0)))
))
index=1 (Request provider=NamedResourceProvider resourcename=backup6m-bkp.NBU_CLIENT.MAXJOBS.MP-Oracle-Prod  userSequence=-1 (CountedResourceRequest resourcename=backup6m-bkp.NBU_CLIENT.MAXJOBS.MP-Oracle-Prod max=5))
index=2 (Request provider=NamedResourceProvider resourcename=backup6m-bkp.NBU_POLICY.MAXJOBS.e-Sign-MP-Oracle-Test-App  userSequence=-1 (CountedResourceRequest resourcename=backup6m-bkp.NBU_POLICY.MAXJOBS.e-Sign-MP-Oracle-Test-App max=2147483647)))
))
 
 
Allocations 
(AllocationSeq 
index=0 (Allocation: id={DAEF58AA-7443-11E2-BFB3-11C862433377} provider=NamedResourceProvider resourcename=backup6m-bkp.NBU_CLIENT.MAXJOBS.000-DC-DR-PrivCloud-bkp masterserver=backup6m-bkp groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=27808" named resource allocation)
index=1 (Allocation: id={DAEF5918-7443-11E2-A7CE-21B9AD5C9B66} provider=NamedResourceProvider resourcename=backup6m-bkp.NBU_POLICY.MAXJOBS.000-DC-DR-PrivCloud masterserver=backup6m-bkp groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=27808" named resource allocation)
index=2 (Allocation: id={DAEF553A-7443-11E2-8A3B-3F9F263824BF} provider=MPXProvider resourcename=anMPXGroup masterserver=backup6m-bkp groupid={DAEF5440-7443-11E2-9339-2EB668609696} userSequence=-1 userid="jobid=27808" (MPXGroupAllocation: groupId={DAEF5440-7443-11E2-9339-2EB668609696}))
index=3 (Allocation: id={DAEE8D12-7443-11E2-9B66-C0B9BF0347BB} provider=MPXProvider resourcename=backup6m-bkp-hcart2-robot-tld-0 masterserver=backup6m-bkp groupid={DAEF5440-7443-11E2-9339-2EB668609696} userSequence=0 userid="jobid=27808" (Media_Drive_Allocation_Record: allocationKey=13724 (Media_Drive_Record: MediaKey=4000044 MediaId=000019 MediaServer=backup6m-bkp DriveKey=2000089 DriveName=IBM.ULT3580-HH5.001 PrimaryPath=/dev/nst6 PoolName=Data-Recovery RobotNum=0 RobotType=8 MediaTypeName=NetBackup HCART2 DriveTypeName=NetBackup HCART2 NdmpControlHost= RetentionLevel=1 PolicyType=2 JobType=1 MasterServer=backup6m-bkp) (Storage_Unit_Record: STU=backup6m-bkp-hcart2-robot-tld-0 STUType=2 MasterServer=backup6m-bkp MediaServer=backup6m-bkp RobotType=8 RobotNumber=0 Density=14 OnDemandOnly=0 ConcurrentJobs=4 ActiveJobs=0 MaxMultiplexing=4 NdmpAttachHost= AbsolutePath=) (Bptm_Strings_Record: 0="MEDIADB 1 13724 000019 4000044 ------ 14 1360361309 1360392503 1361602103 0 1241188001 5 5 1 5 0 512 1024 0 19393577 0" 1="VOLUME 1 000019 4000044 000019 Data-Recovery IMATION 0D85852240 14 8 0 12 0 {00000000-0000-0000-0000-000000000000} 0" 2="DRIVE 3 IBM.ULT3580-HH5.001 2000089 1068017282 /dev/nst6 -1 -1 -1 -1 -1 -1 -1 -1 *NULL* *NULL* *NULL* *NULL* 1 0 0 1 0 0" 3="STORAGE 1 backup6m-bkp-hcart2-robot-tld-0 14 1048575 2 1 0 0 backup6m-bkp backup6m-bkp *NULL*" 4="DISKGROUP 0 6 *NULL* 6 *NULL* 6 *NULL*" 5="DISKVOLUME 0 6 *NULL* 6 *NULL* 0" 6="DISKMOUNTPOINT 0 6 *NULL*" ) TpReqFileName=)))
 
MDS allocations in EMM: 
 
MdsAllocation: allocationKey=13255 jobType=16 mediaKey=4000030 mediaId=000007 driveKey=0 driveName= drivePath= stuName= masterServerName=backup6m-bkp mediaServerName=backup6m-bkp ndmpTapeServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0
MdsAllocation: allocationKey=13257 jobType=16 mediaKey=4000033 mediaId=000010 driveKey=0 driveName= drivePath= stuName= masterServerName=backup6m-bkp mediaServerName=backup6m-bkp ndmpTapeServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0
MdsAllocation: allocationKey=13259 jobType=16 mediaKey=4000031 mediaId=000008 driveKey=0 driveName= drivePath= stuName= masterServerName=backup6m-bkp mediaServerName=backup6m-bkp ndmpTapeServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0
MdsAllocation: allocationKey=13724 jobType=1 mediaKey=4000044 mediaId=000019 driveKey=2000089 driveName=IBM.ULT3580-HH5.001 drivePath=/dev/nst6 stuName=backup6m-bkp-hcart2-robot-tld-0 masterServerName=backup6m-bkp mediaServerName=backup6m-bkp ndmpTapeServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fatPipeKey=0 scsiResType=1 serverStateFlags=1
RamNagalla's picture

you have 3 jobs with the job type =16 which is  Media contents, and this job will trigger when you run the command 

bpmedialist -U -mcontents [-m <media_id>]

it looks like they are orphaned jobs.

release those, mds allocations

nbrbutil -releaseMDS 13255

nbrbutil -releaseMDS 13257

nbrbutil -releaseMDS 13259

and if that does not help,  i would prefer to go with Netbackup restart.

SOLUTION
Marianne's picture

You can see 4 device allocations - all Media:

MDS allocations in EMM: 
 
MdsAllocation: allocationKey=13255 jobType=16 mediaKey=4000030 mediaId=000007 driveKey=0 driveName= drivePath= stuName= masterServerName=backup6m-bkp mediaServerName=backup6m-bkp ndmpTapeServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0
MdsAllocation: allocationKey=13257 jobType=16 mediaKey=4000033 mediaId=000010 driveKey=0 driveName= drivePath= stuName= masterServerName=backup6m-bkp mediaServerName=backup6m-bkp ndmpTapeServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0
MdsAllocation: allocationKey=13259 jobType=16 mediaKey=4000031 mediaId=000008 driveKey=0 driveName= drivePath= stuName= masterServerName=backup6m-bkp mediaServerName=backup6m-bkp ndmpTapeServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=0
MdsAllocation: allocationKey=13724 jobType=1 mediaKey=4000044 mediaId=000019 driveKey=2000089 driveName=IBM.ULT3580-HH5.001 drivePath=/dev/nst6 stuName=backup6m-bkp-hcart2-robot-tld-0 masterServerName=backup6m-bkp mediaServerName=backup6m-bkp ndmpTapeServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fatPipeKey=0 scsiResType=1 serverStateFlags=1
 
Each of these 4 Media allocations assume one drive for each media-id.
3 of those media allocations do not have drives allocated - only media id 00019 is associated with a drive.
You need to release the first 3 Allocation Id's one-by-one, or do a ResetAll.
nbrbutil -releaseMDS 13255
nbrbutil -releaseMDS 13257
nbrbutil -releaseMDS 13259
Please note that ResetAll will cancel any active jobs.
 
***EDIT****
Oops - too slow!! 
(Multi-tasking not doing very well today...)

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Npalma's picture
I believe this to work, then I could explain a little more detailed than what cause?
 
 
 
 
Greetings.
Marianne's picture

No. If we look at the Allocation Ids, we can see that the Ids are a lot smaller than the current active one (13724). I cannot see any timestamp.

This means that 'something' happened some time ago - maybe YOU can tell us?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Npalma's picture
I do not know, back from my vacation I found this in the system :) just added a new set of tapes, maybe the process damage the existing inventory?
 
Point to part Marianne, Nagalla, thank you very much!
Marianne's picture

As per Nagalla's post:

 3 jobs with the job type =16 which is  Media contents

Seems someone started 'Media contents report' and these jobs got hung/orphaned. More than this cannot be explained with the amount of information available.

What you now have is the ability to troubleshoot and solve the problem if it ever happens again...

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links