Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Queued Jobs

Created: 16 Jun 2013 | 5 comments

Hi Everyone,

This was an issue which i have come acrossed while i was doing the regular health check.

 

There were a a few jobs which were in Queue for around 14 hrs and are not being active inspite of the drives being available.

So, i had checked the library console and found a few RAS tickets which i had resolved and everything seems to be fine from the Hardware level.

 

Still i am observing that the Queued jobs are still in Q and not being active.

 

Could someone please help me out in this.

 

Thanks in advance

Operating Systems:

Comments 5 CommentsJump to latest comment

Marianne's picture

What is the exact reason for jobs being queued in Job Details?

Please also show us output from 'vmoprcmd -d' as well as the MDS Allocation part of 'nbrbutil -dump' output.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

sriharishkandula's picture

This is the output for vmoprcmd -d

 

 

  1 hcart    TLD                -                     No       -         0
  1 hcart    TLD                -                     No       -         0
  1 hcart    TLD                -                     No       -         0
  1 hcart    TLD                -                     No       -         0
  1 hcart    TLD                -                     No       -         0
  1 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  3 hcart    TLD                -                     No       -         0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  4 hcart    TLD               Yes   SG2225  SG2225   Yes     Yes        0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  6 hcart    TLD                -                     No       -         0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  7 hcart    TLD               Yes   SG0564  SG0564   Yes     Yes        0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  8 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0
  9 hcart    TLD                -                     No       -         0

                             ADDITIONAL DRIVE STATUS

Drv DriveName            Shared    Assigned        Comment
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  0 IBM.ULTRIUM-TD4.009   Yes      tvmwpbkp54.ad.b
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  1 IBM.ULTRIUM-TD4.008   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  2 IBM.ULTRIUM-TD4.010   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  3 IBM.ULTRIUM-TD4.011   Yes      -
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  4 IBM.ULTRIUM-TD4.012   Yes      tvmwpbkp55.ad.b
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  5 IBM.ULTRIUM-TD4.013   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  6 IBM.ULTRIUM-TD4.014   Yes      -
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  7 IBM.ULTRIUM-TD4.015   Yes      tvmwpbkp54.ad.b
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  8 IBM.ULTRIUM-TD4.016   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -
  9 IBM.ULTRIUM-TD4.017   Yes      -

sriharishkandula's picture

MDS Allocation:

 

MDS allocations in EMM:

        MdsAllocation: allocationKey=51378 jobType=1 mediaKey=4001591 mediaId=TV
0626 driveKey=2000823 driveName=IBM.ULTRIUM-TD4.003 drivePath=/dev/nst3 stuName=
tvlxpbkp01-hcart-robot-tld-1 masterServerName=tvmwpbkp51 mediaServerName=tvlxpbk
p01 ndmpTapeServerName= diskVolumeKey=0 mountKey=0 linkKey=0 fatPipeKey=0 scsiRe
sType=1 serverStateFlags=1
        MdsAllocation: allocationKey=51384 jobType=1 mediaKey=4010362 mediaId=SG
2220 driveKey=2000936 driveName=IBM.ULTRIUM-TD4.009 drivePath=c128t0l1 stuName=t
vmwpbkp54-hcart-robot-tld-0-SBIDPNS03DM2 masterServerName=tvmwpbkp51 mediaServer
Name=tvmwpbkp54 ndmpTapeServerName=SBIDPNS03DM2 diskVolumeKey=0 mountKey=0 linkK
ey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=1
        MdsAllocation: allocationKey=51385 jobType=1 mediaKey=4009060 mediaId=SG
0564 driveKey=2000930 driveName=IBM.ULTRIUM-TD4.015 drivePath=c144t0l4 stuName=t
vmwpbkp54-hcart-robot-tld-0-SBIDPNS03DM4 masterServerName=tvmwpbkp51 mediaServer
Name=tvmwpbkp54 ndmpTapeServerName=SBIDPNS03DM4 diskVolumeKey=0 mountKey=0 linkK
ey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=1
        MdsAllocation: allocationKey=51387 jobType=1 mediaKey=4010357 mediaId=SG
2225 driveKey=2000927 driveName=IBM.ULTRIUM-TD4.012 drivePath=c144t0l1 stuName=t
vmwpbkp55-hcart-robot-tld-0-SBIDPNS03DM3 masterServerName=tvmwpbkp51 mediaServer
Name=tvmwpbkp55 ndmpTapeServerName=SBIDPNS03DM3 diskVolumeKey=0 mountKey=0 linkK
ey=0 fatPipeKey=0 scsiResType=0 serverStateFlags=1

Marianne's picture

You forgot to show us reason for queued jobs in Activity Monitor Details tab.

Were vmoprcmd and nbrbutil commands done at more or less the same time?

vmoprcmd shows drives in use by media servers tvmwpbkp54 and tvmwpbkp55.
nbrbutil shows allocations for one more media server: tvlxpbkp01. Is there a job active for this media server?

I don't understand the multiple lines per drive in vmoprcmd output. Please tell us more about your environment:

Are all of these drives in one robot?
How many media servers? All sharing the drives?
Amount of drives and MPX per STU?
Are jobs queued for a particular STU?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

sriharishkandula's picture

Hi All sorry for the delayed response...

 

The issue has been resolved...As per Vendor's recommendation we have upgraded the library and the drive firmware after which the services have been bounced across all the master nd media servers after that the issue has been resolved..

 

Thanks a  lot for ur help..cheers..