Video Screencast Help

Drives were down and struck with media

Created: 08 May 2013 • Updated: 03 Jun 2013 | 12 comments
This issue has been solved. See solution.

Hi Team,

We are running Netbackup Master on AIX 5.3 with netbackup version of 7.1.0.4.

Now we have configured the drives on windows media server.

Below is the output from master server with vmoprcmd -h -d ds

                                  DRIVE STATUS
 
Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart3 DOWN-TLD            Yes   X36185  X36185   Yes     Yes        0
  1 hcart2   TLD                -                     No       -         0
  2 hcart    TLD               Yes   X41132  X41132   Yes     Yes        0
  3 hcart2   TLD                -                     No       -         0
  4 hcart2   TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  6 hcart    TLD               Yes   X45582  X45582   Yes     Yes        0
  7 hcart    TLD               Yes   X45996  X45996   Yes     Yes        0
  8 hcart    TLD               No            X46978   Yes     Yes        0
  9 hcart  DOWN-TLD            No            X46979   Yes     Yes        0
 10 hcart2   TLD                -                     No       -         0
 11 hcart3 DOWN-TLD             -                     No       -         0
 12 hcart3 DOWN-TLD             -                     No       -         0
 13 hcart3 DOWN-TLD            Yes   X31520  X31520   Yes     Yes        0
 14 hcart2   TLD                -                     No       -         0
 15 hcart3 DOWN-TLD            Yes   X32783  X32783   Yes     Yes        0
 16 hcart3 DOWN-TLD             -                     No       -         0
 17 hcart3 DOWN-TLD             -                     No       -         0
 18 hcart3 DOWN-TLD            Yes   X36436  X36436   Yes     Yes        0
 19 hcart  DOWN-TLD             -                     No       -         0
 20 hcart  DOWN-TLD            Yes   X45334  X45334   Yes     Yes        0
 21 hcart  DOWN-TLD            Yes   X46631  X46631   Yes     Yes        0
 22 hcart  DOWN-TLD             -                     No       -         0
 23 hcart  DOWN-TLD             -                     No       -         0
 24 hcart  DOWN-TLD             -                     No       -         0
 25 hcart  DOWN-TLD            Yes   X45836  X45836   Yes     Yes        0
 26 hcart2   TLD                -                     No       -         0
 27 hcart    TLD                -                     No       -         0
 28 hcart    TLD                -                     No       -         0
 29 hcart    TLD               Yes   X45240  X45240   Yes     Yes        0
 30 hcart    TLD                -                     No       -         0
 

When I try to reset or bring the drive up I get an error invalid drive number.

I tried restarting the nb services on the media server still the issue persists

I see missing paths for the drives which are down above.

Operating Systems:

Comments 12 CommentsJump to latest comment

mph999's picture

If you have missing paths, this means the operating system does not have a path to the drive, NetBackup is not at fault.

Fix the SAN issue, them check all drives are visible in the output of /usr/openv/volmgr/bin/scan.

Any drives missing in scan, need to be fixed at the os level.

Once scan shows everything iy should, reconfigure dirives in NBU.

Martin

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
yngrblajay's picture

Its a windows media server I can see the drives available under device manager in windows server.

But it shows the missing path

Marianne's picture

What is the Windows version on the media server?

If 2003, have you stopped and disabled Removable Storage Manager?

Please show us output of all of the following commands on the media server:

(cd to <install-path>\veritas\volmgr\bin)

scan

tpconfig -l

vmoprcmd -d

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

yngrblajay's picture

------------------------------------------------------------
Device Name  : "Tape4"
Passthru Name: "Tape4"
Volume Header: ""
Port: 2; Bus: 0; Target: 4; LUN: 0
Inquiry    : "IBM     ULT3580-TD4     C7Q0"
Vendor ID  : "IBM     "
Product ID : "ULT3580-TD4     "
Product Rev: "C7Q0"
Serial Number: "0007869349"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "IBM     ULT3580-TD4     0007869349"
Device Type    : SDT_TAPE
NetBackup Drive Type: 3
Removable      : Yes
Device Supports: SCSI-3
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "Tape5"
Passthru Name: "Tape5"
Volume Header: ""
Port: 2; Bus: 0; Target: 5; LUN: 0
Inquiry    : "IBM     ULT3580-TD4     C7Q0"
Vendor ID  : "IBM     "
Product ID : "ULT3580-TD4     "
Product Rev: "C7Q0"
Serial Number: "0007867903"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "IBM     ULT3580-TD4     0007867903"
Device Type    : SDT_TAPE
NetBackup Drive Type: 3
Removable      : Yes
Device Supports: SCSI-3
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "Tape6"
Passthru Name: "Tape6"
Volume Header: ""
Port: 2; Bus: 0; Target: 6; LUN: 0
Inquiry    : "IBM     ULT3580-TD4     C7Q0"
Vendor ID  : "IBM     "
Product ID : "ULT3580-TD4     "
Product Rev: "C7Q0"
Serial Number: "0007875760"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "IBM     ULT3580-TD4     0007875760"
Device Type    : SDT_TAPE
NetBackup Drive Type: 3
Removable      : Yes
Device Supports: SCSI-3
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "Tape7"
Passthru Name: "Tape7"
Volume Header: ""
Port: 2; Bus: 0; Target: 7; LUN: 0
Inquiry    : "IBM     ULT3580-TD3     93GP"
Vendor ID  : "IBM     "
Product ID : "ULT3580-TD3     "
Product Rev: "93GP"
Serial Number: "0007809028"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "IBM     ULT3580-TD3     0007809028"
Device Type    : SDT_TAPE
NetBackup Drive Type: 16
Removable      : Yes
Device Supports: SCSI-3
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : ""
Passthru Name: ""
Volume Header: ""
Port: 2; Bus: 0; Target: 7; LUN: 1
Inquiry    : "IBM     03584X32        8980"
Vendor ID  : "IBM     "
Product ID : "03584X32        "
Product Rev: "8980"
Serial Number: "0000078182230694"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "IBM     03584X32        0000078182230694"
Device Type    : SDT_CHANGER
NetBackup Robot Type: 8
Removable      : Yes
Device Supports: SCSI-3
Number of Drives : 15
Number of Slots  : 794
Number of Media Access Ports: 32
Drive 1 Serial Number      : "0007809028"
Drive 2 Serial Number      : "0007809032"
Drive 3 Serial Number      : "0007809034"
Drive 4 Serial Number      : "0007809072"
Drive 5 Serial Number      : "0007809003"
Drive 6 Serial Number      : "0007808854"
Drive 7 Serial Number      : "0007808916"
Drive 8 Serial Number      : "0007809078"
Drive 9 Serial Number      : "0007809006"
Drive 10 Serial Number      : "0007809051"
Drive 11 Serial Number      : "0007809026"
Drive 12 Serial Number      : "0007809019"
Drive 13 Serial Number      : "0007837777"
Drive 14 Serial Number      : "0007825085"
Drive 15 Serial Number      : "0007827969"
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "Tape8"
Passthru Name: "Tape8"
Volume Header: ""
Port: 2; Bus: 0; Target: 8; LUN: 0
Inquiry    : "IBM     ULT3580-TD3     93GP"
Vendor ID  : "IBM     "
Product ID : "ULT3580-TD3     "
Product Rev: "93GP"
Serial Number: "0007809032"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "IBM     ULT3580-TD3     0007809032"
Device Type    : SDT_TAPE
NetBackup Drive Type: 16
Removable      : Yes
Device Supports: SCSI-3
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "Tape9"
Passthru Name: "Tape9"
Volume Header: ""
Port: 2; Bus: 0; Target: 9; LUN: 0
Inquiry    : "IBM     ULT3580-TD3     93GP"
Vendor ID  : "IBM     "
Product ID : "ULT3580-TD3     "
Product Rev: "93GP"
Serial Number: "0007809034"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "IBM     ULT3580-TD3     0007809034"
Device Type    : SDT_TAPE
NetBackup Drive Type: 16
Removable      : Yes
Device Supports: SCSI-3
Flags : 0x0
Reason: 0x

VERITAS\Volmgr\bin>tpconfig -l
Device Robot Drive       Robot                    Drive          Device
 
Type     Num Index  Type DrNum Status  Comment    Name           Path
 
robot      1    -    TLD    -       -  -          -              mdsrvlcow01
  drive    -    0 hcart3   14    DOWN  -          Rob003-Fx-14   MISSING_PATH:{2
,0,14,0}:00078250
  drive    -   11 hcart3   13    DOWN  -          Rob003-Fx-13   MISSING_PATH:{2
,0,13,0}:00078377
  drive    -   12 hcart3    6    DOWN  -          Rob003-Fx-6    MISSING_PATH:{2
,0,12,0}:00078088
  drive    -   13 hcart3    5    DOWN  -          Rob003-Fx-5    MISSING_PATH:{2
,0,11,0}:00078090
  drive    -   15 hcart3    4    DOWN  -          Rob003-Fx-4    MISSING_PATH:{2
,0,10,0}:00078090
  drive    -   16 hcart3    3    DOWN  -          Rob003-Fx-3    MISSING_PATH:{2
,0,9,0}:000780903
  drive    -   17 hcart3    2    DOWN  -          Rob003-Fx-2    MISSING_PATH:{2
,0,8,0}:000780903
  drive    -   18 hcart3    1    DOWN  -          Rob003-Fx-1    MISSING_PATH:{2
,0,7,0}:000780902
robot      2    -    TLD    -       -  -          -              mdsrvlcow01
  drive    -    1 hcart2    2      UP  -          Rob002-Fx-2    {2,0,24,0}
 
  drive    -    3 hcart2    1      UP  -          Rob002-Fx-1    {2,0,23,0}
 
  drive    -    4 hcart2    6      UP  -          Rob002-Fx-6    {2,0,27,0}
 
  drive    -   10 hcart2    4      UP  -          Rob002-Fx-4    {2,0,26,0}
 
  drive    -   14 hcart2    3      UP  -          Rob002-Fx-3    {2,0,25,0}
 
  drive    -   26 hcart2    5      UP  -          Rob002-Fx-5    {2,0,28,0}
 
robot      4    -    TLD    -       -  -          -              mdsrvlcow01
  drive    -    2  hcart   28      UP  -          Rob4-F7-Drv6A  {2,0,20,0}
 
  drive    -    5  hcart   16      UP  -          Rob4-F6-Drv6A  {2,0,21,0}
 
  drive    -    6  hcart    7      UP  -          Rob4-F5-Drv5B  {2,0,22,0}
 
  drive    -    7  hcart    6      UP  -          Rob004-Fx-6    {2,0,29,0}
 
  drive    -    8  hcart    4      UP  -          Rob4-F5-Drv2A  {2,0,30,0}
 
  drive    -    9  hcart    1    DOWN  -          Rob4-F4-Drv4A  MISSING_PATH:{2
,0,15,0}:00078534
  drive    -   19  hcart    9    DOWN  -          Rob4-F5-Drv7B  MISSING_PATH:{2
,0,6,0}:000787576
  drive    -   20  hcart   24    DOWN  -          Rob4-F7-Drv2A  MISSING_PATH:{2
,0,5,0}:000786790
  drive    -   21  hcart   13    DOWN  -          Rob4-F6-Drv3A  MISSING_PATH:{2
,0,4,0}:000786934
  drive    -   22  hcart   25    DOWN  -          Rob4-F7-Drv3A  MISSING_PATH:{2
,0,3,0}:000786796
  drive    -   23  hcart   23    DOWN  -          Rob4-F7-Drv1A  MISSING_PATH:{2
,0,2,0}:000786804
  drive    -   24  hcart   12    DOWN  -          Rob4-F6-Drv2A  MISSING_PATH:{2
,0,1,0}:000787134
  drive    -   25  hcart   11    DOWN  -          Rob4-F6-Drv1A  MISSING_PATH:{2
,0,0,0}:000787116
  drive    -   27  hcart   15      UP  -          Rob4-F6-Drv5A  {2,0,16,0}
 
  drive    -   28  hcart   26      UP  -          Rob4-F7-Drv4A  {2,0,17,0}
 
  drive    -   29  hcart   27      UP  -          Rob4-F7-Drv5A  {2,0,18,0}
 
  drive    -   30  hcart   14      UP  -          Rob4-F6-Drv4A  {2,0,19,0}
 
 
Volmgr\bin>vmoprcmd -d
 
                                PENDING REQUESTS
 
                                     <NONE>
 
                                  DRIVE STATUS
 
Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Emdsrvl.  ReqId
  0 hcart3 DOWN-TLD             -                     No       -         0
  1 hcart2   TLD                -                     No       -         0
  2 hcart    TLD                -                     No       -         0
  3 hcart2   TLD                -                     No       -         0
  4 hcart2   TLD                -                     No       -         0
  5 hcart    TLD                -                     No       -         0
  6 hcart    TLD               Yes   X45582  X45582   Yes     Yes        0
  7 hcart    TLD                -                     No       -         0
  8 hcart    TLD               No            X46978   Yes     Yes        0
  9 hcart  DOWN-TLD             -                     No       -         0
 10 hcart2   TLD                -                     No       -         0
 11 hcart3 DOWN-TLD             -                     No       -         0
 12 hcart3 DOWN-TLD             -                     No       -         0
 13 hcart3 DOWN-TLD             -                     No       -         0
 14 hcart2   TLD                -                     No       -         0
 15 hcart3 DOWN-TLD            Yes   X32783  X32783   Yes     Yes        0
 16 hcart3 DOWN-TLD             -                     No       -         0
 17 hcart3 DOWN-TLD             -                     No       -         0
 18 hcart3 DOWN-TLD             -                     No       -         0
 19 hcart  DOWN-TLD             -                     No       -         0
 20 hcart  DOWN-TLD             -                     No       -         0
 21 hcart  DOWN-TLD            Yes   X46631  X46631   Yes     Yes        0
 22 hcart  DOWN-TLD             -                     No       -         0
 23 hcart  DOWN-TLD             -                     No       -         0
 24 hcart  DOWN-TLD             -                     No       -         0
 25 hcart  DOWN-TLD             -                     No       -         0
 26 hcart2   TLD                -                     No       -         0
 27 hcart    TLD                -                     No       -         0
 28 hcart    TLD                -                     No       -         0
 29 hcart    TLD               Yes   X45240  X45240   Yes     Yes        0
 30 hcart    TLD                -                     No       -         0
 
                             ADDITIONAL DRIVE STATUS
 
Drv DriveName            Shared    Assigned        Comment
  0 Rob003-Fx-14          Yes      -
  1 Rob002-Fx-2           Yes      -
  2 Rob4-F7-Drv6A         Yes      -
  3 Rob002-Fx-1           Yes      -
  4 Rob002-Fx-6           Yes      -
  5 Rob4-F6-Drv6A         Yes      -
  6 Rob4-F5-Drv5B         Yes      mdsrvlcow02.owesp
  7 Rob004-Fx-6           Yes      -
  8 Rob4-F5-Drv2A         Yes      mdsrvlcow01.owesp
  9 Rob4-F4-Drv4A         Yes      -
 10 Rob002-Fx-4           Yes      -
 11 Rob003-Fx-13          Yes      -
 12 Rob003-Fx-6           Yes      -
 13 Rob003-Fx-5           Yes      -
 14 Rob002-Fx-3           Yes      -
 15 Rob003-Fx-4           Yes      mdsrvucpow01.corp.
 16 Rob003-Fx-3           Yes      -
 17 Rob003-Fx-2           Yes      -
 18 Rob003-Fx-1           Yes      -
 19 Rob4-F5-Drv7B         Yes      -
 20 Rob4-F7-Drv2A         Yes      -
 21 Rob4-F6-Drv3A         Yes      mdsrvUSGow01
 22 Rob4-F7-Drv3A         Yes      -
 23 Rob4-F7-Drv1A         Yes      -
 24 Rob4-F6-Drv2A         Yes      -
 25 Rob4-F6-Drv1A         Yes      -
 26 Rob002-Fx-5           Yes      -
 27 Rob4-F6-Drv5A         Yes      -
 28 Rob4-F7-Drv4A         Yes      -
 29 Rob4-F7-Drv5A         Yes      mdsrvlcow02.owesp
 30 Rob4-F6-Drv4A         Yes      -
 
 

Mark_Solutions's picture

Back to basics for a moment here ... just as a quick check because the title of the thread is something i have heard a lot of times....

How do you change your tapes?

Do you use eject to the load ports and load them from the load ports or could some have been loaded manually directly into slots?

What i have seen is when taoes are being writtent to when new tapes are put into the library - but those tapes are put into slots that belong to the tapes in the drive (and they cannot be ejected into any other slot automatically) - so NetBackup leaves the tape in the drive and downs it.

Such events do get logged to the Event logs on Windows servers so you should find the reason for the down drive in the event viewer

You can try using robtest to check this.

s d will show the drives - a tape in a drive should show the slot number it belongs to - it is doesn't something has gone wrong - if it does then try s s to show the slots and see if a tape is already in that slot number

If so use m d1 s10 (as an example of moving the tape in drive 1 to slot 10) to move the tapes to an empty slot

You can then re-run the inventory so that NBU knows where they all are after which you should be able to UP the drive.

One other cause can be the TUR issue that happens on window so rgaldless of the cause of your issue (and i have seen a faulty library cause your symptoms too) it is worth following this tech note - it applies to all windows versions:

http://support.microsoft.com/kb/842411

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

mph999's picture

"Its a windows media server I can see the drives available under device manager in windows server."

Unfortunately, this does NOT mean that the drives will work.  I am not sure how often windows updates its devices, but if you deleted the drives in device manager and rescanned, it is possible they wont all come back.  I've  also seen pleanty of cases where the drives are visible, but just don't communacte properly (the case that springs to mind turned out to be an HBA issue).

The library shows 15 drives, scan output shows x6 drives.  If all the drives are meant to be on this server (???) then where are the other 9.

Martin

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
yngrblajay's picture

I have tried working with the robtest on the master server where the drives are physically attached.

But didn't find the tape which media server is showing.

Tell me if i reconfigure with the device configuration wizard does that need restart of the master server nb services.

I dont have issues in recycling the media server services but I will not be able to perform the recycle activity on master server.

Marianne's picture

tpconfig and vmoprcmd shows 31 drives!

Do you REALLY have that many tape drives connected to this media server? I have not seen ANY media server on any OS being able to handle that many tape drives.

The ONLY ones seen by the OS are the 6 drives that are reported by 'scan'.

You will have to delete all devices on this media server, check zoning, ensure Persistent Binding is enabled at HBA level, confirm Removable Storage Manager service is stopped and disabled (if W2003), clean out tape drives in Device Manager, then rescan for devices.

After this you should be able to re-add devices in NBU.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

yngrblajay's picture

@Marianne,

The drives were working fine till yesterday, I have rescan the devices in the device management on the windows server.

Even the Removable Storage Manager is disabled

Now my question is if I start reconfiguring the devices using device configuration wizard do I need to restart the services on my master server.

Mark_Solutions's picture

Device config wizard wil only re-start the device manager service on the media servers affected by it

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

SOLUTION
yngrblajay's picture

Issue is resolved now.

I have performed the device configuration on the media server and it has found all the drive paths which are fixing.

Now the drives are up and running successfully

Marianne's picture

Please be fair and mark solution for one of us who recommended device reconfig?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links