Video Screencast Help
Protect Your POS Environment Against Retail Data Breaches. Learn More.

ec2009

Created: 04 Dec 2012 • Updated: 21 Dec 2012 | 20 comments
This issue has been solved. See solution.

Hi All,

I am using a Master(V7.1) and a media (V6.5)..Backup for the media server which is being taken is failing with ec2009.

12/03/2012 22:25:12 - requesting resource asprd434-ebr_lto
12/03/2012 22:25:12 - requesting resource asprd212.aldc.att.com.NBU_CLIENT.MAXJOBS.asprd434-ebr.aldc.att.com
12/03/2012 22:25:12 - requesting resource asprd212.aldc.att.com.NBU_POLICY.MAXJOBS.asprd434
12/03/2012 22:25:17 - Error nbjm (pid=5547) NBU status: 2009, EMM status: All compatible drive paths are down, but media is available
All compatible drive paths are down but media is available  (2009)
 

I have restarted the services and also restarted PBX but still getting the same error.

Any suggestions.

Comments 20 CommentsJump to latest comment

Marianne's picture

All compatible drive paths are down

Which OS on Media server?

Troubleshooting needs to start at OS level.
We can provide troubleshooting steps once we have more info.
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Shekaib's picture

@Marianne..OS of media server os Solaris 10..

Marianne's picture

Add VERBOSE entry to /usr/openv/volmgr/vm.conf on the media server.

Restart NBU on media server.

Try to UP the drives.

If drives go DOWN again, check /var/adm/messages for reason.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Shekaib's picture

output for vm.conf

cat vm.conf
VERBOSE
MM_SERVER_NAME = asprd434-ebr.aldc.att.com
ACS_TCP_RPCSERVICE
ACS_CSI_HOSTPORT = aacsls01.aldc.att.com 30031
ACS_SSI_INET_PORT = aacsls01.aldc.att.com 30031
 

 

Please check and also this is ACS tape library.

Marianne's picture

Good!

What does 'vmoprcmd -d' on the media server show?

Have you checked /var/adm/messages for errors?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Shekaib's picture

more messages | grep error
Dec  3 01:01:08 asprd434 automountd[4101]: [ID 784820 daemon.error] server cstadmin.sbc.com not responding
Dec  3 11:49:10 asprd434 vmd[10909]: [ID 715111 daemon.error] volume daemon terminating because it received a signal (15)
Dec  3 11:49:10 asprd434 vmd[10909]: [ID 164182 daemon.error] terminating - daemon terminated (7)
Dec  3 11:52:16 asprd434 tldd[9163]: [ID 886097 daemon.error] daemon terminating because no devices are configured, tldcd has not been started
Dec  4 09:23:07 asprd434 acsd[5290]: [ID 578017 daemon.error] ACS(0) [5290] timed out after waiting 901 seconds for ready, drive (0,8,1,8)
Dec  4 09:24:03 asprd434 ltid[9125]: [ID 926221 daemon.error] Operator/EMM server has DOWN'ed drive A00_LI001_LS8_D08_L3 (device 3)

vmoprcmd -d ds

                                  DRIVE STATUS

Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart3   ACS                -                     No       -         0
  1 hcart3   ACS                -                     No       -         0
  2 hcart    ACS                -                     No       -         0
  3 hcart    ACS                -                     No       -         0

 

Marianne's picture

"grep error" does not help to get the reason for drives being DOWN'ed.
Anyhow, drives are currently up/ACS control. Are backups still failing? If so, use nbrbutil command on the master to reset Media server.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Nagalla's picture

hi,

does this drives are shared?

do you have 2 different denistity of Drives, Hcart3 and Hcart?

please post the outputs of  /usr/openv/volmgr/bin/scan

/usr/openv/volmgr/bin/vmoprcmd -d

/usr/openv/volmgr/bin/tpconfig -d

/usr/openv/volmgr/bin/tpautoconf -t

Shekaib's picture

i did nbrbutil -resetMediaServer  name

restarted backups and they again failed with same ec.

2/04/2012 21:28:41 - requesting resource asprd434-ebr_lto
12/04/2012 21:28:41 - requesting resource asprd212.aldc.att.com.NBU_CLIENT.MAXJOBS.asprd434-ebr.aldc.att.com
12/04/2012 21:28:41 - requesting resource asprd212.aldc.att.com.NBU_POLICY.MAXJOBS.asprd434
12/04/2012 21:28:59 - Error nbjm (pid=5547) NBU status: 2009, EMM status: All compatible drive paths are down, but media is available
All compatible drive paths are down but media is available  (2009

Marianne's picture

Are these 4 ACS drives the only devices on the media server?

How many Storage Units configured for this media server?

We see the following error in /var/adm/messages:

Dec  3 11:52:16 asprd434 tldd[9163]: [ID 886097 daemon.error] daemon terminating because no devices are configured, tldcd has not been started

This indicates that there should be a direct-attached TLD robot with drive(s) on this media server.

What does 'bpstulist -label asprd434-ebr_lto -U' show?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Shekaib's picture

bpstulist -label asprd434-ebr_lto -U

Label:                asprd434-ebr_lto
Storage Unit Type:    Media Manager
Host Connection:      asprd434-ebr.aldc.att.com
Number of Drives:     2
On Demand Only:       no
Max MPX/drive:        4
Density:              hcart - 1/2 Inch Cartridge
Robot Type/Number:    ACS / 0
Max Fragment Size:    2048 MB

Marianne's picture

Is master/EMM server aware of the UP state of the drives?

What does the following command on Master show?

vmoprcmd -h asprd434-ebr.aldc.att.com -d ds

Please also post 'MDS allocations' section of 'nbrbutil -dump'. 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Shekaib's picture

 

 

vmoprcmd -h asprd434-ebr.aldc.att.com -d ds

                                  DRIVE STATUS

Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart3   ACS                -                     No       -         0
  1 hcart3   ACS                -                     No       -         0
  2 hcart  DOWN-ACS             -                     No       -         0
  3 hcart    ACS                -                     No       -         0

 

when i try to reset and up the drive..

 vmoprcmd -reset 2
Unknown drive name
 vmoprcmd -up 2
Invalid Drive Number
 

 

 

Marianne's picture

Are you issuing -reset and -up from master on from media server?

If master, remember to add -h <hostname>.

You need to check messages file to see what is wrong with drive(s). 
Find DOWN in messages file, then look at the lines above that to see what caused this.

This could be part of the problem:

 timed out after waiting 901 seconds for ready, drive (0,8,1,8)

Some issue between OS and device?

Try to troubleshoot outside of NBU - mount tape in drive using ACSLS.

Check status from OS:

mt -f /dev/rmt/## status

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Shekaib's picture

thanks for reminding..i gave cmd from media server and it worked and drive also came up..

Problem which i think is in the tape mounting...

 

2/05/2012 18:45:56 - requesting resource asprd434-ebr_lto
12/05/2012 18:45:56 - requesting resource asprd212.aldc.att.com.NBU_CLIENT.MAXJOBS.asprd434-ebr.aldc.att.com
12/05/2012 18:45:56 - requesting resource asprd212.aldc.att.com.NBU_POLICY.MAXJOBS.asprd434
12/05/2012 18:46:04 - granted resource  asprd212.aldc.att.com.NBU_CLIENT.MAXJOBS.asprd434-ebr.aldc.att.com
12/05/2012 18:46:04 - granted resource  asprd212.aldc.att.com.NBU_POLICY.MAXJOBS.asprd434
12/05/2012 18:46:04 - granted resource  GX0467
12/05/2012 18:46:04 - granted resource  A00_LI001_LS8_D08_L3
12/05/2012 18:46:04 - granted resource  asprd434-ebr_lto
12/05/2012 18:46:05 - started process bpbrm (pid=19317)
12/05/2012 18:46:06 - connecting
12/05/2012 18:46:06 - connected; connect time: 0:00:00
12/05/2012 18:47:07 - mounting GX0467

generally it fails after this.

Try to troubleshoot outside of NBU - mount tape in drive using ACSLS--for this we did try to mount and unmount tape..and it worked fine.

Marianne's picture

Did you confirm that OS is seeing tape mount with 'mt -f /dev/rmt/xx status'?

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Nagalla's picture

Shekaib,

did you ever have the succuess full backup with this Drives?

Did you set the access control in the ACLSLS servers?

do you have 4 Drives connected in OS level?

Shekaib's picture

 mt -f /dev/rmt/0cbn status
/dev/rmt/0cbn: write protected or reserved.

Marianne's picture

Not good....

Status should show something like this:

 

   sense key(0x0)= No Additional Sense   residual= 0   retries= 0
   file no= 0   block no= 0

 

write protected or reserved means one of the systems accessing this drive is holding SCSI reservation on the drive.

Is drive shared or dedicated to this media server? SCSI or Fibre connected?
Have you checked /var/adm/messages for HBA or device-related errors?

To release reservation, try the following:
mt -f <drive_path_name> forcereserve
followed by:
mt -f <drive_path_name> release

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

SOLUTION