Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Robotics Library down on AIX Media server

Created: 10 Jan 2013 | 20 comments

HI

I got fixed missing drive path on my master server, after building sg drivers

Now I am getting error on Aix media server

I have rebuild the devices with this procedure

tpconfig -delete

 rm /dev/rmt34.1

rmdev -dl rmt0

 lsdev -Cc tape

cfgmgr -v

 after these missing drive path got fixed...on AIX media server with SSO

 

/tpconfig -d
Id DriveName Type Residence
Drive Path Status
****************************************************************************
0 Drive040 hcart TLD(1) DRIVE=9
/dev/rmt4.1 UP
1 Drive041 hcart TLD(1) DRIVE=10
/dev/rmt5.1 UP
2 Drive051 hcart TLD(1) DRIVE=11
/dev/rmt6.1 UP
31 Drive036 hcart TLD(1) DRIVE=1
/dev/rmt0.1 UP
32 Drive037 hcart TLD(1) DRIVE=2
/dev/rmt1.1 UP
33 Drive047 hcart TLD(1) DRIVE=3
/dev/rmt2.1 UP
34 Drive048 hcart TLD(1) DRIVE=4
/dev/rmt3.1 UP
49 Drive043 hcart TLD(4) DRIVE=5
/dev/rmt7.1 UP
50 Drive044 hcart TLD(4) DRIVE=6
/dev/rmt8.1 UP
51 Drive045 hcart TLD(4) DRIVE=7
/dev/rmt9.1 UP
52 Drive046 hcart TLD(4) DRIVE=8
/dev/rmt10.1 UP

Currently defined robotics are:
TLD(1) robot control host =
TLD(4) robot control host =

EMM Server =

  

with the scan command i am able to see on drives but not robots

scan | grep -i "Device Type"

Device Type : SDT_TAPE
Device Type : SDT_TAPE
Device Type : SDT_TAPE
Device Type : SDT_TAPE
Device Type : SDT_TAPE
Device Type : SDT_TAPE
Device Type : SDT_TAPE
Device Type : SDT_TAPE
Device Type : SDT_TAPE
Device Type : SDT_TAPE
Device Type : SDT_TAPE

Master server= solaris nbu 7.1

media server = aix nbu 6.5

 

 

Comments 20 CommentsJump to latest comment

RamNagalla's picture

hi,

 

what is your robot control host?

does it your master server or media server?

you can only see the robot connected physically to the robot contorol host

 

what is the robot control host its showing when you run the tpconfig -d in Media server?

does it showing master server?

 

Please provide the output of tpconfig -d and vmoprcmd -d of Media server

Ah45's picture

Robot control host is master server

I can see robots are connected to master server

and here is the output from media server

 

/tpconfig -d
Id DriveName Type Residence
Drive Path Status
****************************************************************************
0 Drive040 hcart TLD(1) DRIVE=9
/dev/rmt4.1 UP
1 Drive041 hcart TLD(1) DRIVE=10
/dev/rmt5.1 UP
2 Drive051 hcart TLD(1) DRIVE=11
/dev/rmt6.1 UP
31 Drive036 hcart TLD(1) DRIVE=1
/dev/rmt0.1 UP
32 Drive037 hcart TLD(1) DRIVE=2
/dev/rmt1.1 UP
33 Drive047 hcart TLD(1) DRIVE=3
/dev/rmt2.1 UP
34 Drive048 hcart TLD(1) DRIVE=4
/dev/rmt3.1 UP
49 Drive043 hcart TLD(4) DRIVE=5
/dev/rmt7.1 UP
50 Drive044 hcart TLD(4) DRIVE=6
/dev/rmt8.1 UP
51 Drive045 hcart TLD(4) DRIVE=7
/dev/rmt9.1 UP
52 Drive046 hcart TLD(4) DRIVE=8
/dev/rmt10.1 UP

Currently defined robotics are:
TLD(1) robot control host = <master server>
TLD(4) robot control host =<master server>

EMM Server = <master server>

 

RamNagalla's picture

so this is fine,

robots are connected to Master server, and you will not see the robots in Media server.

you do not have any issues

unless you are getting the failurs due to dirve errors 

Do you facing any issues and suspecting this might be the reason? lets us know

 

Ah45's picture

Nagalla,

What do you think the problem is for getting error robotics library is down media server when i try to run backup

 

 

Will Restore's picture

Try running Device Configuration Wizard again 

(GUI  Configure Storage Devices)

Will Restore -- where there is a Will there is a way

RamNagalla's picture

 

 

Did you restart the ltid or Netbackup after configuring the Drives?

Provide the output of 

vmoprcmd -d and bpps -x from Media server

 

RamNagalla's picture

and also you may need to run below command from the Master server and try restart netbackup.

nbrbutil -resetmediaserver <mediaservername>  

Ah45's picture

 I have restarted..

 

./vmoprcmd -d

PENDING REQUESTS

<NONE>

DRIVE STATUS

Drv Type Control User Label RecMID ExtMID Ready Wr.Enbl. ReqId
0 hcart AVR - No - 0
1 hcart AVR Yes J40121 J40121 Yes Yes 0
2 hcart AVR - No - 0
31 hcart AVR - No - 0
32 hcart AVR - No - 0
33 hcart AVR - No - 0
34 hcart AVR - No - 0
49 hcart AVR Yes L01073 L01073 Yes Yes 0
50 hcart AVR Yes L00926 L00926 Yes Yes 0
51 hcart AVR Yes L01247 L01247 Yes Yes 0
52 hcart AVR Yes L01228 L01228 Yes Yes 0

ADDITIONAL DRIVE STATUS

Drv DriveName Shared Assigned Comment
0 Drive040 Yes - *NULL*
1 Drive041 Yes <hostname> *NULL*
2 Drive051 Yes - *NULL*
31 Drive036 Yes - *NULL*
32 Drive037 Yes - *NULL*
33 Drive047 Yes - *NULL*
34 Drive048 Yes - *NULL*
49 Drive043 Yes <hostname> *NULL*
50 Drive044 Yes <hostname> *NULL*
51 Drive045 Yes <hostname> *NULL*
52 Drive046 Yes <hostname> *NULL*
 

 

./bpps -x
NB Processes
------------
root 4325404 1 0 Dec 28 - 0:49 /usr/openv/netbackup/bin/nbsl
root 6356996 1 0 Dec 28 - 9:45 /usr/openv/netbackup/bin/nbrmms
root 1573148 1 0 Dec 28 - 0:02 /usr/openv/netbackup/bin/bpcompatd
root 8126830 1 0 Dec 28 - 1:15 /usr/openv/netbackup/bin/nbsvcmon

MM Processes
------------
root 10027152 3867062 0 Dec 28 - 0:06 avrd
root 3867062 1 0 Dec 28 - 1:01 /usr/openv/volmgr/bin/ltid
root 7799136 1 0 Dec 28 - 0:46 vmd

Shared Symantec Processes
-------------------------
root 2424844 1 0 Sep 29 - 0:30 /opt/VRTSpbx/bin/pbx_exchange
cadompem051:root:/usr/openv/netbackup/bin #

 

revaroo's picture

AVR = robot / drive communication issue.

Let's see on this media the contents of /usr/openv/volmgr/vm.conf

 

RamNagalla's picture

Tldd is not running on the Media server.

I am see you have restated the Media server on Dec 28th 

Dec 28 - 1:01 /usr/openv/volmgr/bin/ltid

 

first reset the media server using 

nbrbutil -resetmediaserver <mediaservername>  

then do the clean bounce(restart) of netabckup in media server.

make sure ltid is dead and started again.

Ah45's picture

Nagalla,

Thanks for your reply

I am working on this case more than 15 days...

i have restarted netbackup after missing drive path fix..

and i have run nbrbutil many times..

and i have try to start tldd

 

/usr/openv/volmgr/bin/tldd -v
TLD: No robots are configured 

 

RamNagalla's picture

hi,

tldd suppose to start from ltid.

 

lets gets start again.. to make sure we did not miss something... 

1) create a ltid and tldd directorys in /usr/openv/volmgr/debug/ of media server

2) keep VERBOSE = 5 in vm.conf and bp.conf of media server

3)reset the media server from master server with below command

nbrbutil -resetmediaserver <mediaservername>  

4) bounce the Netabckup in Media server (make sure all process are dead when you try to start again)

5) check vmoprcmd -d and see the Drive status.

if you still see the Drives as AVR mode.

6)then go to master server and run Robtest and check if you are able to perfrom the list the Drives or  not.

7) get the scan -changer output from master server and post

8) Provide the logs of ltid and tldd from the media server as attachment.

9) alsolatest  lastest output of vmoprcmd -d , tpconfig -d and tpautoconf -t commands from both master and Media servers.

 

<<Edit>>

before all those above steps, check if tldcd is running on your robot contorl host(master server),if that is not running , you need to start that first. by restarting ltid.

<<Edit >>

 

 

 

Yasuhisa Ishikawa's picture

If you got failed again, please post output of "bpps -x", "vmoprcmd -d" on both master and media server.

Also check if any relevant errors are logged in /var/adm/messaes on your solaris master server.

Authorized Symantec Consultant(ASC) Data Protection in Tokyo, Japan

Marianne's picture

Never try to start tldd manually. All Media Manager processes are started when ltid is restarted.

We can see in tpconfig that you have two robots configured:

TLD(1) robot control host = <master server>
TLD(4) robot control host =<master server>

I agree with revaroo - AVR status = comms problem with master.

When you restarted NBU on media server, did you also restart PBX?
Restart should be done as follows:
netbackup stop
(wait till all processes are stopped)
/opt/VRTSpbx/bin/vxpbx_exchanged stop
/opt/VRTSpbx/bin/vxpbx_exchanged start
netbackup

Check output carefully while NBU is coming up.
Post output of the following:
bpps -x
tpconfig -l
vmoprcmd -d

Check/verify comms from master server as follows:
nbemmcmd -listhosts -verbose 
nbemmcmd -getemmserver

PS: Are you aware of the fact that NBU 6.5 reached EOSL in Oct 2012?
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Ah45's picture

Marianne,

Netbackup and PBX restarted as you instructed..

still the same error--"Robotics are down on Media server"

bpps -x, tpconfig -l and vmoprcmd -d outputs from Media server are

 

./tpconfig -l
Device Robot Drive Robot Drive Device
Type Num Index Type DrNum Status Comment Name Path
robot 1 - TLD - - - - <master server>
drive - 0 hcart 9 UP *NULL* Drive040 /dev/rmt4.1
drive - 1 hcart 10 UP *NULL* Drive041 /dev/rmt5.1
drive - 2 hcart 11 UP *NULL* Drive051 /dev/rmt6.1
drive - 31 hcart 1 UP *NULL* Drive036 /dev/rmt0.1
drive - 32 hcart 2 UP *NULL* Drive037 /dev/rmt1.1
drive - 33 hcart 3 UP *NULL* Drive047 /dev/rmt2.1
drive - 34 hcart 4 UP *NULL* Drive048 /dev/rmt3.1
robot 4 - TLD - - - - <master server>
drive - 49 hcart 5 UP *NULL* Drive043 /dev/rmt7.1
drive - 50 hcart 6 UP *NULL* Drive044 /dev/rmt8.1
drive - 51 hcart 7 UP *NULL* Drive045 /dev/rmt9.1
drive - 52 hcart 8 UP *NULL* Drive046 /dev/rmt10.1
 

 

./vmoprcmd -d

PENDING REQUESTS

<NONE>

DRIVE STATUS

Drv Type Control User Label RecMID ExtMID Ready Wr.Enbl. ReqId
0 hcart AVR - No - 0
1 hcart AVR - No - 0
2 hcart AVR - No - 0
31 hcart AVR - No - 0
32 hcart AVR - No - 0
33 hcart AVR - No - 0
34 hcart AVR - No - 0
49 hcart AVR - No - 0
50 hcart AVR - No - 0
51 hcart AVR - No - 0
52 hcart AVR - No - 0

ADDITIONAL DRIVE STATUS

Drv DriveName Shared Assigned Comment
0 Drive040 Yes - *NULL*
1 Drive041 Yes - *NULL*
2 Drive051 Yes - *NULL*
31 Drive036 Yes - *NULL*
32 Drive037 Yes - *NULL*
33 Drive047 Yes - *NULL*
34 Drive048 Yes - *NULL*
49 Drive043 Yes - *NULL*
50 Drive044 Yes - *NULL*
51 Drive045 Yes - *NULL*
52 Drive046 Yes - *NULL*  

 

./bpps -x
NB Processes
------------
root 4260018 1 0 11:53:05 - 0:00 /usr/openv/netbackup/bin/nbsvcmon
root 7340204 1 0 11:53:03 - 0:00 /usr/openv/netbackup/bin/nbrmms
root 3735930 1 0 11:52:37 - 0:00 /usr/openv/netbackup/bin/bpcompatd
root 3867084 1 0 11:53:05 - 0:00 /usr/openv/netbackup/bin/nbsl

MM Processes
------------
root 2424886 1 0 11:52:36 - 0:00 vmd
root 4456588 1 0 11:52:36 - 0:00 /usr/openv/volmgr/bin/ltid
root 10027244 4456588 0 11:53:02 - 0:00 avrd

Shared Symantec Processes
-------------------------
root 9240778 1 0 11:51:45 - 0:00 /opt/VRTSpbx/bin/pbx_exchange

and Output from Master server are

ClusterName = ""
        MachineName = "<media server>"
        FQName = "<media server>"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x15
        MachineNbuType = media (1)
        MachineState = active for tape and disk jobs (14)
        MasterServerName = "<master server>"
        NetBackupVersion = 6.5.5.0 (655000)
        OperatingSystem = rs6000 (5)
        ScanAbility = 5
 

 

Ah45's picture

ltid log from Media server

here something i find..

12:10:33.201 [7405814] <16> emmlib_DrivePathQuery: (0) GetDrivePaths failed, emmError = 2000000, nbError = 0

12:10:33.201 [7405814] <4> ProcessAllDrivePaths: Failed to retrieve all paths for drive Drive040 from emm. Error = 13

12:10:33.293 [7405814] <16> emmlib_DrivePathQuery: (0) GetDrivePaths failed, emmError = 2000000, nbError = 0

12:10:33.293 [7405814] <4> ProcessAllDrivePaths: Failed to retrieve all paths for drive Drive041 from emm. Error = 13

12:10:33.377 [7405814] <16> emmlib_DrivePathQuery: (0) GetDrivePaths failed, emmError = 2000000, nbError = 0

12:10:33.377 [7405814] <4> ProcessAllDrivePaths: Failed to retrieve all paths for drive Drive051 from emm. Error = 13

12:10:33.470 [7405814] <16> emmlib_DrivePathQuery: (0) GetDrivePaths failed, emmError = 2000000, nbError = 0

12:10:33.470 [7405814] <4> ProcessAllDrivePaths: Failed to retrieve all paths for drive Drive036 from emm. Error = 13

12:10:33.735 [7405814] <16> emmlib_DrivePathQuery: (0) GetDrivePaths failed, emmError = 2000000, nbError = 0

12:10:33.735 [7405814] <4> ProcessAllDrivePaths: Failed to retrieve all paths for drive Drive037 from emm. Error = 13

12:10:33.822 [7405814] <16> emmlib_DrivePathQuery: (0) GetDrivePaths failed, emmError = 2000000, nbError = 0

12:10:33.822 [7405814] <4> ProcessAllDrivePaths: Failed to retrieve all paths for drive Drive047 from emm. Error = 13

12:10:33.905 [7405814] <16> emmlib_DrivePathQuery: (0) GetDrivePaths failed, emmError = 2000000, nbError = 0

12:10:33.905 [7405814] <4> ProcessAllDrivePaths: Failed to retrieve all paths for drive Drive048 from emm. Error = 13

12:10:33.987 [7405814] <16> emmlib_DrivePathQuery: (0) GetDrivePaths failed, emmError = 2000000, nbError = 0

12:10:33.987 [7405814] <4> ProcessAllDrivePaths: Failed to retrieve all paths for drive Drive043 from emm. Error = 13

12:10:34.095 [7405814] <16> emmlib_DrivePathQuery: (0) GetDrivePaths failed, emmError = 2000000, nbError = 0

12:10:34.095 [7405814] <4> ProcessAllDrivePaths: Failed to retrieve all paths for drive Drive044 from emm. Error = 13

12:10:34.241 [7405814] <16> emmlib_DrivePathQuery: (0) GetDrivePaths failed, emmError = 2000000, nbError = 0

12:10:34.241 [7405814] <4> ProcessAllDrivePaths: Failed to retrieve all paths for drive Drive045 from emm. Error = 13

12:10:34.362 [7405814] <16> emmlib_DrivePathQuery: (0) GetDrivePaths failed, emmError = 2000000, nbError = 0

12:10:34.362 [7405814] <4> ProcessAllDrivePaths: Failed to retrieve all paths for drive Drive046 from emm. Error = 13

Marianne's picture

Certainly seems like comms issue form media server side with EMM.

Did you run 'nbemmcmd -getemmserver' on master server? Is output from this media server correct?

If above command gives correct output, all I can suggest is to run Device Config wizard again for this media server. Cmd equivalent of wizard:

vmoprcmd -h <media server name> -timeout 3600 -autoconfig  -a

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

RamNagalla's picture

 

12:10:34.362 [7405814] <16> emmlib_DrivePathQuery: (0) GetDrivePaths failed, emmError = 2000000, nbError = 0

12:10:34.362 [7405814] <4> ProcessAllDrivePaths: Failed to retrieve all paths for drive Drive046 from emm. Error = 13

 

What is tpautoconf -t showing in Medid server?