Video Screencast Help

Master @7.5.0.3 Media Servers @ 6.0MP4

Created: 03 Jul 2012 • Updated: 07 Jul 2012 | 13 comments
vamsivaddi's picture
This issue has been solved. See solution.

 

#./ltid

emmlib_GetDeviceList failed, status=334

# ./tpautoconf

The device_mappings file has invalid license info

 

Need Suggestions immediately

 

Master Server Netbmast

OS: Solaris 10

Net Backup version 7.5.0.3

 

Media servers sf6900a, sf6900b, sf6900c-bkp

OS Solaris 10

Net Backup Version 6.0 MP4

 

Recently Upgraded NBU to 7.5.0.3 everything worked fine for a week including all backups and restores.

Since yesterday Media Server sf6900c-bkp  can’t be able to backup or restore.

Seems this passing to other media servers as well.

 

Suspecting Communication between EMM server and Media Server.

And Licensing issues based on below two errors.

 

#./ltid

emmlib_GetDeviceList failed, status=334

 

# ./tpautoconf

The device_mappings file has invalid license info

 

Please find below the logs.

 

emmlib_GetDeviceList failed, status=334

root@sf6900c # ./tpautoconf

The device_mappings file has invalid license info

 

root@sf6900c # ./scan

************************************************************

*********************** SDT_TAPE    ************************

*********************** SDT_CHANGER ************************

*********************** SDT_OPTICAL ************************

************************************************************

Unable to intialize the device mappings table, status = 2

------------------------------------------------------------

Device Name  : "/dev/sg/c1tw500104f0009accbdl0"

Passthru Name: "/dev/sg/c1tw500104f0009accbdl0"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "STK     SL500           1232"

Vendor ID  : "STK     "

Product ID : "SL500           "

Product Rev: "1232"

Serial Number: "559000200248"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_CHANGER

NetBackup Robot Type: Not Found(6)

Removable      : Yes

Device Supports: SCSI-5

Number of Drives : 2

Number of Slots  : 100

Number of Media Access Ports: 15

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name  : "/dev/rmt/2cbn"

Passthru Name: "/dev/sg/c1tw500104f0009accc7l0"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "HP      Ultrium 3-SCSI  M21S"

Vendor ID  : "HP      "

Product ID : "Ultrium 3-SCSI  "

Product Rev: "M21S"

Serial Number: "HU10726FGY"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_TAPE

NetBackup Drive Type: Not Found(6)

Removable      : Yes

Device Supports: SCSI-3

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name  : "/dev/rmt/3cbn"

Passthru Name: "/dev/sg/c1tw500104f0009acccal0"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry    : "HP      Ultrium 3-SCSI  M21S"

Vendor ID  : "HP      "

Product ID : "Ultrium 3-SCSI  "

Product Rev: "M21S"

Serial Number: "HU10726FF1"

WWN          : ""

WWN Id Type  : 0

Device Identifier: ""

Device Type    : SDT_TAPE

NetBackup Drive Type: Not Found(6)

Removable      : Yes

Device Supports: SCSI-3

Flags : 0x0

Reason: 0x0

 

 

 

./sgscan

#

#WARNING: detected StorEdge Network Foundation connected devices not in

#         SG configuration file:

#

#    Device World Wide Port Name 50060e80104d1762

#    Device World Wide Port Name 50060e80104d1761

 

#    Device World Wide Port Name 50060e80104d1763

#    Device World Wide Port Name 50060e80104d1760

#

#    See /usr/openv/volmgr/MediaMgr_DeviceConfig_Guide.txt chapter

#    "Special configuration for "Sun StorEdge Network Foundation" HBA/Driver"

#    for information on how to use sg.build and sg.install to

#    configure these devices

#

/dev/sg/c0t8l0: Disk (/dev/rdsk/c6t8d0): "SEAGATE ST373455LSUN72G"

/dev/sg/c0t9l0: Disk (/dev/rdsk/c6t9d0): "SEAGATE ST373455LSUN72G"

/dev/sg/c1tw500104f0009accbdl0: Changer: "STK     SL500"

/dev/sg/c1tw500104f0009accc7l0: Tape (/dev/rmt/2): "HP      Ultrium 3-SCSI"

/dev/sg/c1tw500104f0009acccal0: Tape (/dev/rmt/3): "HP      Ultrium 3-SCSI"

/dev/sg/c2t10l0: Disk (/dev/rdsk/c0t10d0): "SEAGATE ST373455LSUN72G"

/dev/sg/c2t11l0: Disk (/dev/rdsk/c0t11d0): "SEAGATE ST373455LSUN72G"

/dev/sg/c2t5l0: Processor: "SUN     StorEdge 3120  D"   

Comments 13 CommentsJump to latest comment

mph999's picture

From one of the affected media servers what does this show

nbemmcmd -listhosts

If this fails, look in pbx logs, the media server contacts pbx on the master when you run this command.

vxlogview -p 50936 -o 103 -d all -t 00:20:00

This will give the past 20 mins of the pbx log

Personally I would upgrade the media servers 6.0 to 7.5 is a very big gap ...

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
vamsivaddi's picture

I have upgraded the media servers all the backup's are workoing fine.

 

But i cant rstore from mediaservers it is taking long time for restoration and it is not failing.

 

When i restore from one of the media servers it is detecting the tape but but notthing is moving ahead.

restore job is showing in running status for long time.

Thanks & Regards,

Vamsi Krishna

mph999's picture

What do you see in the details tab of one of the restore jobs in activity monitor.

This will show where abouts the job is getting delayed.

Has anything else changed, an upgrade should not cause this issue ?

Once we know where it is getting stuck, we can advise further, we may need to lok in the logs.

 

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
vamsivaddi's picture

When you start the restore job it is saying this media is required etc..

but is not mounting the media into drive to start the restore process

 

restore is not failing it is taking long time in running state.

It  is the same case with all the media servers.

and the master itself is the media but it is working fine there.

Thanks & Regards,

Vamsi Krishna

mansoor.sheik's picture

Hi

Please tell us that restoration is failing or taking long time ?

If it taking long time ensure the data flow is through the media server or not.

In job details

Waiting for mount of media id XXXXXX on server servername1 for reading. Here servername1 is the server through the data flow is going to process.Here you have to get your mediaserver name.

---------------------

When i restore from one of the media servers it is detecting the tape but but notthing is moving ahead.

restore job is showing in running status for long time.

-----------------------

If the tape selection is done and it is not moving ahead. You can go ahead with nbrbutil -dump comamnd and release the media MDS key of the tape ( nbrbutil -releaseMDS xxxx) and restart the job again.

 

vamsivaddi's picture

restore is not failing it is taking long time to mount media.

It  is the same case with all the media servers.

and the master itself is the media but it is working fine there.

Thanks & Regards,

Vamsi Krishna

Marianne's picture

You need bprd log on the master to troubleshoot this.

If Activity Monitor is only listing 'media needed', it means that the job is still 'stuck' in bprd process on the master. Restore request has not yet been handed over to media server.

Please also post output of the following on the master when you see these "hung' restore jobs:

bpps |grep bprd

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

vamsivaddi's picture

 

    #bpps -x |grep bprd
   
    root   953     1   0   Jul 04 ?           3:41 /usr/openv/netbackup/bin/bprd
    root  1396   953   0 00:15:05 ?           0:30 /usr/openv/netbackup/bin/bprd
    root 27728   953   0 22:49:14 ?           0:32 /usr/openv/netbackup/bin/bprd
    root 22350     1   0 21:03:50 ?           0:44 bprd -dontfork -mpxmain

Thanks & Regards,

Vamsi Krishna

vamsivaddi's picture

Hey Marianne u seem to be correct!

 

None of the restores are not working.

What is the work around to solve this issue.

 

Even restarted the Netbackup stop and start on master server

Thanks & Regards,

Vamsi Krishna

Marianne's picture

My experience has been that this one is the culprit:
bprd -dontfork -mpxmain

My experience has also been that stopping NBU does not terminate this hung bprd process. So, when NBU is down, do bpps again. Any left-over, hung bprd processes must be killed with kill -9 before starting NBU again.

PLEASE check that bprd log folder exists before starting up NBU again.

In most instances, a new restore attempt should be successful.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

SOLUTION
mph999's picture

Correct - in the bprd log you would see it complaining about mpxmain already existing, if i remember correctly .... Not seen one for a while ...

M

 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mph999's picture

As Marianne suggested, we need the bprd log from the master, and I would add in bpjobd log also.

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
vamsivaddi's picture

Guys got the Solution.

bprd went to hung state.

Killed it manually restores working now. 

Thanks & Regards,

Vamsi Krishna