Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Catalog backup failing with 87 using DD

Created: 23 Jan 2013 | 13 comments
PhaniKumar82's picture

Team

Need solution on the below issue 

Netbackup CATALOG_DRIVEN_BACKUP is failing with status 87 using datadomain but same time normal backups using same DD storage unit is completing successfully 

Master server : Netbackup 7.1.0.4 on Windows 2003 

Media server : Master server act as media server itself as this is remote master server 

 

Please let me know how to proceed with the above

 

Regards,

Kumar 

 

 

Comments 13 CommentsJump to latest comment

mph999's picture

The writing part of the catalog backup is not really any different than a normal backup, the 'clever' bit of the catalog backup is performed before the write is made.

There is one difference however which you probably don't have sent on you regular backups, and that is the TIR option is set.

I would suggest enabling the TIR option on a regulat policy (set on the attributes screen) and run a test - this will narrow the issue down, if the regular backup than fails.

Additionally, we would need the logs from the catalog backup - bptm, bpdm, bpbrm, bpbkar and bpcd 

M

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
PhaniKumar82's picture

Martin

i have attached bpdm bpbrm bpcd and bpbkar logs

Please let me know if you need any other information

Regards,
Kumar

AttachmentSize
logs.zip 28.51 KB
mph999's picture

Missing the bptm log (and bpdm was empty, though there might not be anything logged in here).

Can you create the bptm dir, set the verbose level to 5 and general to 2, and repost the logs.

One really important question - did this ever work ?

Is this a big catalog - how long do you think the catalog backup should run for - are we taking a 1 or so or many hours ?  Just need an estimate.

Thanks,

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mph999's picture

 

Can you also look at the logs on the datadomain - I suspect it might log something (apologies, I am not too famliar with DD so cannot really advise on which logs).

For info, this is how to set logs/ change verbose levels - see part 2.
 
 
Setting up logs in NetBackup
 
 
For a given issue, it may be necessary to gather multiple logs.  This MUST cover the time the issue happens.
If an additional log is required, that has to be created, then ALL the logs must be supplied again.
 
There are two types of logs in NetBackup.  Legacy logs and VX logs.
 
 
1) Creating Legacy Logs
2) Setting Verbose Level for Legacy Logs
3) Collecting Legacy Logs
4) Creating VX Logs and setting the log level
5) Collecting VX logs
6) Volmgr Logs
 
1) Creating Legacy Logs
************************
 
These are created in either 
 
Unix
/usr/openv/netbackup/logs/
/usr/openv/volmgr/debug/
 
Windows
<install path>\veritas\netbackup\logs
<install path>\veritas\volmgr\debug
 
 
For example to create bptm log, simply create a directory called the <process> name.
 
mkdir /usr/openv/netbackup/logs/bptm
 
A newly created log will not log anything /detect a change of verbose level until the process is restarted.  For logs such as bptm, this will be when the next backup runs.  Other logs such as bprm and bpdm may require a restart of the NBU services.  I say 'may', if the process starts a child process, then this would write to a newly created log or pick up a verbose level change.
 
 
2) Setting Verbose Level for Legacy Logs
****************************************
 
There are two ways this can be done :
 
(Unix) 
 
To increase the verbose level of all logs (except vault)
Add the entry VERBOSE = <level> into /usr/openv/netbackup/bp.conf.  <level> is a value between 0 and 5, with 5 being the highest.
 
 
(Windows)
 
On the server you are gathering the logs from, run the BAR GUI
From the File menu, select Client Properties and in the pop-up window, goto the Troubleshooting tab.
Set General to 2 and Verbose to 5
 
 
3) Collecting Legacy Logs
*************************
 
The log file is simply found in the <process> name directory.  There is one log per day.
 
The name of the log file will be log.<date>
 
If you are sending multiple log files in, they will all have the same name.  Please therefore rename the log files to :
 
<process>.log.<date>
 
 
4) Creating VX Logs and setting the log level
**********************************************
 
These are more complex, and have to be set with specific commands.  NOTE: Some of these logs, for example, 'mds' do NOT create a log file.  Instead the lines are entered in to other log files.  In the case of mds (143), it logs into EMM (111).
 
 
The vxlogs cover various processes, for example, nbemm, nbrb, nbjm, nbrb, mds
 
 
To set these up on either Unix or Windows, use this command :
 
vxlogcfg -a -p 51216 -o <oid> -s DebugLevel=<1-6> -s DiagnosticLevel=<1-6>
 
For example, to set the EMM and MDS logs to levels 6 and 6 use
 
vxlogcfg -a -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6
vxlogcfg -a -p 51216 -o 143 -s DebugLevel=6 -s DiagnosticLevel=6
 
 
To confirm the log level has been set, simply look in the nblog.conf file, which is located in the netbackup diorectory.
(NOTE: DO NOT EDIT THIS FILE MANUALLY)
 
 
5) Collecting VX Logs
*********************
 
 
 
To collect the vx logs, use the nbcplogs command.  This copies the raw logs, which is the preference of Technical Support.
 
NOTE:  The destination directory MUST be empty.
 
 
nbcplogs --no-nbsu -d 2hrs --logs nbemm,nbjm /tmp/logs   (Ex.  Coleect the past 2 hrs of logs, RELATIVE, to when the command is run )
  
nbcplogs --no-nbsu -s 07/11/2012-10:17:58 -e 07/11/2012-12:17:58 --logs nbjm,nbpem /tmp/logs (Collect the logs between two times -s <start> -e <end>  )
 
In these examples, the nbpem and nbjm logs would be copied to /tmp/logs
 
 
For details of using vxlogview please see TN:  http://www.symantec.com/docs/TECH75805
 
 
6) Volmgr Debug Logs
********************
 
These are very similar to legacy logs, the difference being the location and the verbose setting.
There is no value for verbose level, simply it is turned up by adding the work VERBOSE to a line in the vm.conf file.
 
To turn on Media Manager logging:
 
UNIX:
 
Add VERBOSE to the /usr/openv/volmgr/vm.conf file.  If this file does not exist, just create it.
If necessary, create the directory /usr/openv/volmgr/debug
 
mkdir /usr/openv/volmgr/debug/acssi
mkdir /usr/openv/volmgr/debug/acsd
mkdir /usr/openv/volmgr/debug/robots
mkdir /usr/openv/volmgr/debug/daemon
mkdir /usr/openv/volmgr/debug/ltid
mkdir /usr/openv/volmgr/debug/oprd
mkdir /usr/openv/volmgr/debug/reqlib
mkdir /usr/openv/volmgr/debug/tpcommand
 
The following empty files increase the details logged further
 
touch /usr/openv/volmgr/DRIVE_DEBUG
touch /usr/openv/volmgr/ROBOT_DEBUG 
touch /usr/openv/volmgr/AVRD_DEBUG 
touch /usr/openv/volmgr/SSO_DEBUG 
 
 
Restart ltid :
 
/usr/openv/volmgr/bin/stopltid
/usr/openv/volmgr/bin/ltid -v
 
 
Windows:
 
Add VERBOSE to the <install path>\veritas\volmgr\vm.conf file.  If this file does not exist, just create it.
If necessary, create the directory /usr/openv/volmgr/debug
 
<install path>\veritas\volmgr\debug/acssi
<install path>\veritas\volmgr\debug/acsd
<install path>\veritas\volmgr\debug/robots
<install path>\veritas\volmgr\debug/daemon
<install path>\veritas\volmgr\debug/ltid
<install path>\veritas\volmgr\debug/oprd
<install path>\veritas\volmgr\reqlib
<install path>\veritas\volmgr\debug\tpcommand
 
Create the following empty files to increase the details logged further
Ensure that windows does not craete a 'suffix', for example .txt
 
<install path>\veritas\volmgr\DRIVE_DEBUG
<install path>\veritas\volmgr\ROBOT_DEBUG 
<install path>\veritas\volmgr\AVRD_DEBUG 
<install path>\veritas\volmgr\SSO_DEBUG 
 
 
Restart ltid :
 
<install path>\veritas\volmgr\bin/stopltid
<install path>\veritas\volmgr\bin/ltid -v

 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
PhaniKumar82's picture

Martin

We dont have any tape backups in our environment

do you still need bptm logs ?

Regards,
Kumar

Marianne's picture

As from NBU 6.5, ALL backups log to bptm - disk and tape.
bpdm only logs disk cleanup info.

So, yes - we need bptm log.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

mph999's picture

Hmm, seems that there can be many causes of 87 on disk - I've been looking through a few previous cases.

Network issues seem quite common

Time out issues (but this was on a netapp)

Disk failure

However, the one over riding factor, which is making this a bit confusing is you only report the issue on catalog backups, so it would appear something that is different about this type of backup.

The only thing I can think of (as mentioned) that is probably different is that TIR is always enabled for a catalog backup (hence my request to test a 'normal' backup with this enabled.

The other possibe difference of a catalog backup, is that it is backing up (part of the master) (other backups are probably of different clients) and the length of the catlog backup could be very long compared to others.

These are just a few ideas, and odd as it seems, sometimes these sort of things are a cause or contributing factor.

Is there anything on the datadomain that is unique for the catalog backup - apologies, not familar with these, but my thoughts are if the catalog backups use a different 'location' of the DD (if this is possible ?).

M

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mph999's picture

Yep, bptm also writes to disk ...

TBH, I'm not convinced that we going to get much more info than we have from the details you have posted, but we should look.

Most imporantly are the questions I asked.

Has this worked in the past

If it did work previously, what has been changed

What happens if you enable TIR on a regular policy

How big is the catalog backup 

Anything special on the DD

Thanks,

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
Nicolai's picture

Can you post the detailed job text from the activitry monitor ?

Have you tried to delete the catalog backup policy and re-create it ?

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

mph999's picture

Right I've got my brain in gear tonight re datadomains ...  The DD has luns mapped to diskpools in NBU, are you using a separate diskpool for the catalog backup ?

Thanks,

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
William Jansen van Nieuwenhuizen's picture

Hi

As far as I know TIR is not yet supported on NBU and DD(need 2.6 ost which is GA and 5.3 DDOS that isn't GA yet). Sonif you have it configured, turn it off and try again.

Like martin said, check the STU you selected and possibly create a new catalog backup policy and see if that works.

mph999's picture

You can't turm TIR off on Catalog backups, it's always on.

M

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
William Jansen van Nieuwenhuizen's picture

Sorry my bad, I interprited TIR as True Image Replication. The stuff that goes with AIR. Guess I have some reading to do in the manuals :-)