Video Screencast Help
Search Video Help Close Back
to help
New in the Rewards Catalog: Vouchers for "Symantec Technical Specialist" and "Symantec Certified Specialist" exams.

Catalog backup on 5.1 MP2 fails with status 124

Updated: 23 May 2010 | 10 comments
afacey's picture
0 0 Votes
Login to vote

In trying to get to 6.5.x on new hw, I have migrated production 5.1MP2 (solaris 8 SPARC) to new hw (solaris 10) as a stepping stone. All backups and restores run as expected. The catalog will not backup sucessfully though.

The paths I have listed are consistant with the guidelines and worked without issue on the old hardware. The only thing that has changed is our catalog filesystem is now NFS mounted.

polaris-T2000# pwd
/opt/openv/netbackup/logs/admin
13:27:54.360 [15340] <2> job_disconnect: Disconnected
13:27:54.374 [15340] <16> bpbackupdb: NB database backup to media id A00054 FAILED
13:27:54.374 [15340] <2> bpbackupdb: executing - /usr/openv/netbackup/bin/dbbackup_notify TAPE A00054 FAIL
13:27:54.441 [15340] <2> get_behavior_rec: ?
13:27:54.463 [15340] <2> LOCAL CLASS_ATT_DEFS: Product ID = 6
13:27:54.466 [15340] <2> get_behavior_rec: MD5 digest line

I have not had a sucessful backup since we migrated to this new HW. The catalog backup runs for about 3 hours and then fails. The size is ~200GB to LTO3 tape
How do I find the path that is failing?

discussion Filed Under:

Comments

schmaustech's picture
13
Jul
2009
0 Votes 0
Login to vote

What about...

Couple questions:

-On the old master, were their media servers as well?  Can the new system access the media severs?

-Do you see any nfs errors in /var/log/messages?

Regards,

Benjamin Schmaus

John Stockard's picture
13
Jul
2009
1 Vote +1
Login to vote

Logging

Try enabling the bpbkar log on the master server.  Since a catalog backup in NBU 5.1 is always a cold backup, it uses the same NBU processes as a normal filesystem backup (bpbkar, bptm, bpbrm).  I have a gut feeling that bpbkar is throwing errors because it doesn't like trying to backup an NFS mount point in this context.

afacey's picture
14
Jul
2009
0 Votes 0
Login to vote

Master and media are same

Master and media are same machine (both new and old HW). Not seeing any NFS errors in /var/adm/messages. Strangely though the backup runs for nearly 3 hours and then fails.
I'm going to run another attempt when my fulls complete with golobal logging set to 5 and see what turns up.

Alex Vasquez's picture
14
Jul
2009
0 Votes 0
Login to vote

Update

I'm not sure how much time I would spend troubleshooting something that isn't patched to where it needs to be.  You're five maintenance packs behind on 5.1.x.  My advice to you, would be to update to the latest maintenance pack, which is MP7.  Then see what you get: http://seer.entsupport.symantec.com/docs/303010.htm

That may not resolve the issue, but at least if you take the case to support you'll have already done the first thing that they would advise. ;)  John's suggestion is also solid and will help you further diagnose the issue.

"I’m an early bird and a night owl. So I’m wise and I have worms."
- Michael Scott

afacey's picture
14
Jul
2009
0 Votes 0
Login to vote

True - but two things: 1. I

True - but two things:
1. I migrated from old HW to new; it worked fine on old - no config changes on new HW
2. Before upgrading to 5.1MP7, wouldn't it be wise to have a successful catalog backup?

John Stockard's picture
14
Jul
2009
0 Votes 0
Login to vote

Config changes

Actually, you did make a couple of config changes to your environment when you moved your NBU master/media server from an old server to a new server:

1. The OS changed from Solaris 8 to Solaris 10.
2. The NetBackup catalogs are now hosted on a remote machine via an NFS export.

It's possible that the problem lies more with NBU interacting with the NFS export than with the change from Solaris 8 to Solaris 10 (since Solaris 10 was supported starting with NBU 5.1 MP2).

Alex Vasquez's picture
14
Jul
2009
0 Votes 0
Login to vote

True that plus 1

3. It is wise to have a good catalog backup before upgrading, do you have enough local disk to redirect your catalog backup, rather than using tape?

"I’m an early bird and a night owl. So I’m wise and I have worms."
- Michael Scott

John Stockard's picture
14
Jul
2009
2 Votes +2
Login to vote

Upgrade to MP7

If you can't get the catalog backup working at NBU 5.1 MP2, you could manually do an equivalent catalog backup, like this:

1. Wait for a time period when there is no activity in your NetBackup environment (no backups, duplications, or restores).

2. Stop all of the NetBackup daemons on your master/media server by running the script "/opt/openv/netbackup/bin/goodies/bp.kill_all".

3. Manually use the Solaris "tar" command (or an equivalent utility) to backup the following directories to a separate tape or a flat tarball file:

/opt/openv/netbackup/db
/opt/openv/var
/opt/openv/volmgr/database

These are the same three directories that get backed up by NetBackup during a catalog backup.  The first path contains your images and media databases, the second path contains your license keys (and a few other things), and the third path contains your volume database and the device configuration.  If you choose to put this manual backup on a tape, pick a tape that you won't be using with NetBackup.  Once a utility such as "tar" writes to the tape, it will destroy the NetBackup header on the tape and NetBackup will refuse to use it (unless you checked the "Overwrite tar" option in the media server properties, and then you'll run the risk of destroying your manual catalog backup with NetBackup backups).

At this point you should be in a good position to upgrade to NBU 5.1 MP7.

You should also be aware that NBU 5.1 reached it's "end-of-support" date back on March 31, 2008.  Symantec is not producing any more patches or maintenance packs for NBU 5.1, and you may get the cold shoulder from Symantec tech support.  I would bet that Symantec tech support will tell you to upgrade to NBU 6.0 or NBU 6.5.x.  You may have to push back at them to get them to help you troubleshoot an issue on NBU 5.1.

afacey's picture
14
Jul
2009
0 Votes 0
Login to vote

NBU catalog backup

thanks for your input. The intended goal is to get to 6.5.4, however I must ensure the current system is working properly before attempting an upgrade. Since that is the goal, support is helping to some extent. If it is indeed an issue with NFS, I will most likely proceed as you suggested in making a copy on disk.

I'm running the catalog backup again now with logging set to 5.

Thanks to all that have replied thus far!

afacey's picture
16
Jul
2009
1 Vote +1
Login to vote

snapshots from filer caused the issue

Update on issue:

Ended up being an issue with the .snapshots being visable in the /usr/openv/netbackup/db/images dir. NBU was failing trying to reset acces times or related. The fix is to turn off snapshots visabliity at the filer level. Catalogs completed without issue.

Thanks to all.