Video Screencast Help

Catalog backup aborted with status 42 on master server

Created: 27 Feb 2013 • Updated: 06 Mar 2013 | 11 comments
This issue has been solved. See solution.

Hi All,

Our catalog backup is getting aborted with status 42. It was working fine till yesterday.

Only catalog backup is failing & other client backups are running fine.I opened a case with symantec and waiting for their response too.

please find the below logs

 

02/27/2013 09:47:49 - Info nbjm (pid=22317) starting backup job (jobid=78370) for client bnymwdnb1, policy bnymwdnb1_CBT, schedule Daily
02/27/2013 09:47:49 - Info nbjm (pid=22317) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=78370, request id:{A81C8348-80EC-11E2-9E2D-0021280A25EE})
02/27/2013 09:47:49 - requesting resource bnymwdnb1-hcart-robot-tld-0
02/27/2013 09:47:49 - requesting resource bnymwdnb1.NBU_CLIENT.MAXJOBS.bnymwdnb1
02/27/2013 09:47:49 - requesting resource bnymwdnb1.NBU_POLICY.MAXJOBS.bnymwdnb1_CBT
02/27/2013 09:47:50 - granted resource bnymwdnb1.NBU_CLIENT.MAXJOBS.bnymwdnb1
02/27/2013 09:47:50 - granted resource bnymwdnb1.NBU_POLICY.MAXJOBS.bnymwdnb1_CBT
02/27/2013 09:47:50 - granted resource 300366
02/27/2013 09:47:50 - granted resource IBM.ULTRIUM-TD4.002
02/27/2013 09:47:50 - granted resource bnymwdnb1-hcart-robot-tld-0
02/27/2013 09:47:50 - estimated 0 kbytes needed
02/27/2013 09:47:50 - Info nbjm (pid=22317) started backup (backupid=bnymwdnb1_1361976470) job for client bnymwdnb1, policy bnymwdnb1_CBT, schedule Daily on storage unit bnymwdnb1-hcart-robot-tld-0
02/27/2013 09:47:51 - Info bpbrm (pid=22949) bnymwdnb1 is the host to backup data from
02/27/2013 09:47:51 - Info bpbrm (pid=22949) reading file list from client
02/27/2013 09:47:51 - Info bpbrm (pid=22949) starting bpbkar on client
02/27/2013 09:47:51 - started process bpbrm (pid=22949)
02/27/2013 09:47:51 - connecting
02/27/2013 09:47:51 - connected; connect time: 0:00:00
02/27/2013 09:47:52 - Info bpbkar (pid=22957) Backup started
02/27/2013 09:47:52 - Info bpbrm (pid=22949) bptm pid: 22958
02/27/2013 09:47:53 - Info bptm (pid=22958) start
02/27/2013 09:47:53 - Info bptm (pid=22958) using 65536 data buffer size
02/27/2013 09:47:53 - Info bptm (pid=22958) using 30 data buffers
02/27/2013 09:47:53 - Info bptm (pid=22958) start backup
02/27/2013 09:47:53 - Info bptm (pid=22958) media id 300366 mounted on drive index 2, drivepath /dev/rmt/3cbn, drivename IBM.ULTRIUM-TD4.002, copy 1
02/27/2013 09:47:53 - mounted 300366
02/27/2013 09:47:53 - positioning 300366 to file 9
02/27/2013 09:47:56 - positioned 300366; position time: 0:00:03
02/27/2013 09:47:56 - begin writing
02/27/2013 10:02:15 - Warning bpbrm (pid=22949) from client bnymwdnb1: WRN - /usr/openv/netbackup/db/error/daily_messages.log is only being backed up as a symbolic link
02/27/2013 10:02:16 - Info bpbkar (pid=22957) bpbkar waited 1402 times for empty buffer, delayed 6420 times
02/27/2013 10:02:16 - Info bptm (pid=22958) waited for full buffer 30333 times, delayed 35555 times
02/27/2013 10:02:23 - Error bpbrm (pid=22949) db_flist_complete failed: events out of sequence - image inconsistency (229) bnymwdnb1 bnymwdnb1_CBT 4 1361976470
02/27/2013 10:10:37 - Error bptm (pid=22958) get_string() failed, I/O error (5), premature end of file encountered
02/27/2013 10:10:37 - Info bptm (pid=22958) EXITING with status 42 <----------
02/27/2013 10:10:42 - Info bpbkar (pid=22957) done. status: 42: network read failed
02/27/2013 10:10:42 - end writing; write time: 0:22:46
network read failed (42)

 

 

Regards,

Musthaq

 

 

Operating Systems:

Comments 11 CommentsJump to latest comment

Marianne's picture

Best to log a Support call.

Seems master server is also the media server? If so, ensure all of the following log folders exist on the master:

bpdbm  (restart NBU to enable this log)
bpbrm
bptm
bpbkar

In the meantime, increase Client Read Timeout on master server to something like 1800. We have seen that increased timeout solved similar issues in the past.

You may also want to have a look at this TN:

 

Key performance considerations for NetBackup 7.5 master servers 

http://www.symantec.com/docs/TECH202840

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

mohamedmusthaq's picture

Hi Marianne,

 

Thanks for the reply.

 

I tried after changing the client read timeout but still getting the same error :(

Marianne's picture

Then please follow my initial advice:

Best to log a Support call.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Nagalla's picture

02/27/2013 10:02:23 - Error bpbrm (pid=22949) db_flist_complete failed: events out of sequence - image inconsistency (229) bnymwdnb1 bnymwdnb1_CBT 4 1361976470

try to fix this inconsistency then try the catalog backup again.

Run /usr/openv/netbackup/bin/bpdbm -consistency 1 > /tmp/bpdbm_consistency.txt

The command will verify all backup images and report those that have problem. After fixing / removing the bad images, try cagtalog backup.

mohamedmusthaq's picture

Hi NAgalla,

Thanks for the reply. please find the attached below logs. Is there any command to make these image as consistent

 

 

 

Nagalla's picture

I dont think so, you would need to delete the associated images files ,, becuase its saying failed backup.

 

cking image file <bnymwdnb1_CBT_1362036644_INCR>
>image is invalid
>>failed backup
>>COPIES not zero
checking files file <bnymwdnb1_CBT_1362036644_INCR.f>
</usr/openv/netbackup/db/images/bnymwdnb1/1362000000/bnymwdnb1_CBT_1362036644_INCR.f> does not exist
checking image file <bnymwdnb1_CBT_1362038306_INCR>
>image is invalid
>>failed backup
>>COPIES not zero
checking files file <bnymwdnb1_CBT_1362038306_INCR.f>
</usr/openv/netbackup/db/images/bnymwdnb1/1362000000/bnymwdnb1_CBT_1362038306_INCR.f> does not exist
checking image file <bnymwdnb1_CBT_1362040004_INCR>
>image is invalid
>>failed backup
>>COPIES not zero
checking files file <bnymwdnb1_CBT_1362040004_INCR.f>
</usr/openv/netbackup/db/images/bnymwdnb1/1362000000/bnymwdnb1_CBT_1362040004_INCR.f> does not exist
checking image file <bnymwdnb1_CBT_1362041698_INCR>
>image is invalid
>>failed backup
>>COPIES not zero
checking files file <bnymwdnb1_CBT_1362041698_INCR.f>
</usr/openv/netbackup/db/images/bnymwdnb1/1362000000/bnymwdnb1_CBT_1362041698_INCR.f> does not exist
checking image file <bnymwdnb1_CBT_1362043393_INCR>
>image is invalid
>>failed backup
>>COPIES not zero
checking files file <bnymwdnb1_CBT_1362043393_INCR.f>
</usr/openv/netbackup/db/images/bnymwdnb1/1362000000/bnymwdnb1_CBT_1362043393_INCR.f> does not exist
checking image file <bnymwdnb1_CBT_1362045187_INCR>
>image is invalid
>>failed backup
>>COPIES not zero
checking files file <bnymwdnb1_CBT_1362045187_INCR.f>
</usr/openv/netbackup/db/images/bnymwdnb1/1362000000/bnymwdnb1_CBT_1362045187_INCR.f> does not exist
checking image file <bnymwdnb1_CBT_1362046967_INCR>
>image is invalid
>>failed backup
>>COPIES not zero
checking files file <bnymwdnb1_CBT_1362046967_INCR.f>
</usr/openv/netbackup/db/images/bnymwdnb1/1362000000/bnymwdnb1_CBT_1362046967_INCR.f> does not exist
checking image file <bnymwdnb1_CBT_1362048734_INCR>
>image is invalid
>>failed backup
>>COPIES not zero
checking files file <bnymwdnb1_CBT_1362048734_INCR.f>
</usr/openv/netbackup/db/images/bnymwdnb1/1362000000/bnymwdnb1_CBT_1362048734_INCR.f> does not exist
checking image file <bnymwdnb1_CBT_1362050499_INCR>
>image is invalid
>>failed backup
>>COPIES not zero
checking files file <bnymwdnb1_CBT_1362050499_INCR.f>
</usr/openv/netbackup/db/images/bnymwdnb1/1362000000/bnymwdnb1_CBT_1362050499_INCR.f> does not exist
checking image file <bnymwdnb1_CBT_1362058434_INCR>
>image is invalid
>>still being backed-up
>>COPIES not zero
checking files file <bnymwdnb1_CBT_1362058434_INCR.f>
</usr/openv/netbackup/db/images/bnymwdnb1/1362000000/bnymwdnb1_CBT_1362058434_INCR.f> does not exist
Marianne's picture

Phone Support and request that your call be escalated to severity 2 and transferred to an engineer.

You NEED to have bpdbm log as a minimum.

This is what I see:

You have image headers files in EMM database:

bnymwdnb1_CBT_1362036644_INCR
bnymwdnb1_CBT_1362038306_INCR
bnymwdnb1_CBT_1362040004_INCR
etc...

but no matching .f files.

There is nothing that you can do to delete these headers files because they are part of EMM database.
You need assistance from Support.

Did anything happen recently (this morning) on your master server?

UNIX time 1362036644 is 02/28/2013 7:30am GMT.
UNIX time 1362038306 is 02/28/2013 7:58am GMT.
UNIX time 1362048734 is 02/28/2013 10:52am GMT.

It does not explain why catalog backup failed yesterday (27th).

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

mohamedmusthaq's picture

We sent the required logs to engineer and waiting for their response. will update everyone once the issue resolve

It does not explain why catalog backup failed yesterday (27th).

Because i had created that log directory yesterday only

 

Mark_Solutions's picture

Were any backups running when you ran the catalog consistency command?

Looking at the date stamps they are all from yesterday which implies they may have been in progress when you ran the command and hence picked up as invalid images - that could be throwing a red herring into the mix here.

I would concentrate on the one bad image, but see what support say.

It is a long thread now .. can you remind me of the O/S of the Master and Media Servers and what version of NetBackup you are running - also where these backups were writtent to (disk, tape, de-dupe)

Thanks

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

mohamedmusthaq's picture

Hi All,

Thank you very much for your valuable response.

Symantec engineer resolved the issue.

He found there were error in the last full backup, so he asked us to run the catalog full backup and then daily catalog backup.

We tried full and then daily, this time it ran successfully. Check the below notes from the engineer

 

-------------------------------------------------------------------------------------------------------------------------------------------

-Had customer run: bpdbm -consistency 1

-Checked the output, no image issues.

-Turned up logging to verbosity 5.

-Had customer re-produce the problem, schedule is an incremental.

-Issue occurred as usual, about 20 minutes into the backup.

-Looked at the bpdbm log at the time of the error. Error seen is:

db_flist_complete failed: events out of sequence - image inconsistency (229) bnymwdnb1 bnymwdnb1_CBT 4 1361976470 -Expected to see a bad image referenced but there was none. Mentioned a problem with a previous image, but does not say which.

-Had customer run a full catalog backup, it completed successfully.

-Had customer run a backup on another machine to change the catalog.

-Had customer run an incremental catalog backup, it completed successfully.

-It appears that there was something wrong with the last full catalog backup.

-Customer will verify that the next incremental catalog backup runs properly on Monday.

---------------------------------------------------------------------------------------------------------------------------------

 

Regards,

Musthak

SOLUTION