Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Slow catalog backup

Created: 26 Sep 2012 • Updated: 09 Jan 2013 | 5 comments

Hi,

Recently I have upgraded my one of site from Netbackup Version 6.5.6 to 7.5.0.3.

Upgraded completed smooth and successfully.

The catalog Backup size is about to 120 GB and backup speed is about to 25-30mb/sec on LTO-3 Tape Drives.

But I am also seeing some waiting/delayed messages

<< 

Catalog Backup
09/26/2012 07:12:48 - Info bpbkar (pid=2588) bpbkar waited 3 times for empty buffer, delayed 109 times
09/26/2012 07:12:48 - Info bptm (pid=2611) waited for full buffer 149171 times, delayed 222126 times
==
Vault Catalog
09/26/2012 02:11:28 - Info bptm (pid=20129) waited for full buffer 161968 times, delayed 243634 times
09/26/2012 02:11:28 - Info bpbkar (pid=20128) bpbkar waited 0 times for empty buffer, delayed 0 times
==
Catalog Backup
09/25/2012 07:14:33 - Info bpbkar (pid=2290) bpbkar waited 0 times for empty buffer, delayed 0 times
09/25/2012 07:14:33 - Info bptm (pid=2306) waited for full buffer 152506 times, delayed 229106 times
==
Catalog Backup
09/24/2012 07:13:30 - Info bpbkar (pid=5863) bpbkar waited 0 times for empty buffer, delayed 0 times
09/24/2012 07:13:30 - Info bptm (pid=5866) waited for full buffer 152405 times, delayed 228318 times
==
Catalog Backup
09/23/2012 07:13:43 - Info bpbkar (pid=10867) bpbkar waited 0 times for empty buffer, delayed 0 times
09/23/2012 07:13:43 - Info bptm (pid=10868) waited for full buffer 148710 times, delayed 223246 times

>> 

Seems need to do some fine tuning on the buffers side.

With Regards

Harpreet Singh

Comments 5 CommentsJump to latest comment

Marianne's picture

Which OS on your master?

Catalog backup going to local tape or over the network to a media server?

Please post full bpbkar and bptm logs.
Rename logs to reflect process name (e.g. bpbkar.txt) and post as File attachments.

The massive waits in bptm needs to evaluated against existing buffer sizes (can be seen in beginning of bptm log).

Have you tried to perform the bpbkar test as per the thread where you have originally posted? (https://www-secure.symantec.com/connect/forums/netbackup-7503-catalog-backup-really-slow )

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Harpreet_Singh's picture

Hi Marianne,

Master Server is running on Sun Solrais 10 (sparc).

Please find attached the logs for your info please.

With Regards

Harpreet

AttachmentSize
bptm_log.7z 854.08 KB
bpbkar_log.7z 3.46 MB
NBU_TSE's picture

This article is very handy when troubleshooting performance issues.

http://www.symantec.com/docs/TECH18422  - DOCUMENTATION: How to configure buffers for NetBackup in a Windows NT/2000/2003/2008 environment to improve performance

I think your symptoms match Example 2 from that technote and the solution...

Investigate disk read performance and network throughput performance.

As NetBackup doesn't have much tuning other that buffer settings, its worth finding out where the bottleneck is, often the tape drive, the disk read speed or network are the main issues.

The bpbkar nocont is useful to determine the max speed it can read from disk.

Snr Tech Support Engineer
NetBackup Database Team
Symantec

Marianne's picture

bpbkar log is for 26 Sept and ends in incomplete line before backup was complete:

01:56:57.178 [20128] <2> bpbkar do_nbu_

bptm log is for 25 Sept.

Please give us a set of matching logs that contain beginning and end of catalog backup?

As per my previous post - have you done the bpbkar test to test local disk read speed?

Command usage on Unix/Linux:

/usr/openv/netbackup/bin/bpbkar -nocont -dt 0 -nofileinfo -nokeepalives <filesystem> > /dev/null
(use actual path - /netbackup/openv/netbackup/db)
 
Please also drop down logging level. 
Level 5 is only required for in-depth troubleshooting and slows down performance.
Level 1 is sufficient for troubleshooting done here.
 
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Mark_Solutions's picture

I have done several NBU 7.5 upgrades recently and i have also noticed the large values for waited and delays

If you actually apply the formula then it would have waited for longer than the job actually ran from what i have seen!

Seems to be a "Feature" btu i haven't had a minute to report it anywhere yet.

So lets look at your figures for the one job ...

bpbkar waited 0 times for empty buffer, delayed 0 times
bptm waited for full buffer 152506 times, delayed 229106 times

so bpbkar time delay is 0 x 0.02s, bptm is 229106 x 0.03 s

so total is 0 + 6873.18 s = 114 minutes = 1.9 hours

No problem with bpbkar on that occasion but all issues were with bptm (writing to tape) so some buffer tuning for tape may help but LTO3 is not great to tune so this may all just mean that it can get the backup from the O/S faster than it can write it to tape.

Possibly set you NUMBER_DATA_BUFFERS to 64 but note that if you change the SIZE_DATA_BUFFERS file you will need to bplabel all previsouly used (but empty!) tapes before re-using them as their block size is written in the header so that needs to be overwritten to take on a new value

Hope this all helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.