Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Exchange 2010 DAG bkups wait for full buffer too frequently

Created: 17 Aug 2013 | 3 comments
tythecellist's picture

Hey friends, I could use a sanity check here.  Goal is to improve backup performance of a DAG cluster node, as (to me, and I'm fairly new to DAG backups) the number of buffer fills and delays is excessive (see below.)   Backups are completing successfully ... just slow (i.e. 5-6 MB/sec, written to a PureDisk STU on the media server.)   I understand that slow perf is linked to high #s of fills and delays, so I understand why the slow write rate is there ...

Environment is as follows:  NetBackup 5230 Appliance (master and media servers) running 7.5.0.6, and Windows 2008R2 running Exchange 2010SP2 (7.5.0.5.)     There are approximately 3000 mailboxes associated with this particular database.

On the media server we've configured

* NUMBER_DATA_BUFFERS_DISK to 512

* SIZE_DATA_BUFFERS_DISK to 1048576

On the Exchange clients (Win2008) we've configured

* communication buffer size to 256K

...per the various technotes, other postings here, etc.

=======================================================================================
08/16/2013 19:09:56 - Info nbjm (pid=15283) starting backup job (jobid=3841) for client <client>, policy nadag01_DBCG01, schedule Full
08/16/2013 19:09:56 - Info nbjm (pid=15283) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=3841, request id:{5959258E-06D1-11E3-B833-F9234B1C1687})
08/16/2013 19:09:56 - requesting resource stu_disk
08/16/2013 19:09:56 - requesting resource .NBU_CLIENT.MAXJOBS.
08/16/2013 19:09:56 - requesting resource .NBU_POLICY.MAXJOBS.nadag01_DBCG01
08/16/2013 19:10:00 - granted resource  .NBU_CLIENT.MAXJOBS.
08/16/2013 19:10:00 - granted resource  .NBU_POLICY.MAXJOBS.nadag01_DBCG01
08/16/2013 19:10:00 - granted resource  MediaID=@aaaae;DiskVolume=PureDiskVolume;DiskPool=dp_disk_;Path=PureDiskVolume;StorageServer=;MediaServer=
08/16/2013 19:10:00 - granted resource  stu_disk_
08/16/2013 19:10:01 - estimated 0 kbytes needed
08/16/2013 19:10:01 - Info nbjm (pid=15283) started backup (backupid=nadag01_1376698201) job for client , policy nadag01_DBCG01, schedule Full on storage unit stu_disk_
08/16/2013 19:10:02 - started process bpbrm (pid=25323)
08/16/2013 19:10:04 - connecting
08/16/2013 19:10:05 - connected; connect time: 0:00:00
08/16/2013 19:11:11 - Info bpbrm (pid=25323)  is the host to backup data from
08/16/2013 19:11:11 - Info bpbrm (pid=25323) reading file list from client
08/16/2013 19:11:14 - Info bpbrm (pid=25323) starting bpbkar on client
08/16/2013 19:11:26 - Info bpbkar (pid=18612) Backup started
08/16/2013 19:11:26 - Info bpbrm (pid=25323) bptm pid: 25426
08/16/2013 19:11:26 - Info bptm (pid=25426) start
08/16/2013 19:11:26 - Info bptm (pid=25426) using 1048576 data buffer size
08/16/2013 19:11:26 - Info bptm (pid=25426) setting receive network buffer to 1048576 bytes
08/16/2013 19:11:26 - Info bptm (pid=25426) using 512 data buffers
08/16/2013 19:11:28 - Info bptm (pid=25426) start backup
08/16/2013 19:12:21 - begin writing
08/16/2013 19:13:30 - Info bptm (pid=25426) backup child process is pid 25659
08/17/2013 08:42:22 - end writing; write time: 13:30:01
08/17/2013 08:42:57 - Info bpbrm (pid=25323) DB_BACKUP_STATUS is 0
08/17/2013 08:43:00 - Info bpbkar (pid=18612) 5000 entries sent to bpdbm
08/17/2013 08:43:01 - Info bpbkar (pid=18612) 10000 entries sent to bpdbm
08/17/2013 08:43:03 - Info bpbkar (pid=18612) 15000 entries sent to bpdbm
08/17/2013 08:43:04 - Info bptm (pid=25426) waited for full buffer 103105 times, delayed 1030255 times
08/17/2013 08:43:22 - Info bptm (pid=25426) EXITING with status 0 <----------
08/17/2013 08:43:22 - Info  (pid=25426) StorageServer=PureDisk:; Report=PDDO Stats for (): scanned: 278687770 KB, CR sent: 48961491 KB, CR sent over FC: 0 KB, dedup: 82.4%
08/17/2013 08:43:30 - Info bpbrm (pid=25323) validating image for client nadag01
08/17/2013 08:43:31 - Info bpbkar (pid=18612) done. status: 0: the requested operation was successfully completed
the requested operation was successfully completed  (0)
====================================================================================
 
Feels like I'm missing something here .. help?
Operating Systems:

Comments 3 CommentsJump to latest comment

captain jack sparrow's picture

You can try reducing size_data_buffers to 512/256 and check if it helps (i knw sounds bit odd, but truely performance tuning varies from environment to environment)

Never saw this high values in number_data_buffers (512 !!!!)

try it with 16 or 32

Refer performance tuning guide for 7.1 (applicable for 7.5 as well)

Also have a look at http://www.mass.dk/netbackup/guides/49-netbackup-buffer-tuning.html

 Cheers !!!

CJS

tythecellist's picture

Thanks for the reply!    Your comment holds true still where the target device is DISK, correct?   (in other words, the # of buffers is being specified in NUMBER_DATA_BUFFERS_DISK.)   

Not doubting your suggestion, just confirming.  Again, thanks.

captain jack sparrow's picture

yes. your assessment is correct :-)

 Cheers !!!

CJS