How to configure buffers for NetBackup in a Windows NT/2000/2003/2008 environment to improve performance

Article:TECH18422  |  Created: 2006-01-17  |  Updated: 2013-10-24  |  Article URL http://www.symantec.com/docs/TECH18422
Article Type
Technical Solution

Product(s)


Issue



DOCUMENTATION: How to configure buffers for NetBackup in a Windows NT/2000/2003/2008 environments to improve performance


Solution



For most configurations, the default NetBackup buffer settings are correct and there is no need to adjust them for the purpose of performance.  Furthermore, there are factors outside of NetBackup which effect performance and should be reviewed.  Some of these external factors include Host Bus Adapter (HBA) cards, SCSI cards, network interface card (NIC) settings, client disk I/O speed, network latency, and tape drive I/O.  All of these should be reviewed to determine their respective impact on backup and restore speeds prior to any attempts to tune NetBackup.  

On a Windows server, four different buffer settings can be modified to enhance backup performance.  Those settings are:
  • NUMBER_DATA_BUFFERS: The number of buffers used by NetBackup to buffer data prior to sending it to the tape drives.  The default value is 16.
  • SIZE_DATA_BUFFERS: The size of each buffer setup multiplied by the NUMBER_DATA_BUFFERS value.  The default value is 65536.
  • NET_BUFFER_SZ: The network receive buffer on the media server.  It receives data from the client. The default value is 256K.  
  • Buffer_size: The size of each data package sent from the client to the media server.  The default value is 32K.

Overview:
When a backup is initiated, the client packages data of the amount specified by the Buffer_size value, and transfers the information to the media server, which in turn, buffers that data in the NET_BUFFER_SZ.  When the NET_BUFFER_SZ is full, it transfers data to the array of space created by a combination of NUMBER_DATA_BUFFERS and SIZE_DATA_BUFFERS.  As soon as at least one of the SIZE_DATA_BUFFERS is full, assuming the drive is ready to write, the information is written to the tape drive.  


To troubleshoot performance issues related to buffer settings, enable and review the bptm log on the media server and the bpbkar log on the client.  On the media server, go to the <INSTALL_PATH>\Veritas\NetBackup\Logs directory, and create a  bptm  folder.  On the client, go to the <INSTALL_PATH\Veritas\NetBackup\Logs directory and create a bpbkar folder.  Then, when there are no backups or restores running, stop and restart all NetBackup services on the media server and client.

Examine the bptm and bpbkar logs for references to waits and buffers and compare one side with the other.  Individually, each side's number of waits means nothing.  Only when compared to the opposite side, can you deduce where a potential bottleneck is:

Example 1:
--------------------
bptm:
13:16:06.546 [2776.2620] <2> mpx_read_data: waited for empty buffer 0 times, delayed 0 times
 
bpbkar:
1:16:6.875 PM: [2532.1304] <4> tar_backup::OVPC_EOFSharedMemory: INF - bpbkar waited 0 times for empty buffer, delayed 0 times
 
Conclusion: Neither the bptm or bpbkar logs are waiting.  This means that you could increase the buffers on each side to pass greater amounts of data and possibly see greater performance.

Example 2:
--------------------
bptm:
12:32:25.937 [2520.2176] <2> write_data: waited for \b full buffer\b0  19285 times, delayed 18012 times
 
bpbkar:
12:32:44.875 PM: [1372.135] <4> tar_backup::OVPC_EOFSharedMemory: INF - bpbkar waited 612 times for empty buffer, delayed 651 times
 
Conclusion: The bptm process is waiting to receive data from the client many thousands of times more then the client is waiting on the bptm process.  The bottleneck here is on the client.  Increasing SIZE_DATA_BUFFERS or NUMBER_DATA_BUFFERS will not improve performance.  Finding out why the client is slow to pass data to the media server, is the key.  Investigate disk read performance and network throughput performance.

Example 3:
--------------------
bptm:
13:31:42.343 [1800.2108] <2> fill_buffer: [2016] socket is closed, waited for \b empty buffer\b0  18834 times, delayed 33954 times, read 3326864 bytes
 
bpbkar:
1:30:25.301 PM: [1242.916] <4> tar_backup::OVPC_EOFSharedMemory: INF - bpbkar waited 27420 times for empty buffer, delayed 26525 times
 
Conclusion: The quantity of waits listed in bpbkar indicates the problem is on the media server.  In the bptm log we see it waiting a significant amount of times for empty buffers. The client is waiting to send data, until there is a place to put it.  This indicates the data is not passing to the tape drives fast enough.  Increasing the SIZE_DATA_BUFFERS or NUMBER_DATA_BUFFERS will not help.  The key here is to figure out if the performance bottleneck is the tape drive write speed, or the HBA/SCSI transfer speed.

NUMBER_DATA_BUFFERS
To change the NUMBER_DATA_BUFFERS, create the <INSTALL_PATH\NetBackup\db\config\NUMBER_DATA_BUFFERS file.  It should contain the number of buffers to be created at the start of backup.  If the file is not present, the default of 16 will be used.

SIZE_DATA_BUFFERS
Remember this is the size of each buffer setup on the media server, the number of which is defined by NUMBER_DATA_BUFFERS.  Exercise caution when changing the value from the default setting as some SCSI cards, HBA cards, and tape drives cannot transfer buffers greater than 65536 in size.  After changing this value, it is important to test both backups and restores, as sometimes data can be written at the modified size, but can not be read at the modified size.   Please review the specifications of the HBA card, SCSI card and tape drive to confirm the value is not being exceeded.  
 
To change the SIZE_DATA_BUFFERS setting, create a file called SIZE_DATA_BUFFERS in the <INSTALL_PATH>\NetBackup\db\config.  Add the desired value in this file, expressed in bytes, in multiples of 1024. Ensure the file has no extensions after saving.  If the file is not present, the default for the SIZE_DATA_BUFFER is 65536.  

NET_BUFFER_SZ
Note the NET_BUFFER_SZ value is the size of the buffer on the media server which receives data from the client server.  The NET_BUFFER_SZ is set on the media server by creating the NET_BUFFER_SZ file in the <INSTALL_PATH>\NetBackup directory.  The value in the file needs to be a multiple of 1024.  If the file is not present, the default value of 262144 (or 256K) is being used.  If the file is created for tuning purposes, be sure the file has no extensions.  
 
NOTE: It is frequently helpful to match the Buffer_Size on the clients to NET_BUFFER_SZ on the media server.

Buffer_size
Please note the value of the Buffer_size does not show up in the bptm log file, but can be viewed and set from the master server or by modifying a registry value.  The default value for the Buffer_size setting is 32K.  
 
On the client server, the buffer size (Buffer_size) can be modified in two ways.  
 
1. From the NetBackup Administration console, Host Properties | Clients, double-click the client you are configuring.  Expand the options under Windows Client | select Client Settings.  Change the Communications buffer to the desired size in KB.
 
Figure 1
 
 
2. Edit the following registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\VERITAS\NetBackup\CurrentVersion\Config\Buffer_Size
 
Change the decimal value to the desired setting.  
 
NOTE: Performance can sometimes be enhanced by setting this value the same as the NET_BUFFER_SZ on the media server.

Viewing values of the buffer settings:
A review of the bptm log file will show the current settings for the NET_BUFFER_SZ, SIZE_DATA_BUFFERS, and NUMBER_DATA_BUFFERS   The following is an excerpt from the bptm log from a media server showing the value of these three buffers.
 
16:54:40 [284.2260] <2> io_set_recvbuf: setting receive network buffer to 32768 bytes (NET_BUFFER_SZ)
16:54:40 [284.2260] <2> io_init: using 32768 data buffer size (SIZE_DATA_BUFFERS)
16:54:40 [284.2260] <2> io_init: CINDEX 0, sched Kbytes for monitoring = 30000
16:54:40 [284.2260] <2> io_init: using 16 data buffers (NUMBER_DATA_BUFFERS)
 



Legacy ID



244652


Article URL http://www.symantec.com/docs/TECH18422


Terms of use for this information are found in Legal Notices