Waits and delays for full and empty buffers message recorded in NetBackup logs and also presented in the Activity Monitor for NetBackup 7.1 and later

Article:TECH145578  |  Created: 2010-12-05  |  Updated: 2013-11-15  |  Article URL http://www.symantec.com/docs/TECH145578
Article Type
Technical Solution


Issue



Understanding waits and delays for full and empty buffers message produced in NetBackup logs, and also presented in the Activity Monitor for NetBackup 7.1 and later.


Error



From bptm log:

08:18:26.087 [5312.6256] <2> write_data: waited for full buffer 24552 times, delayed 1221273 times

From bpbkar log:

08:18:26.072 AM: [5416.6628] <4> BufferManagerLegacySharedMemory::~BufferManagerLegacySharedMemory(): INF - bpbkar waited 64737 times for empty buffer, delayed 66780 times.
 

 


Solution



To calculate how much delay was experienced by bptm, we will need the following two lines from bptm log:

08:18:26.087 [5312.6256] <2> write_data: waited for full buffer 24552 times, delayed 1221273 times
00:49:29.035 [5312.6256] <2> io_init: child delay = 10, parent delay = 15 (milliseconds)

Delay and Wait counters will help to determine which process has to wait more often: data producer or data consumer side of the read/write operation. During a backup, if the client and the media server is the same system, bpbkar process on the client that reads the data from the storage (disk) and places it into the shared memory segments is the Data Producer, and then single bptm process that reads the data from the shared memory segments and writes to the storage (tape or disk) is the Data Consumer. If the client and media server are different systems, bpbkar process on the client reads the data from the disk and places it to the network, then bptm child process that reads the data coming from the clients and places the data into the shared memory segments on the media server is the Data Producer. And bptm parent process that reads the data from the shared memory segments on the media server and writes to the destination storage (tape or disk) is the Data Consumer.

Data Producer needs an empty buffer, while Data Consumer needs a full buffer. So, the log events above are from the Data Consumer bptm side. 

If the Data Consumer has to wait for a full buffer, it increments the wait and delay counters to indicate that it had to wait for a full buffer. After a delay, the Data Consumer checks again for a full buffer. If a full buffer is still not available, the Data Consumer increments the delay counter to indicate that it had to wait (delay) for a full buffer. Data Consumer repeats the delay and full buffer check steps until a full buffer is available.

Analysis of the wait and delay counter values recorded in bptm log and also presented in the activity monitor events for the job (in NetBackup 7.1 only and later versions) indicates which process, Producer or Consumer, has had to wait most often and for how long.

Now it is shown below how much delay is experienced by the bptm process shown above:

Total delay = how many times bptm was delayed x parent delay = 221273 x 0.015 = 18319 secs = 305 mins = over 5 hours

To find out how to solve this problem using the following configuration files, how these processes (Consumer and Producer) work for backups and restores,  please read NetBackup Planning and Performance Tuning Guide for additional information :

 

SIZE_DATA_BUFFERS

NUMBER_DATA_BUFFERS

PARENT_DELAY

CHILD_DELAY

 




Article URL http://www.symantec.com/docs/TECH145578


Terms of use for this information are found in Legal Notices