Video Screencast Help

Netbackup Data Consumer waiting for full buffer, delayed 2186355 times

Created: 03 Sep 2013 | 17 comments

Hi There,

Please suggest some adjustments to reduce the above delay.

This message is from the catalog backup which ran for 9.5 hours, but was delayed for > 6.5 hours.

io_init: using 65536 data buffer size
io_init: using 30 data buffers
io_init: child delay = 10, parent delay = 15 (milliseconds)
...
Info bptm(pid=4400) waited for full buffer 632305 times, delayed 2186355 times.
fill_buffer: [4400] socket is closed, waited for empty buffer 3920 times, delayed 9758 times, read 343558062 Kbytes

NETBACKUP 7.1.0.3
The win2003 master server is on the same subnet as the win2003 media server and the library, they are all connected via fibre.
The Library is a SL500 with 4 * LTO4 drives.
Mulitiplexing is set to 6
Reduce fragment size to 1048575

The NUMBER_DATA_BUFFERS = 32
The SIZE_DATA_BUFFERS = 262144

 

Thanking you in advance,

John

Operating Systems:

Comments 17 CommentsJump to latest comment

mph999's picture

bptm waiting for full is the issue, you have 30 buffers which can't be filled quick enough to keep bptm happy (edit) or, bptm is emptying the buffers quickly and is then waaiting, hence you need more buffers ...
Other possibles would be slow read speed (disk) of the clients, or network issues.

What do you see if you run a backup of the media server itself ??

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
JohnOBrien's picture

HI Martin,

Here's the info for the media server backup:

<2> write_data: waited for full buffer 2986 times, delayed 37550 times
<2> write_data: Total Kbytes transferred 2562821

job time =10m
write time = 4m
delay time = 6m

 Been reading the tuning guide for 7.1 data paths but cannot take clear direction on where to begin.

Thanking you in advance,

John

mph999's picture

If you run a bpbkar -nocont test on the data you backed up, can you work out the read speed.

http://www.symantec.com/docs/HOWTO56131

Increase the number of buffers to 128 - I don't think this is your issue, but, 32 won't give good performance when everything else is working.

I'm out and about today, so will check back when I can.

M

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mph999's picture

Ahh I'm not thinking ...

NUMBER buffers could cause this issue ... it can affect both ends of things ...

If too low a number, the drives empty out what is there quickly, and have to wait for more data ...

So, firstthing, increase NUMBER to 128 and see if the performance changes.

 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
Mark_Solutions's picture

In your first message i see two things that conflict (unless i am reading it incorrectly)

1. io_init: using 65536 data buffer size

2. The SIZE_DATA_BUFFERS = 262144

So your data buffer size is set to 256kb (I assume you have this set on every media server) but the job uses 65kb when it runs

There are three possible reasons for this that immediately come to mind

a) The job is not running to tape

b) The media server running the backup does not have that SIZE_DATA_BUFFERS file (or it is not in the right place - /program files/veritas/netbackup/db/config)

c) The tape has been used for a while and was originally used on a system having a 64kb block size for tape.

Check a) and b) but if c) is a possibility then label the next tape with the verify option unchecked. This forces the media header to be overwritten and should label it using the correct 256kb block size which will then be used in the future.

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

rsm.gbg's picture

Hi,

I'm looking at the same problem and sort of narrowed down to some older tapes labeled wit NBU 6.5

Is ther a way of checking all the tapes for what blocksize the label has?
Then I know which tapes to relabel.

NBU Enterprise 7.1.0.3 running on Solaris 10. (LTO4 drives)

- Roland

JohnOBrien's picture

HI All,

 

Set buffers on (Master Server) to 128 and there are less delays:

- Info bptm(pid=10408) waited for full buffer 392147 times, delayed 1632874 times.

Also noticed that we don't have the

NUMBER_DATA_BUFFERS
The SIZE_DATA_BUFFERS

files on the media servers.

I will add these now and report on tomorrows performance.

Appreciate all the help guys,

John

JohnOBrien's picture

HI Guys,

Do I need to restart the box/services for new size and data buffers files to commit?

John

Mark_Solutions's picture

No need for a reboot or restart - they will take effect when the next backup runs

On tapes you do sometimes need to label them for the new buffer sizes to take effect

You can check this in the job log or the bptm log on th emedia server where it will show the block size (buffer size) being used

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

JohnOBrien's picture

HI All,

Looks like its still using 30 & 65536

 

16:38:55.366 [5840.8244] <2> io_init: bpbrm_handle = 1228, buffer address = 0x2380000
16:38:55.366 [5840.8244] <2> io_init: found index: 0
16:38:55.366 [5840.8244] <2> io_init: using 65536 data buffer size
16:38:55.366 [5840.8244] <2> io_init: CINDEX 0, sched Kbytes for monitoring = 40000
16:38:55.366 [5840.8244] <2> io_set_recvbuf: setting receive network buffer to 263168 bytes
16:38:55.366 [5840.8244] <2> io_set_recvbuf: receive network buffer is 263168 bytes
16:38:55.366 [5840.8244] <2> io_init: using 30 data buffers
16:38:55.366 [5840.8244] <2> io_init: child delay = 10, parent delay = 15 (milliseconds)

a) Job is running to tape

b)added to media server:
  c:\program files\veritas\netbackup\db\config\SIZE_DATA_BUFFERS (262144)
  c:\program files\veritas\netbackup\db\config\NUMBER_DATA_BUFFERS (128)

c) Labeled tapes

I noticed the media servers have a directory within veritas
c:\program files\veritas\netbackupdb\

This location doesn't have a config folder

Thank you,

John
 

JohnOBrien's picture

UPdate -

my files had the .txt appended

)added to media server:
  c:\program files\veritas\netbackup\db\config\SIZE_DATA_BUFFERS.txt (262144)
  c:\program files\veritas\netbackup\db\config\NUMBER_DATA_BUFFERS.txt (128)

removed and testing again.

apologies.

John

mph999's picture

Super, hopefully that will give an improvement. If so, increase the number buffers to 256 and see if there is further improvement, if not go back to 128.

M

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
JohnOBrien's picture

Ok  - changes commited:

9/4/2013 6:48:02 PM - Info bptm(pid=11032) start           
9/4/2013 6:48:03 PM - Info bptm(pid=11032) using 262144 data buffer size       
9/4/2013 6:48:03 PM - Info bptm(pid=11032) setting receive network buffer to 1049600 bytes     
9/4/2013 6:48:03 PM - Info bptm(pid=11032) using 128 data buffers

will report tomorrow on performance

Any value on going the whole hog with the buffers?

Tape:

/usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS

262144

/usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS

32

Disk:

/usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS_DISK

1048576

/usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS_DISK

32

SAN client/FT server:

/usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS_FT

262144

/usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS_FT

16

NDMP:

/usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS_NDMP

262144

/usr/openv/netbackup/db/config/NUMBER_DATA_BUFFERS_NDMP

32

 

 

thanks again,

 

John

JohnOBrien's picture

HI All,

Quick update

changed # of buffers to 128 and got errors on catalog job, error 40  -network connection issue.

Trying again and will report.

Thanks,

John

JohnOBrien's picture

Update:

128 buffers still gave me 1.7 millions delays which is a significant improvement but still needs tweaking.

Will report once source of delays are found and fixed.

John

mph999's picture

Have you tried 256 buffers ?

M

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805