Please ensure you check out this post whenstarting new threads.

https://www-secure.symantec.com/connect/blogs/minimum-information-required-when-logging-problem-details

Did this every work ?

If it did, when did it last work

What has changed (if it did work, something HAS changed)

Is the master servre also the media server runnign the backup

Does it fail every time, always at the same point ?

Can you run other similar sized backups from this server, run a test if necessary

From this, the performance looks poor :

1/26/2012 10:55:28 AM - Info bptm(pid=960) waited for full buffer 31786 times, delayed 40142 times    
1/26/2012 10:55:28 AM - Info bpbkar32(pid=3124) bpbkar waited 10922 times for empty buffer, delayed 11000 

This could well be the main issue :

1/26/2012 11:00:33 AM - Info bpbkar32(pid=3124) done. status: 44: network write failed       
network write failed(44)

The main question I guess at the moment, is if the master is acting as the media server.

If the master is the media, then tyhe problem may be a little mopre complex

If the media server is separate from the master server, I would suspect the Network.

Martin

This may help...or not.

Try running the backup to NULL to test local performance. 
http://www.symantec.com/docs/TECH17541

Then move onto checking disk defragmentation.

Check network speed on uplinks and switch ports are matched. (Had a mismatch last week on new client and this fixed it.)

Also, please explain what troubleshooting steps you have taken, and what you suspect the issue to be.

Apologies, but I am unsure why Accredited members are not posting up some seriously detailed troubleshooting steps, and are not running basic testing before logging a new post.

These lines:

1/26/2012 10:55:28 AM - Info bptm(pid=960) waited for full buffer 31786 times, delayed 40142 times    
1/26/2012 10:55:28 AM - Info bpbkar32(pid=3124) bpbkar waited 10922 times for empty buffer, delayed 11000 times.   Thanks,

... should show that some performance testing needs to be done, it shows that the issue is most likely between the client and the media server (could be same box) - hence my questions, and Stuarts suggestions.

The questions in here https://www-secure.symantec.com/connect/blogs/minimum-information-required-when-logging-problem-detailsMartin

should be supplied from accredited memebers on every post, without having to be asked for them - I appreciate some questions will not be relevant to every problem, but they are a good guide, and show the level of details needed.  If a question is not relevant, think "what other details can I give that may be needed in place of this question".

Thanks,

Martin

Unpatched NBU 7.1?

W2008 patch level? Physical resources on master?

From the opening post, The Master seems to be Media server as well - NBU-Server-hcart-robot-tld-0 .... client NBU-Server

There seems to a disconnect between bpbrm and bpdbm:

Error bpbrm(pid=5288) db_FLISTsend failed: network write failed (44) 

You will need all relevant logs....

bpbrm and bptm at VERBOSE = 5 please ... 

@ mph999

The same machine is Master and Media Server. (Other Backups of different machine are running fine. And the Catalog bacjup is being backed up from Netbackup Server so the NBU does not seems loaded)

@ Yasuhisa

Client read timeout is 6000

@ Stuart

My Network seems fine and all backups are going good.

=========================

Will share the bpbrm and bptm logs soon

Thanks all for kind inputs and follow-up on my Post.

I increased the Browse time out to 6000 from 300. The read time out was already 6000 while backup was failing. But after increasing the Browse time out to 6000 from 300 my backup got successful. I triggered the backup again and trying to notice that this may be the cause.

Yes you are right. That wqas not the solution. Again failed. I opened a case with Symantec and will share the status in the end

Re-reading the original post it looks like the backup actually completed but then failed to update itself with the final result.

You have said that the system is not busy but it could actually be a port lock out issue causing this.

The internal communication gets its ports locked out so by the time the backup finishes it cannot communicate

It writes for 13 minutes 58 seconds (guess your catalog is not very big?), but then has the failure exactly after 5 minutes.

Four things worth doing here:

1. Add the following registry key to the Master Server:

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\

DWORD – TcpTimedWaitDelay  - Decimal Value of 30

2. From a "run as administrator" from command line

 Netsh int ipv4 set dynamicport tcp start=10000 num=50000

 This gives it 60000 connections, the default is 16383

3. On the Master Servers Host Properties ensure that in the Timeouts section the Client Connect, Client Read, File Browse and Media Server Connect timeouts are set to 600, just to be sure.

4. Now it could be something to do with bpdbm or bpcd which have hard coded 5 minute timeouts.

Check to make sure you dont have a load of bpdbm processes running. If the system is quiet do a bpdown and see if the bpdbm processes are still running - then do another bpdown and see if they go. If not reboot the Master for a full cleanup - regular process cleanups / reboots are always worth doing.

Also plain 7.1 has issues, especially for bpdbm, and there is an EEB to help overcome this which comes to light for GRT Restore browsing errors. This increases the bpdbm hardcoded timeout and I believe this is included in 7.1.0.3 so I would strongly reccomend patching your Master Server

Hope this helps

Why have you opened a case witgh Symantec - you should open the case with either your Network support guys, or whoever supports the operating systems.

This is not a NetBackup issue.

Martin

I have a single machine which is Master / Media Server. My Catalog is on the same machine. All other backups of different Server(which included SQL,Flat file backup and Exchange2010 backup) on DSU and Tape Library is going fine and 100% perfect. How this could be the Network error ?

How much disk space do you have on the drive where your databases are held?

You are starting to get low on disk space

I asked because during the catalog backup the databses are staged which takes up additional space

When EMM detects a shortage of disk space it can shut itself down, which would cause this error during the catalog backup.

This is not supposed to happen until 1 or 2 % free space but I have seen many issues when it gets to 10% as well

See if you can clear down your logs to see how much free space you can make then see if it works

OK, my error :

Try this ...

 bpimmedia -mediaid ddd

Thanks,

MArtin

I would clear down your logs to free up the disk space and try the catalog backup again.

If that works then I would look at re-locating your catalog files (the db folder can be relocated relatively easy) or as an alternative you can relocate your logs elsewhere to save them using your space up

Give it a try and I will get you the links for moving your db folder and the logs - but lets see if it works first

See the below logs. It seems that the space was out of space. But now my backup is running successful with less space.

For example

When my backup was failing that time the space was around 12GB and now the space is around 9GB and backup is running fine.

See the below logs for reference:

log entry in bpdbm

08:55:27.560 [6420.1388] <2> get_adaptable_string: (4) network read() error:  No buffer space available.

Network buffer space and disk space is not the same...

Please do not leave logging level at 5 if you are not troubleshooting a specific problem.

Level 5 logs grow large, make your system slow, consume system resources, etc... ONLY increase logging level while troubleshooting specific problems when you have an open case with Symantec. Drop down logging levels to 0 as soon as all necessary logs have been collected.
Level 0 logs are sufficient to troubleshoot day-to-day issues.

Please also check at Windows level for disk fragmentation.

This couldnt be the problem ?

08:55:27.560 [6420.1388] <2> get_adaptable_string: (4) network read() error:  No buffer space available.

Yes, that could be causing the problem.

I was trying to say that it points to a network problem, not disk space.

What sort of that buffer could be ?

(4) network read() error:  No buffer space available

I also pointed out it could be a network problem, but my advice was discounted ...

To answer the latest questions ...

The send /receive buffers on the NIC ... 

OK, lets straighten something out ...

If we suggest something, or ask for detail that seems not relevent, please provide it anyway - we are not asking for the fun of of it ...

Just becuase you can see no reason for the network being the issue, does not mean it is not the issue, until it has been PROVED 100%.

A little story to demonstrate ...

 I had a case recently, NBU would not mount a tape in the drive, hung when mounting.

Now, I am the first to point out that tape issues are usually not NBU ... but ...

Robtest could load a tape

The operating system could access that tape, and, write to it using tar ... etc ...

Only fault, was NBU not mounting the tape ...

So, even I had to admit it was looking like NBU was the fault ....

Nope, turns out, it was the firmware on the drive(s).

So, you see, no matter what the issue does, or does not look like, it can be the most unexpected thing that casues the problem.

Martin

GOOGLE found some info for us.

Firstly a Symantec TechNote with a different status code but the same network buffer error.
The TN is very old, so I cannot tell if it will be applicable to W2008.

http://www.symantec.com/docs/TECH55906

The error is caused when the master server ran out of network buffer space.

Resolution:
Check the boot.ini on the master and verify that the /3GB switch is not used.  What the /3GB switch does is allocate 1GB of address space to the Windows operating system and 3GB of address space to user mode processes.  This allows Windows to better accommodate demanding applications such as Exchange and SQL Servers.

By only allowing the operating system to allocate 1GB of address space, a limit is also placed on the amount of network buffers your operating system can use.  Remove the /3GB entry from your boot.ini and the "No buffer space available" error from the operating system should not reappear.

Second Google find:

http://docs.dal.net/docs/connection.html#6

6 · [10055] No buffer space available

Also check out the post I did earlier in the thread to tune your network  - did you do these and reboot the server?:

1. Add the following registry key to the Master Server:

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\

DWORD – TcpTimedWaitDelay  - Decimal Value of 30

2. From a "run as administrator" from command line

 Netsh int ipv4 set dynamicport tcp start=10000 num=50000

 This gives it 60000 connections, the default is 16383

Also, what NET_BUFFER_SZ values are you using (if any - \ntebackup\bin\)?

mph999 first of all I would like to pay bundle of thanks in the regard of your kins help and I do value it. If at some place if you feel that your suggesstions are being ignored so sorry for that.

Mark and Marianne let me do this and share the result