Slow Filesystem Backup Performance after Upgrading to 7.1
Created: 22 Mar 2011 • Updated: 25 Mar 2011 | 18 comments
This issue has been solved. See solution.
Ladies and Gents,
I am facing strange filesystem backup performance issues after upgrading to NetBackup 7.1 and wonder if anyone has experienced a similar behavior.
Customer setup is
- master on RHEL 5.4 x86_64, upgraded from 6.5.5 to 7.1
- a couple of media servers on RHEL 5.4 x86_64, upgraded from 6.5.5 to 7.1
- a couple of media servers on SPARC Solaris 10, mixed versions 6.5.5 and 7.1
- Solaris, Linux and Windows clients with version 6.5.5 and only a few with 7.1
After upgrading the master and most of the media servers filesystem backup performance over LAN dropped below 1MB/sec for full and diffincr backups. Prior to the upgrade we were seeing write performance rates of 10 to 30MB/sec.
Now comes the interesting part:
- On the same client that is showing the poor filesystem backup performance.
- Restore performance is good at around 30MB/sec
- Oracle RMAN Backup (Archive Jobs) are writing with around 30MB/sec (on the same client!)
Extensive testing showed
- SAN Backups are doing fine.
- NDMP Backup with LAN transport are doing fine as well.
- The poor filesystem LAN backup performance is not limited to a single client of platform (Linux, Solaris and Windows, 6.5.5 and 7.1 are all affected)
- The same poor filesystem backup performance can be observed when writing to a 7.1 media server as well as to a 6.5.5 media server
- The same poor filesystem backup performance can be observed when writing to disk (Advanced Disk) or tape
- We've tested the network throughput and didn't find any glitches
- We didn't test the client disk read performance but then Oracle backup jobs are doing fine on the same system anyway (see above)
- Changing the NET_BUFFER_SZ values didn't change anything at all
- NUMBER_DATA_BUFFERS and SIZE_DATA_BUFFERS are not set and hence on their default values. Changing those parameters for testing purposes didn't change anything either.
- I can't see any hints, let alone errors, in the bpbkar and bptm logs.
Other customers that we've upgraded to 7.1 don't see any performance issues but they are running Windows master servers.
Anyone here with a RHEL master on 7.1 who can or cannot confirm a similar behavior?
Am I missing something obvious?
Any help is greatly appreciated.