Video Screencast Help

Netbackup Performance Issues.

Created: 18 Mar 2010 • Updated: 20 Sep 2010 | 3 comments

Greetings.. I am having some performance issues that I was wondering if I could get some possible pointers on.

The architecture involved here is:

Master server - netbackup 6.5.4 running on windows 32bit - XEON 2.66 - 8 gigs of RAM - Dual HBA and Dual Teamed gigabit ethernet

Client in Question - netbackup client 6.5.4 running on windows 32bit - XEON 2.66 - 4 gigs of RAM - Dual HBA and Dual Teamed gigabit ethernet - client also runs replication exec.

During the evening, replication exec replicates about 4TB worth of data to storage on the client.

During the day, I back the data up to tape - or recently - to a new quantum DXi7500 VTL.

The problem is, the backups are going extremely slow.. I am seeing the following for a portion of the job that is running right now:

kbps - 2499, 1814, 2051, 1215, 651, 1312, 2553, 2595 - and that is being backed up to a VTL with 20 free drives available, over fiber!!!

I have looked at the CPU load, memory load, and network load on both the master and on the client and both are under 50 percent. The policy is setup currently with about 30 different jobs, all of which consisit of a directory with many subdirectories. The root of each of these directories represents one of the sites I am replicating during the evening. I have about 4 "NEW STREAMS" inserted throughout the jobs as the job used to be configured with no "NEW STREAM" directives and I guess I was in hopes this might help things some.

Here are some additional settings:

Limit Jobs Per Policy - 18
Media Multiplexing - 16
Concurrent Write Drives - 16
Maximum Jobs Per Client - 20
VTL drives are emulating LTO3.

Currently only 8 jobs from the policy are running, but I think thats because I had the "jobs per policy" setting lower than it is now before the job started and I am assuming that the change wont take place "mid job..".

One thought that I had while writing this post up was that I could just replicate sites directly to the VTL - but this might not be a viable option. This was all running fine to 6 tape drives previously - but Quantum professional services came in to setup the new VTL and stole two fo the drives to dedicate to NDMP duplications.. Ever since things have not been right - and dont even talk to me about NDMP and netbackup.. its failing miserably and has been escalated to Symantec engineers...

Any help would be apprecaited - this just seems way too slow.

Comments 3 CommentsJump to latest comment

Nicolai's picture

Testing disk
Try to fire off this command, disk should be very busy, if not you have a disk I/O problem:

[install_path]\netbackup\bin\bpbkar32.exe -nocont e:\path\to\data 1> nul 2> nul

This Netbackup command read data from disk and throw the data in the bit bucket (dev null).

Testing Network
Try using Chesapeake TTCP to test the network throughput. It's a Java based freeware tool - with GUI . Whenever I suspect network performance problems I test with ttcp. You need to change the default values in order to archive a true picture. I always set buffer size to 65536 and "Num Buffers" to 16384.

There is also  the NetBackup 6.5 Backup Planning and Performance Tuning Guide as a last resort :-)

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

Omar Villa's picture

Hi,

       VTL not always is that fast and more based on the number of HBA's you have, having to much drives doesnt help to the VTL performance, so the first thing to check are Media Server Buffers, how are NUMBER_DATA_BUFFERS and SIZE_DATA_BUFFERS this are good factors and recommendation is 128x256KB respectively.

       Check the client side performance through the network, how? just FTP a large file from one of your clients to one media server, also check if you are backing up large amounts of small files in your clients, this can kill any backup even with direct Disk.

       With dd and mt commands run tape drive tests straight from your media, this will remove the Netbackup factor and prove if your issue is with NBU or infrastructure.

       Later check your pipes, what I mean is what is the speeds do you have from:

       Clients --> Media Server (1GB)
       Media Server --> vDrives (4GB)

      On your VTL check how many HBA's are Targets (Drives HBAs) and how are the drives load balanced, I mean if you have 2 HBA's with 10 drives on each HBA or 5/20, this can create a bottleneck

      Ask your vendor for their best processesing performance, I know FS says they can handle only 600MB/Sec on the best conditions.

Check all this and let us know how you do.
Regards.

Omar Villa

Netbackup Expert

Twiter: @omarvillaNBU

 

schmaustech's picture

In my experience, VTL's don't always offer the best speeds for backup, although they provide great speeds for restore.  I was in an environment where we used Protectier as the front end VTL with Hitachi SATA disk on the backend.  When I would run a backup from a Netapp 3040 via NDMP directly over fiber to the VTL, I would see about 20,000kbs.  I could take that same job and route it directly to a SDLT600 fiber attached tape drive and get about 42,000kbs.

The moral of the story is the VTL is probably the bottleneck.

Regards,

Benjamin Schmaus