Video Screencast Help

Improving NDMP performance across multiple jobs

Created: 01 May 2013 | 7 comments
pmj's picture

Hi,

First off, I've gleaned a ton of information from this forum that has helped me solve some tough problems. I have one that I haven't found an answer for or I totally missed......

I am backing up two different systems via NDMP to a DataDomain VTL and I can only get good throughput one job at a time. One system is a Celerra connected to a Windows 2008 media server and the other is an IBM Storwize connected to an AIX media server.

For one job I can attain speeds between 50-70 kb/s, while the other jobs crawl along at 750 or maybe 1300 kb/s . We always thought the Celerra with its 4 stream limit was the bottleneck, but now with the IBM Storwize, we're seeing the same issues.

I have gone through the best practices for NDMP, VTL performance from DD, AIX performance tuning, Windows performance tuning, etc. I've added the size data buffers, the numbers data buffers, turned the logging to zero, confirmed host files, etc.

Here's some of the settings I have on the master and media servers (AIX)

NET_BUFFER_SZ: 1024000

NUMBER_DATA_BUFFERS:128

SIZE_DATA_BUFFERS: 262144

 

Checking the properties of the master/media servers in the NB 7.0.1 console:

               The bandwidth limitations is not configured,

               The firewall only has two servers, neither are NDMP

Checking the policies configuration in the NB 7.0.1. console:

                Alllow multiple data streams is checked,

                limit jobs per policy is NOT checked,

                 job priority is 2,

                Media multiplexing is set to 1 (one)

I have a feeling it's something very simple that I'm overlooking. Any suggestions would be welcome.

 

Thank you,

Philip

 

 

 

Operating Systems:

Comments 7 CommentsJump to latest comment

Jaykullar's picture

"For one job I can attain speeds between 50-70 kb/s, while the other jobs crawl along at 750 or maybe 1300 kb/s"

Do you mean that or have you got your figures the wrong way round?

Is your back from you NDMP device to DD via Media Server LAN?

What type of backup selection are you backing up?

One job at a time being faster than multiple jobs is normal, obviously there is more load, the more you put in the slower it will go, until you increase the performance in your enviornment. I would get some stats from your network cards, HBA adapters, and from DD to see where the bottleneck is and what you need to improve.

pmj's picture

Sorry, I do have my figures backwards, it's been a long road:)

My backup is from the NDMP device to DD via Media server. I am backing up entire volumes which include are basically departmental drives. User drives, common drives, files, etc.

I was considering multiplexing but I read that restores take longer with that option.

 

Thanks for the suggestions....

Jaykullar's picture

Generally NDMP is slower to disk, ideally you want to backup direct to tape, the tape device being attached to your NDMP device.

Which device are you using?

smurphy's picture

NDMP performance troubleshoting can be very dfficult. Other than the BUFFER settings, nothng else in the software will alter it.

First, create a SIZE_DATA_BUFFERS_DISK file in netbackup/db/config and set it to 262144. SIZE_DATA_BUFFERS pertains only to tape.  It would be a very worthwhile test to try this backup to a physical tape storage unit, if you have one available.

NUMBER_DATA_BUFFERS set to 128 is rather high, I would throttle that back to 64 to start.

The one stream on the Celerra that does not get good thoughput, what happens if you run that standalone, no other backup jobs running?

Another test would be an archive-to-null.  On the Celerra, it would be something like this:

server_archive <datamover name> -w -f /dev/null/-J <file path>

I have no idea what it is on the IBM.  Run this and note the timing.

Steve Murphy
NetBackup Technical Support
_________________________________
http://go.symantec.com/nb

jim dalton's picture

Tell us about your data profile...

I would say your figures are very low. Up here I have large (multiGB) files and I can get data off at 300MB/sec, no bother. But the more files and smaller they are the slower it gets, so with 10s of 100s of small files it decreases to 10..20...30MB/s. Much depends on the inode analysis which Ive been unable to improve as its the filers job priority I cant modify.

You can test this: generate a huge file and just back that up , see what happens and let us know.

Jim

Mark_Solutions's picture

One thing you may have missed, as you have covered your touch files in good detail is this one:

SIZE_DATA_BUFFERS_NDMP

NUMBER_DATA_BUFFERS_NDMP

A nice description of them and how / if you can use them is here:

http://www.symantec.com/docs/HOWTO56077

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

watsons's picture

Some filer has specific "backup parameter" for different behavior.

I have no experience working with IBM type of filers, but with EMC Celerra the "set type=" or "set hist=" can have different impact on the backup in terms of performance, restore feature etc.

How does your backup selection look like?

Have you checked with IBM vendor what they recommend for the backup parameter.