Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Staging jobs to disk takes hours longer than writing to tape on AIX client.

Created: 09 Jan 2013 • Updated: 22 Jan 2013 | 20 comments
This issue has been solved. See solution.

Media Operating System:  Server 2003

NetBackup Version:  7.5

 

Client Operating System:  AIX 6.1

NetBackup Version:  7.5

 

 

I run a daily backup job on an AIX 6.1 client, the job usually takes 3:04 on a consistent basis.  Recently I setup a drive enclosure with about 3TB of space using 15k SAS drives for the purpose of disk staging.  Normal write speeds when backing up this client to tape is around 32-35 kb/sec, write speed tests to disk exceeded those numbers.  I set up the policy to stage to disk first then duplicate to tape, the actual write speed while the job was active was 33 kb/sec but the job actually took 5 plus hours to complete. The last test job I conducted took a disappointing 7:24 to complete.  In theory staging to disk should be much faster, any ideas on why staging this job takes so long.  All of the firmware on the Perc 5E controller and drives are up to date.

Comments 20 CommentsJump to latest comment

Mark_Solutions's picture

Do you have any tuning in place for disk backups?

If not try the following files on your media server:

Create the directory \program files\veritas\netbackup\db\config

Create a file (no extension) named NUMBER_DATA_BUFFERS_DISK and put a value of 32 in it

Create a file (no extention) named SIZE_DATA_BUFFERS_DISK and put a value of 1048576 in it

No need to re-start any services - just re-try the backup and see how it goes

Also check that no AV is running - or at least you disk area is excluded

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

SOLUTION
jpichardo's picture

I do not have any disk tuning in place, we just started staging at this site in hopes of shaving some time off that 3hr backup time, instead we have increased it by several hours. In your experience will those settings dramatically impact jobs writing to disk?

Nicolai's picture

And how is the 3TB configured - as a stripe or a simple concatenation ?

If it a simple cincatenation one disk will take the entire load whereas a stripe will share the load.

 

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

Mark_Solutions's picture

The buffer tuning can make a big difference - but as Nicolai says you disk must be OK - I was basing my advise on your test write figures so give the files a try and see what difference they make

They certainly wont do any harm

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

jpichardo's picture

I did implement those changes and will test tonight, the disks are in a RAID 5 configuration.

Yasuhisa Ishikawa's picture

Normal write speeds when backing up this client to tape is around 32-35 kb/sec

is it really around 35 KB/s?

If so, I would think this low performnce is brought by other reason. Can you test with previous configuration again?

Usually low performnce in such order is caused by network or client issue. it is better to measure performance with "bpbkar -nocont" and GEN_DATA directive.

http://www.symantec.com/docs/TECH75213

http://www.symantec.com/docs/HOWTO56131

 

Authorized Symantec Consultant(ASC) Data Protection in Tokyo, Japan

jpichardo's picture

Yasuhisia-

I will use the info in the links you provided to work with the Unix admin, perhaps we can achieve additional performance.

jpichardo's picture

Last night's backup has improved dramtically after making the disk tuning changes, it went from 5-7 hours on previous jobs to 3:04 . This is exactly how long is takes to write the same job to tape so I am still looking to improve upon that time and shave off 30 min to 1 hour.

Mark_Solutions's picture

If you are happy that the client can feed data that quickly and the media server can accept it and write it to disk then try the following:

Increase the number data buffer disk to 64 (try 128 after that)

You can also try tuning the network buffer size - this is set on the clients host properties (client and media server - having them match does help)

Some people find that 16 works great - others ramp it right up .. 32, 64, 128 etc

Just make sure that it does not have any adverse affect on any other backups while doing this - eventually you will hit a limit anyway as every system has its limits - such as the hardware of the NIC etc

Try these and see if it gets even better

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

jpichardo's picture

I increased the number data buffer disk to 64 and will post the results tomorrow. 

Is there any way to enable email alerts to get status of failed jobs and job completion?

Tomer Gurantz's picture

If you end up getting the exact same results, perhaps your bottleneck (and quoted tape / backup speed) is really the client disk, or network between client and media server. That would explain why you'd see the almost exact same performance from 2 quite different backup destinations.

Senior Principal Technical Education Consultant with Symantec Corporation

Mark_Solutions's picture

e-mail alerts can be setup via Master Servers Host Properties and installing blat on the Master, or better still use OpsCenter (it is free!)

 

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

jpichardo's picture

Can OpsCenter be installed on the Master server or do you suggest installing on a seperate server, maybe a VM?

Will Restore's picture

Installation of OpsCenter server software on a NetBackup master server or

media server is possible if you want to monitor only one master server. An

example is the master server on which the OpsCenter server software is

installed.

To monitor more than one master server, Symantec recommends that you

install the OpsCenter server software on a separate standalone server.

 

 

OpsCenter Server and Agent are supported in a VMware virtual machine guest

operating system environment.

 

 

Will Restore -- where there is a Will there is a way

Mark_Solutions's picture

I usually advise my customers to have a dedicated server for OpsCenter - purely due to disk space and processor usage - I dont like to add anything to a Master that doesn't need to be there

How are the tests going?

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

jpichardo's picture

I tested again over the weekend and the job took around 4hrs to complete after bumping up the disk buffer to 64. There may have been other factors that contributed to the 4hr backup time instead of the previously achieved 3hrs. I will test again tonight with the buffer set at 64.

Marianne's picture

Please go back to Yasuhisa's post and test client's read speed from disk. 
If client has inefficient disk/lun layout or suffers from bad fragmentation, any network or buffer tuning will have limited effect.

Another way to cut down on backup time is to implement multistreaming to enable simultaneous backup of 2 - 4 streams from the client, effectively cutting backup time in half (or more).

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

jpichardo's picture

Mark’s initial disk tuning recommendation provided the desired results; the backup went from 7 hours down to 3 hours, reducing the time by more than half.  I will use the suggestions provided by the other members as well to further tune this job. 

 

The UNIX client has more than one raid array on which the data resides, is it safe to send multiple streams to the disk storage unit which is one large volume in a RAID 5 setup? Should I set up a stream for each RAID array on the Client?

Mark_Solutions's picture

If you have more than one RAID then you should be fine to use a stream from each RAID set - as long as the backbone of the server and the network can handle more throughput then it may give you even better results

Glad things are working out for you

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

jpichardo's picture

Update: After configuring multiple streams I was able to reduce the time it takes to backup this client by another 30 mins, bringing the total backup time to 2 1/2 hours.

In summary by using a combination of disk tuning and multi streaming I was able to significantly reduce my backup time on this AIX client. Thank you all for your expertise.