Staging jobs to disk takes hours longer than writing to tape on AIX client.
Media Operating System: Server 2003
NetBackup Version: 7.5
Client Operating System: AIX 6.1
NetBackup Version: 7.5
I run a daily backup job on an AIX 6.1 client, the job usually takes 3:04 on a consistent basis. Recently I setup a drive enclosure with about 3TB of space using 15k SAS drives for the purpose of disk staging. Normal write speeds when backing up this client to tape is around 32-35 kb/sec, write speed tests to disk exceeded those numbers. I set up the policy to stage to disk first then duplicate to tape, the actual write speed while the job was active was 33 kb/sec but the job actually took 5 plus hours to complete. The last test job I conducted took a disappointing 7:24 to complete. In theory staging to disk should be much faster, any ideas on why staging this job takes so long. All of the firmware on the Perc 5E controller and drives are up to date.
Comments 20 Comments • Jump to latest comment
Do you have any tuning in place for disk backups?
If not try the following files on your media server:
Create the directory \program files\veritas\netbackup\db\config
Create a file (no extension) named NUMBER_DATA_BUFFERS_DISK and put a value of 32 in it
Create a file (no extention) named SIZE_DATA_BUFFERS_DISK and put a value of 1048576 in it
No need to re-start any services - just re-try the backup and see how it goes
Also check that no AV is running - or at least you disk area is excluded
Hope this helps
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
I do not have any disk tuning in place, we just started staging at this site in hopes of shaving some time off that 3hr backup time, instead we have increased it by several hours. In your experience will those settings dramatically impact jobs writing to disk?
And how is the 3TB configured - as a stripe or a simple concatenation ?
If it a simple cincatenation one disk will take the entire load whereas a stripe will share the load.
Assumption is the mother of all mess ups.
If this post answered your'e qustion - Please mark as a soloution.
The buffer tuning can make a big difference - but as Nicolai says you disk must be OK - I was basing my advise on your test write figures so give the files a try and see what difference they make
They certainly wont do any harm
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
I did implement those changes and will test tonight, the disks are in a RAID 5 configuration.
is it really around 35 KB/s?
If so, I would think this low performnce is brought by other reason. Can you test with previous configuration again?
Usually low performnce in such order is caused by network or client issue. it is better to measure performance with "bpbkar -nocont" and GEN_DATA directive.
http://www.symantec.com/docs/TECH75213
http://www.symantec.com/docs/HOWTO56131
Authorized Symantec Consultant(ASC) Data Protection in Tokyo, Japan
Yasuhisia-
I will use the info in the links you provided to work with the Unix admin, perhaps we can achieve additional performance.
Last night's backup has improved dramtically after making the disk tuning changes, it went from 5-7 hours on previous jobs to 3:04 . This is exactly how long is takes to write the same job to tape so I am still looking to improve upon that time and shave off 30 min to 1 hour.
If you are happy that the client can feed data that quickly and the media server can accept it and write it to disk then try the following:
Increase the number data buffer disk to 64 (try 128 after that)
You can also try tuning the network buffer size - this is set on the clients host properties (client and media server - having them match does help)
Some people find that 16 works great - others ramp it right up .. 32, 64, 128 etc
Just make sure that it does not have any adverse affect on any other backups while doing this - eventually you will hit a limit anyway as every system has its limits - such as the hardware of the NIC etc
Try these and see if it gets even better
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
I increased the number data buffer disk to 64 and will post the results tomorrow.
Is there any way to enable email alerts to get status of failed jobs and job completion?
If you end up getting the exact same results, perhaps your bottleneck (and quoted tape / backup speed) is really the client disk, or network between client and media server. That would explain why you'd see the almost exact same performance from 2 quite different backup destinations.
Principal Learning Consultant with Symantec Education Course Development
e-mail alerts can be setup via Master Servers Host Properties and installing blat on the Master, or better still use OpsCenter (it is free!)
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
Can OpsCenter be installed on the Master server or do you suggest installing on a seperate server, maybe a VM?
Installation of OpsCenter server software on a NetBackup master server or
media server is possible if you want to monitor only one master server. An
example is the master server on which the OpsCenter server software is
installed.
To monitor more than one master server, Symantec recommends that you
install the OpsCenter server software on a separate standalone server.
OpsCenter Server and Agent are supported in a VMware virtual machine guest
operating system environment.
will restore -- where there is a Will there is a way
I usually advise my customers to have a dedicated server for OpsCenter - purely due to disk space and processor usage - I dont like to add anything to a Master that doesn't need to be there
How are the tests going?
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
I tested again over the weekend and the job took around 4hrs to complete after bumping up the disk buffer to 64. There may have been other factors that contributed to the 4hr backup time instead of the previously achieved 3hrs. I will test again tonight with the buffer set at 64.
Please go back to Yasuhisa's post and test client's read speed from disk.
If client has inefficient disk/lun layout or suffers from bad fragmentation, any network or buffer tuning will have limited effect.
Another way to cut down on backup time is to implement multistreaming to enable simultaneous backup of 2 - 4 streams from the client, effectively cutting backup time in half (or more).
Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links
Mark’s initial disk tuning recommendation provided the desired results; the backup went from 7 hours down to 3 hours, reducing the time by more than half. I will use the suggestions provided by the other members as well to further tune this job.
The UNIX client has more than one raid array on which the data resides, is it safe to send multiple streams to the disk storage unit which is one large volume in a RAID 5 setup? Should I set up a stream for each RAID array on the Client?
If you have more than one RAID then you should be fine to use a stream from each RAID set - as long as the backbone of the server and the network can handle more throughput then it may give you even better results
Glad things are working out for you
Authorised Symantec Consultant
Don't forget to give a "Thumbs Up" or mark as "Solution" if someones advice has helped you.
Update: After configuring multiple streams I was able to reduce the time it takes to backup this client by another 30 mins, bringing the total backup time to 2 1/2 hours.
In summary by using a combination of disk tuning and multi streaming I was able to significantly reduce my backup time on this AIX client. Thank you all for your expertise.
Would you like to reply?
Login or Register to post your comment.