Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

NetBackup Appliance 5220 Slow Rehydration to tape

Created: 14 Nov 2012 | 6 comments

Hi Everyone,

I know there have been lots of comments surrounding slow rehydration to tape from the appliances but mine is just to slow and the appliance can't keep up with the demand.

Master server = Windows 2008 R2 SP1 NetBackup 7.5.0.3

Appliances 5220 = NetBackup 7.5.0.4 (patch 2.5)

Tape drives linked to appliance at 4GB over fibre ibm 3592 drives capable of over 100mb a second native without compression over 200mb a second when compressing.

We have an SLP to backup our Exchange servers to a 5220 appliance then replicated to another 5220 on a different site but same netbackup domain and then rehydrate to tape from there. We have 12 tape drives avalaible but currently the storage unit is set to allow 4 concurrent write drives for the duplication to tape. The exchange servers are the passive nodes and we are using client side dedupe on them as its faster than using the appliance dedupe as there are 6 servers totaling in about 13TB of data.

All buffers are set to appliance defaults at 256kb (262144) and using 30 buffers. we currently use this on our master server catalog backup and get over 120MBps

Currently tape speeds on rehydration are around 15 -25MBps topping out across all 4 streams at 75-85MBps. This speed is appauling when you have 13TB a night to get to tape.

I have tried running a single stream to tape and it still can only produce around 50-60MBps

Has anyone had any luck improving this?

Comments 6 CommentsJump to latest comment

teiva-boy's picture

This has been my experience as well.  It hasn't changed with 6.5 and PDDO to 7.5 MSDP today.

My workaround...  Have customers tapeout less often.  I have policies set anywhere from 30-60days of on disk retention, and I have them tape out bi-weekly to monthly, as well as an SLP replicated copy.

Since the backups are on disk in two different locations, the customers see the value in using less tape over all, and don't worry too much how long it takes to get there.  In one case the monthly export takes 5 days.  But thats okay for their requirements.

The only thing quicker than this is to use an OST from a 3rd party.  They are significantly faster and more predictable.  But it'll cost ya much more than the Symc appliances.

There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."

f.buirey's picture

Hi,

Have you tried to change tthe ReadBufferSize in contentrouter.cfg file ?

Release Notes of Netbackup 7.5 recommends to set this parameter to 1048576 (1M)

I'm used to set NUMBER_DATA_BUFFERS to 64 rather than 30

regards

 

Chad_Wansing's picture

I always bump up the data buffers to well beyond the factory configs, although you should assess RAM utilization on your particular box. As a general rule, 30MB/s is usually on the low end of what I see. For a lot of the customers I deal with I generally see somewhere in the 70-100MB/s as "normal" although even then they may not achieve those speeds on every dataset. I see lows of 30MB/s and highs of 150-180MB/s with what I would call "normal good throughput" in the 70-100MB/s range.

Sometimes due to the nature of certain kinds of data we do tend to see lower rates of rehydration. In particular lots of small files can result in lower average rehydration throughputs. I would be interested if you think this could be relevent here?

I'm not at all a fan of everyday rehydration from deduplicated disk pools. It's generally just not practical, regardless of the dedupe vendor those backups are living on. We can help that situation through the architecture of the NetBackup environment. There are a couple different ways we can approach this:

1) Maintain all "operational recovery" copies of data on deduplicated storage - First and foremost it's important to look at exactly why you might be making tape every day. In some cases perhaps it's a legal or compliance requirement. Could this possibly be addressed by doing replication to another site instead of creating tapes? Even if we can reduce tape to weekly or monthly, that generally gets the environment enough time to make tape creation a more practical option.

2) Perform inline tape copy (ITC)- Yes, that's right, inline tape copy still lives. And you can, in fact, leverage ITC as part of Storage Lifecycle Policies (SLP's). Instead of doing a backup followed by a duplication, you can configure two backup operations as part of the SLP in which case the data stream will get split in the back-end buffers of the media server and send one stream to the tape device and one stream to your dedupe pool. The great news is that these operations don't change your ability to multiplex to the tape devices since it is a standard backup operation. The downside is that this negates the ability to use client-side deduplication and both backup streams are slowed to the rate of the slowest stream, but with the parallelization of the operations, many times this can be a great approach.

3) Leverage some form of disk staging - Also available is the option to stage the data to fully-hydrated disk storage in order to get your backup done and then rolling that data off to disk and/or dedupe storage. The downside here is that, like above, client side dedupe can't be used, but also since it's a duplication operation there's no multiplexing to the tapes. The good side is that these operations are very fast as there's no rehydration needs and using SLP's you can expire the data from the hydrated disk storage after the duplication operations complete.

If you have any additional questions, feel free to message me!

 

-Chad

follow me on twitter @chadwansing_sym to get alerts on patch releases and breaking NetBackup Appliance news

Even's picture

So then, if you have some good numbers to start up with to share.....

please do :-)

Even Hornfelt.

As we all are part of the product, we're in here to improve it.

Mark_Solutions's picture

If these are GRT Exchange backups it will be the cataloging of the GRT data that slows them down when going to tape

Either disable the GRT part when duplicating to tape or try reducing you fragment size of the de-dupe storage unit to 5000 - have found that makes a big difference

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Chad_Wansing's picture

So this is my own general guidelines and not anything that's been run through Symantec QA, but when I do 5220 POC's, I generally configure the standard 262144 buffer size and will assign anywhere from 300 to 1000 data buffers.  Yes, I understand this is a crazy number, but the rationale I use is to match the net buffer size to whatever you're using on your clients (which, by the way is where I generally see the biggest impact in tuning any environment) and multiply that value by the number of simultaneous streams you would have.  Then subtract that from your total RAM amount to get what **should** be left.  take about 90% of that and divide it by your data buffer size to get about how many data buffers you **could** be using without adversely impacting your system. 

Truthfully most PS guys I know always cringe when I tell them how I tune (admittedly I'm a "crank it up until it smokes then back it down a notch" kind of guy) so this probably isn't the best way for every environment, but it has worked out well for me in the POC's I've done (about 70 or so at this point).  Some of my customers have gotten better performance at lower settings and that just proves out the generally accepted knowledge that every environment is different so find the settings that work best for you.

 

-Chad

follow me on twitter @chadwansing_sym to get alerts on patch releases and breaking NetBackup Appliance news