Very slow restore - Backup Exec 12.5 (Tape)

StevenH's picture

Hi Guys,

We've been backing up using v12.5 for quite a while now and having only been at this company for 3 monthes this is the first time I have attempted to restore a mailbox.

We are doing full backups to tape using VSS and GRT even evening.

I am doing a restore of a mailbox from April and the total size of the backup is approximately 306gb with 1.4:1 compression... with a real size of 432.12gb.

Now this restore is running @ 204 MB/min for 25 hours!!! Obviously this is ridiculous as we have had to miss a backup rotation and looks very bad when a director is expecting some data from us.

My first question is - the byte count is currently at 328gb - will this have to go all the way up to 432gb (the real uncompressed size)

Secondly, - why so slow?? I have read lots that state we should be backing up to disk but the backup time of this was also astronomical so it seems to be a lose lose situation.

The staging area is on a SAN attached via a SCSI adaptor. Would this have been faster to do to the local disk?

Thirdly - are there and tips on improving B2D speeds?

Regards
Steve

CraigV's picture

Hi Steven, Restoring to disk

Hi Steven,

Restoring to disk would depend on the size of the file/s, speed of the disks, RAID-type of the disks etc.
Is it a SAN, or DAS you're restoring too? A SAN would use fibre channel switches.
Besides that, I've always duplicated my Exchange backup to disk, and never had any speed issues.
If you don't know how to do that, you can read the article I wrote a few months ago here...:

https://www-secure.symantec.com/connect/articles/r...

As for speeding up the B2D, you can initiate more streams to disk, but that's about it. The rest would be dependant on some of the factors I wrote about above.
You're running at about 3.8MB/s which is very slow. What type of tape drive are you using?

If you find this is a solution, please mark it as such.

StevenH's picture

Many thanks for the response

Many thanks for the response Craig.

It is a SAN we are restoring to.

I'll look into that article has suggested and report back as soon as possible with some results.

The tape drive for info is a HP Ultrium 3 SCSI..

Colin Weaver's picture

When restoreing from tape a

When restoreing from tape a mailbox from a GRT enabled backup - this will always be slower than a restore from a B2D/ (IMG) location as the data from an IMG is taken straight from the IMG to your mail server, BUT in order to do a GRT restore from tape - Backup Exec  restores the complete information store to a temporary/staging folder (look in the Exchange Properties of the Restore job to see or change the location) and once the information store data has been recovered in full to the temporary folder it then extracts the mailbox information to restore it to your mail server.

I would guess there is something about the access to the SAN or the read from tape that is making it slow. If it is the SAN then using a local disk would speed it up, if it is the tape access then it would not make any change - and you might not want to stop the current restore to change location only to find there is no difference.

BTW knowing what the Backup Job rate was might be interesting as a comparison with your restore rate.

Can't really comment on your B2D speed - although there have been some issues with Incremental backups where the catalog process takes an extended time for the later incremental jobs (particularly on large busy information stores) I believe the current workaround for this is only enable GRT on the full backup template and disable GRT on the incremental jobs (but make sure your Exchange Deleted Item retention period is longer than the interval between your full backups)

It might be worth running the B2D test tool against your storage location to see if it identifies anything obvious.
http://entsupport.symantec.com/docs/321584
(links for downloads at bottom)

 

StevenH's picture

I think the solution for us

I think the solution for us will be to move towards B2D as soon as we can get to the root of the actual backup speed problem. That tool could prove useful - i'll give that a go when we get some free time.

The job rate for the current restore is: 204 MB/min (3.4 MB/sec)

The job rate for our average backup to tape is: 1,966 MB/min (32 MB/sec)

Quite a large difference it seems! - although to be fair they seem very slow in the first place. The Ultrium 3 spec seems to suggest it will run at 80 MB/sec uncompressed... a 50mb drop for compressed data seems crazy.

Is the way a tape device restores data that different to the way it backs it up?

We need to look at exactly what configuration the SAN we are going to be using as a test with the B2D and i'll come back with more information regarding that.

Many thanks for the response(s).

Colin Weaver's picture

Hmm might be worth (once your

Hmm might be worth (once your current restore had ended)  running a test to see how long it takes to duplicate one of your Exchange jobs on tape back to a B2D location - this will give you a comparison between the Restore from tape, the duplicate from taoe and the original backup to tape - gut feeling is there is something odd about the large difference in speed between your backup and your restore.

CraigV's picture

Steven: With a SAN in place,

Steven: With a SAN in place, and if your Exchange server is attached to it, I'd look at Backup Exec's SAN SSO option. It's going to backup at the speed of the SAN (also assuming your tape drive is SAN-attached), and will restore at those speeds too (again dependant on your disks). So restoring to an HP EVA for instance is going to be far quicker than restoring to a server disk RAID set.
It is a licensed option though, but well worth it if you can. I used this to great effect, dropping my backup times down about 60%.

If you find this is a solution, please mark it as such.

StevenH's picture

Right guys, Thanks for the

Right guys,

Thanks for the responses above.

I`ve now got the following scenario running;

Full exchange IS is B2D overnight (900gb) which runs at approx 1,400 MB/min which is good.

At the same time I backup our SAN to our tape autoloader which also runs at about 1,500-1700 MB/min which is great. This finishes about the same time as the B2D.

Once both are complete, the exchange IS backup is duplicated to tape.

Now i've tested a restore from the tape that the exchange IS is duplicated on and the restores are still running at 200MB/min... any ideas? This is on a new LTO4 drive using LTO4 media. I've also tried to duplicate a backup from tape to disk... but this duplication is still running at 200 MB/min. Maybe wrongly I thought that if you duplicated to disk... staging was not required.

I can't keep the backups on disk @ 800gb a pop !!! I only have 1tb of SAN available for this particular backup server.

Regards

Steve

CraigV's picture

Doesn't matter if you

Doesn't matter if you duplicate to disk, or restore from tape. It's still going to do a stage of the data. Duplicating to disk would mean your restore sits on a disk allowing you to restore at will. If the restore fails on a tape, you'd have to start it all over again and run for hours once more. Duplication takes care of this.
Those speeds are really slow, so take a look at these 2 articles from the Symantec site (which I have put into my favourites =) ) which detail reasons why backup speeds differ, and how to improve them.

http://seer.entsupport.symantec.com/docs/231488.htm

http://seer.entsupport.symantec.com/docs/285756.htm

Are your drivers the  latest drivers for Backup Exec and for your SCSI card?

If you find this is a solution, please mark it as such.

StevenH's picture

Thanks for the responses

Thanks for the responses Craig - you've been helpful in a few threads i've had.

Are you saying the speeds for the B2D and the speeds for backing up to tape are slow or just the speeds to restore?

I ran liveupdate on backup exec earlier and it picked up 10 hotfixes, however these did nothing to improve speeds. (However one thing i've thought is that I didn't reboot or restart services after this... just restarted backup exec itself)

The tape autoloader is a Quantum and it uses a SAS card... i'm running the drivers from the provided CD (not downloaded them) - perhaps I could update these. For the actual autoloader i'm using the drivers that Backup Exec installed....... should I be using the ones from the device provider, would this make a difference?

Again many thanks for the quick response.

I'll check those links out and report back.

StevenH's picture

Despite upgrading the

Despite upgrading the drivers, installing all hotfixes... upgrading the firmware on the tape device the damn restore from tape still only runs at about 220-230MB/min.

CraigV's picture

Hi Steven, By upgrading the

Hi Steven,

By upgrading the drivers, are you talking about using Symantec's drivers? That's always best.

Are you restoring to another server? Have you tried the B2D tool that Colin mentioned, and how did that go? Have you tried restoring to the local server?

Lastly...tried a total reboot of the backup environment? Down the autoloader, down your server; start up your autoloader, start up your server?

If you find this is a solution, please mark it as such.

StevenH's picture

I used the device drivers

I used the device drivers from the manufacturer (however I was using symantec drivers previous)

The restore speed is still pretty dire - 220 MB/min maximum - going to spend some time running some tests.

One very quick question and apologies if I should be starting a new thread;

I've got 3 B2D folders; - Exchange Full Exchange SG 1/2, Exchange SG 2/4

Now I ideally want Backup Exec to;

- Delete Exchange Full B2D data once the duplication to tape has completed
- Delete SG 1/2 prior to that backup starting
- Delete SG 3/4 prior to that backup starting

Can this be scheduled? I've read in to overwrite, etc but the only information I can find is to set the media to overwrite but only for one folder....!? That's fine, it will overwrite the particular target folder, but what if the other folders are also using up requred space?

CraigV's picture

Hi Steven, Only other option

Hi Steven,

Only other option would be to upgrade the drivers for your SAS card.

Cheers,

Craig

If you find this is a solution, please mark it as such.