Video Screencast Help
Search Video Help Close Back
to help
New in the Rewards Catalog: Vouchers for "Symantec Technical Specialist" and "Symantec Certified Specialist" exams.

Very slow restore - Backup Exec 12.5 (Tape)

Created: 25 Sep 2009 | 20 comments
StevenH's picture
+1 1 Vote
Login to vote

Hi Guys,

We've been backing up using v12.5 for quite a while now and having only been at this company for 3 monthes this is the first time I have attempted to restore a mailbox.

We are doing full backups to tape using VSS and GRT even evening.

I am doing a restore of a mailbox from April and the total size of the backup is approximately 306gb with 1.4:1 compression... with a real size of 432.12gb.

Now this restore is running @ 204 MB/min for 25 hours!!! Obviously this is ridiculous as we have had to miss a backup rotation and looks very bad when a director is expecting some data from us.

My first question is - the byte count is currently at 328gb - will this have to go all the way up to 432gb (the real uncompressed size)

Secondly, - why so slow?? I have read lots that state we should be backing up to disk but the backup time of this was also astronomical so it seems to be a lose lose situation.

The staging area is on a SAN attached via a SCSI adaptor. Would this have been faster to do to the local disk?

Thirdly - are there and tips on improving B2D speeds?

Regards
Steve

Comments

CraigV's picture
25
Sep
2009
5 Votes +5
Login to vote

Hi Steven, Restoring to disk

Hi Steven,

Restoring to disk would depend on the size of the file/s, speed of the disks, RAID-type of the disks etc.
Is it a SAN, or DAS you're restoring too? A SAN would use fibre channel switches.
Besides that, I've always duplicated my Exchange backup to disk, and never had any speed issues.
If you don't know how to do that, you can read the article I wrote a few months ago here...:

https://www-secure.symantec.com/connect/articles/r...

As for speeding up the B2D, you can initiate more streams to disk, but that's about it. The rest would be dependant on some of the factors I wrote about above.
You're running at about 3.8MB/s which is very slow. What type of tape drive are you using?

If you find this is a solution, please mark it as such.

StevenH's picture
25
Sep
2009
2 Votes 0
Login to vote

Many thanks for the response

Many thanks for the response Craig.

It is a SAN we are restoring to.

I'll look into that article has suggested and report back as soon as possible with some results.

The tape drive for info is a HP Ultrium 3 SCSI..

Colin Weaver's picture
25
Sep
2009
3 Votes +3
Login to vote

When restoreing from tape a

When restoreing from tape a mailbox from a GRT enabled backup - this will always be slower than a restore from a B2D/ (IMG) location as the data from an IMG is taken straight from the IMG to your mail server, BUT in order to do a GRT restore from tape - Backup Exec  restores the complete information store to a temporary/staging folder (look in the Exchange Properties of the Restore job to see or change the location) and once the information store data has been recovered in full to the temporary folder it then extracts the mailbox information to restore it to your mail server.

I would guess there is something about the access to the SAN or the read from tape that is making it slow. If it is the SAN then using a local disk would speed it up, if it is the tape access then it would not make any change - and you might not want to stop the current restore to change location only to find there is no difference.

BTW knowing what the Backup Job rate was might be interesting as a comparison with your restore rate.

Can't really comment on your B2D speed - although there have been some issues with Incremental backups where the catalog process takes an extended time for the later incremental jobs (particularly on large busy information stores) I believe the current workaround for this is only enable GRT on the full backup template and disable GRT on the incremental jobs (but make sure your Exchange Deleted Item retention period is longer than the interval between your full backups)

It might be worth running the B2D test tool against your storage location to see if it identifies anything obvious.
http://entsupport.symantec.com/docs/321584
(links for downloads at bottom)

 

StevenH's picture
25
Sep
2009
2 Votes +2
Login to vote

I think the solution for us

I think the solution for us will be to move towards B2D as soon as we can get to the root of the actual backup speed problem. That tool could prove useful - i'll give that a go when we get some free time.

The job rate for the current restore is: 204 MB/min (3.4 MB/sec)

The job rate for our average backup to tape is: 1,966 MB/min (32 MB/sec)

Quite a large difference it seems! - although to be fair they seem very slow in the first place. The Ultrium 3 spec seems to suggest it will run at 80 MB/sec uncompressed... a 50mb drop for compressed data seems crazy.

Is the way a tape device restores data that different to the way it backs it up?

We need to look at exactly what configuration the SAN we are going to be using as a test with the B2D and i'll come back with more information regarding that.

Many thanks for the response(s).

Colin Weaver's picture
25
Sep
2009
2 Votes +2
Login to vote

Hmm might be worth (once your

Hmm might be worth (once your current restore had ended)  running a test to see how long it takes to duplicate one of your Exchange jobs on tape back to a B2D location - this will give you a comparison between the Restore from tape, the duplicate from taoe and the original backup to tape - gut feeling is there is something odd about the large difference in speed between your backup and your restore.

CraigV's picture
25
Sep
2009
5 Votes +5
Login to vote

Steven: With a SAN in place,

Steven: With a SAN in place, and if your Exchange server is attached to it, I'd look at Backup Exec's SAN SSO option. It's going to backup at the speed of the SAN (also assuming your tape drive is SAN-attached), and will restore at those speeds too (again dependant on your disks). So restoring to an HP EVA for instance is going to be far quicker than restoring to a server disk RAID set.
It is a licensed option though, but well worth it if you can. I used this to great effect, dropping my backup times down about 60%.

If you find this is a solution, please mark it as such.

StevenH's picture
06
Nov
2009
1 Vote +1
Login to vote

Right guys, Thanks for the

Right guys,

Thanks for the responses above.

I`ve now got the following scenario running;

Full exchange IS is B2D overnight (900gb) which runs at approx 1,400 MB/min which is good.

At the same time I backup our SAN to our tape autoloader which also runs at about 1,500-1700 MB/min which is great. This finishes about the same time as the B2D.

Once both are complete, the exchange IS backup is duplicated to tape.

Now i've tested a restore from the tape that the exchange IS is duplicated on and the restores are still running at 200MB/min... any ideas? This is on a new LTO4 drive using LTO4 media. I've also tried to duplicate a backup from tape to disk... but this duplication is still running at 200 MB/min. Maybe wrongly I thought that if you duplicated to disk... staging was not required.

I can't keep the backups on disk @ 800gb a pop !!! I only have 1tb of SAN available for this particular backup server.

Regards

Steve

CraigV's picture
06
Nov
2009
4 Votes +4
Login to vote

Doesn't matter if you

Doesn't matter if you duplicate to disk, or restore from tape. It's still going to do a stage of the data. Duplicating to disk would mean your restore sits on a disk allowing you to restore at will. If the restore fails on a tape, you'd have to start it all over again and run for hours once more. Duplication takes care of this.
Those speeds are really slow, so take a look at these 2 articles from the Symantec site (which I have put into my favourites =) ) which detail reasons why backup speeds differ, and how to improve them.

http://seer.entsupport.symantec.com/docs/231488.htm

http://seer.entsupport.symantec.com/docs/285756.htm

Are your drivers the  latest drivers for Backup Exec and for your SCSI card?

If you find this is a solution, please mark it as such.

StevenH's picture
06
Nov
2009
1 Vote +1
Login to vote

Thanks for the responses

Thanks for the responses Craig - you've been helpful in a few threads i've had.

Are you saying the speeds for the B2D and the speeds for backing up to tape are slow or just the speeds to restore?

I ran liveupdate on backup exec earlier and it picked up 10 hotfixes, however these did nothing to improve speeds. (However one thing i've thought is that I didn't reboot or restart services after this... just restarted backup exec itself)

The tape autoloader is a Quantum and it uses a SAS card... i'm running the drivers from the provided CD (not downloaded them) - perhaps I could update these. For the actual autoloader i'm using the drivers that Backup Exec installed....... should I be using the ones from the device provider, would this make a difference?

Again many thanks for the quick response.

I'll check those links out and report back.

Khaled Mashaykh's picture
29
Dec
2009
1 Vote +1
Login to vote

Yes, use the ones from the device provider

"For the actual autoloader i'm using the drivers that Backup Exec installed....... should I be using the ones from the device provider, would this make a difference?"

I had same problem but with HP tapelibrary and I solved it after rollback the tape drive driver from Symantec to HP.

Best regards,
Khaled Mashaykh
PRO Technology - Jordan

PRO Technology - Jordan

StevenH's picture
07
Nov
2009
0 Votes 0
Login to vote

Despite upgrading the

Despite upgrading the drivers, installing all hotfixes... upgrading the firmware on the tape device the damn restore from tape still only runs at about 220-230MB/min.

CraigV's picture
08
Nov
2009
1 Vote +1
Login to vote

Hi Steven, By upgrading the

Hi Steven,

By upgrading the drivers, are you talking about using Symantec's drivers? That's always best.

Are you restoring to another server? Have you tried the B2D tool that Colin mentioned, and how did that go? Have you tried restoring to the local server?

Lastly...tried a total reboot of the backup environment? Down the autoloader, down your server; start up your autoloader, start up your server?

If you find this is a solution, please mark it as such.

StevenH's picture
10
Nov
2009
0 Votes 0
Login to vote

I used the device drivers

I used the device drivers from the manufacturer (however I was using symantec drivers previous)

The restore speed is still pretty dire - 220 MB/min maximum - going to spend some time running some tests.

One very quick question and apologies if I should be starting a new thread;

I've got 3 B2D folders; - Exchange Full Exchange SG 1/2, Exchange SG 2/4

Now I ideally want Backup Exec to;

- Delete Exchange Full B2D data once the duplication to tape has completed
- Delete SG 1/2 prior to that backup starting
- Delete SG 3/4 prior to that backup starting

Can this be scheduled? I've read in to overwrite, etc but the only information I can find is to set the media to overwrite but only for one folder....!? That's fine, it will overwrite the particular target folder, but what if the other folders are also using up requred space?

CraigV's picture
10
Nov
2009
1 Vote +1
Login to vote

Hi Steven, Only other option

Hi Steven,

Only other option would be to upgrade the drivers for your SAS card.

Cheers,

Craig

If you find this is a solution, please mark it as such.

JustinO's picture
15
Dec
2009
0 Votes 0
Login to vote

Staging With Backup Exec 12.5

Restore from tape of a Exchange volume is yes extremely slow. I get about 220 to 250 MB per Minute on the Staging process when I do about once a month for testing integrity of my duplicates. This is from 800/1600 Ultrium 4 HP Branded external on direct attach 4GB Fiber. The drives it's staging to are SAS local raid 5 10k rpm drives, Hp Branded. The hardware is not the bottle neck. Its something to do with Backup Exec.

Upgrading your hardware is not the option it seams.
Changing how you run your backups completely is the answer.
D2D2T is the answer. Disk to Disk to Tape
You run backups to Disk, then duplicate to Tape.
Of course having the disk arrays to do this is what you need. Band-aiding the issue is what you seem to be asking for help on.
If you always have a month of backups on disk including exchange fulls and diffs restoring exchange emails takes seconds.
Tape would then only be used under very extreme issues. (Because of the slow staging time required)

If anyone has a real solution to this issues please give us a write up. If you have never completed an Exchange Restore and never actually found the fix for this, there's not much point in answering with things like upgrade your drivers/sas/harddrive/tape drive ect. Anybody that can help us all with a verifiable solution would be awesome!!!! Thank you to anybody out there can help us! :) woot

Justin

CraigV's picture
16
Dec
2009
0 Votes 0
Login to vote

I'll do a staging to disk of

I'll do a staging to disk of 1 of our Exchange DBs and post the speeds here.
It's going from an MSL6060 connected to 2 SAN Switch 4/16s, and the disk is presented from an HP StorageWorks EVA 6100...
Will let you know how it performs.

PS: I've actually restored on a number of occasions using the disk staging option, as well as a disaster recovery of Exchange from a locally attached autoloader...staging a 50GB Information Store took just under an hour

If you find this is a solution, please mark it as such.

teiva-boy's picture
17
Dec
2009
0 Votes 0
Login to vote

 BTW, HP does make tools to

 BTW, HP does make tools to test your tape drive speeds and disk speeds.  It could help isolate the problem area from media server to HBA to FC fabric to Tape...  Could be as simple as a zoning issue...

Symantec also makes a tool to validate your disk performance. b2dtool.exe I think.


There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."

gdassieu's picture
10
Jun
2010
0 Votes 0
Login to vote

Exact same problem here

Hello,

I'm having the exact same issue in BE2010. I have an open case with Symantec, but so far no cream. Could you find a solution to your problem?

Basically, I have a 256Gb VMDK.

If I restore the VMDK to the local BE media server (using VMware redirection) E:\ drive, it restores the whole VMDK in less than 2 hours at an average speed of more than 2.2Gb / min.

However, if I restore a file from inside that same VMDK using GRT (using the same E:\ drive as staging location), it takes 8 hours to finish! As I check the speed, it begins restoring the VMDK at 3Gb/min, but this logarithmically decreases to 200Mb/s near the end.

There's clearly a problem with BE here. The same VMDK restored to the same local drive has completely different performance depending on whether we are doing a VMDK restore or a GRT restore!

Any help would be appreciated, since Symantec support has not even been able to reproduce the issue at their labs even after two months of exchanges.

Regards,

Gaston

gdassieu's picture
10
Jun
2010
0 Votes 0
Login to vote

I have attached an excel file

I have attached an excel file comparing speed graphs for VMDK restore and GRT restore jobs for 256Gb and 100Gb VMDKs (I used a "timed screenshot" program to capture status every 15 minutes).

AttachmentSize
be_restore_perf.xls 48.5 KB
pkh's picture
10
Jun
2010
0 Votes 0
Login to vote

@gdassieu - This is an old

@gdassieu - This is an old discussion.  Do start a new one for your problem.