BE2012 - Duplicate from DeDup Device is VERY slow
Hopefully someone can help me - I'm running out of ideas:
Problem:
We're using DeDuplication for the firststage-backup, afterwards the data is copied to external media - this time USB-disks (USB3 with USB3-controller)
The first stage is running fine, but the copy to usb is awfully slow - and gets slower every day.
It started with about 1400 MB/min, now it's down to 150-250 Mb/min.
To have a comparison to a "normal" duplicate to usb-job, I've created a b2d-device on the same storage as the dedup-foleder, made a backup and duplicated that to usb-disk - everthing identical, except the b2d-device used as a source instead of the dedup-device.
The result is about 6.000-7.500 MB/min - so the diskchannel and the usb-disk-channel is fine.
I don't have a problem with the slower speed from the dedup-device (due to the rehydrating) - as long it stays above 1000 MB/min, but it mustn't drop that far.
Environment:
BE2012 with actual updates. The hardware is a HP DL180G6, CPU: E5606, 12 GB RAM. 5 SAS 1 TB Harddisks attached to an HP SmartArray P410 with 256 MB Cache and battery. CPU usage below 50%, RAM-usage ~ 50%.
Any idea ?
Thx
Olaf
Comments 18 Comments • Jump to latest comment
Unless you're duplicating to another dedupe folder, you should actually be rehydrating your data...this is 1 reason for slower speeds. Add in a USB drive and you would have the right environment for slow duplication rates.
The server hardware itself looks to be fine, with no changes to be made!
Thanks!
If you find this is a solution, please mark it as such.
The absolut speed of the (beginning) dedup rate isn't the problem.
THe quesion is - why is it getting slower every day?
Without any visible bottleneck regarding disk, cpu or ram ?
Imho that's not the way it should be.
Olaf
Have you used Perfmon.exe to see whether or not there is a bottleneck on that USB drive?
Thanks!
If you find this is a solution, please mark it as such.
I'm pretty sure, that the usb-disk is not the botteneck due to 2 facts:
- Copy from B2D to USB works with ~ 7.000 MB/min
- Copy from DeDup to USB worked in the past with more than 1.500 MB/min.
I haven't used perfmon but the ressourcemonitor in the w2k8 task manager and couldn't see any problems.
Regards
Olaf
As a test what happens when you do a duplicate from your dedupe back to the dedupe folder?
Is that a GRT / non GRT enabled backup you are duplicating?
Is that a GRT / non GRT enabled backup you are duplicating?
...please edit your previous post if you post within minutes of the last one!
Thanks!
If you find this is a solution, please mark it as such.
You seem to read my mind!
Its a duplicate of a backup from a w2k8-server - raws based.
It includes Exchange-Data/mailboxes - so the answer is yes and no.
Does that make any difference ?
When a backupset is duplicated from a dedupe device to a non dedupe device, data is rehydrated i.e reconstructed. The metadata(catalogs) also has to be rebuilt. GRT backups are catalog intensive than a non GRT backup. To isolate the issue, I would suggest you to create a backup without GRT and test a duplicate backup. I assume there are no other operations running on the dedupe folder during the duplicate backup.
Hi all!
a few answers / comments to the suggestions made by you:
1. Yesterday (out of the blue sky) the dup-job ran with unbelievable 1700 MB/min - completly unexpected, because it was the same job / usb-disk / data that was duplicated.
The one and only difference was:
I started the duplicate-job manually with "run now" from the serverconsole itself - not via rdp.
So imho - there is no connection betwenn grt/non-grt data and the dup-speed.
2. Today - same dup job starts automatically at 9 am - and it's running with only 223 MB/min :-(
Changes since then:
I've installed last night the patch 201596 and restarted the server and today it's a different (identical type) usb-disc.
I'm gonna check if there is a difference when:
- The job is started manualy directly on the console without rdp (it shouldn't...)
- The disc from yesterday is used
New facts:
1. Starting the job manually from the serverconsole directly didn't change a thing - still veeeeeeeeery slow.
2. Using a brandnew empty freshly formatted usb-disc afterwords - even slower - 180 MB/min
3. Restarting the be-services before restarting the same job - juhu - speed is up to 1126 MB/min - still accelerating.
Some conclusions:
1. there seem's to be no connection between the usb-disc itself and the speed of the backup
2. the "BackupExec-setup" itself (hardware, software, configuration) seems to be ok, as the system is able to perform the duplicate-job with high-speed
Do you agree with ?
But why change a restart of the be-services speed up the duplicate job ??
Waiting for some ideas..
Olaf
...restarting the BE services is obviously releasing "something".
My suggestions:
1. Check that no AV is actively scanning the dedupe folder, and if so, put in an exclusion for that.
2. Check to see if there is a Known Issue in that section around this:
https://www-secure.symantec.com/connect/backup-and...
Thanks!
If you find this is a solution, please mark it as such.
Craig,
I've got the same impression - it's pretty obvious.
The AV check exclusion is already done - and not modified between the diferrent tests / results, so that couldn't be the reason for different behaviour.
I couldn't find a similar problem / error in the list.
Regards
Olaf
I've opened a case with symantec.
I'll keep this post updated.
Olaf
Hi Olaf,
In connection with my post on the link below, if you updated BE 2012 which caused the issues you now face, see if you can uninstall those patches and make sure you can operate as-per-normal. Once done, install the patches one-by-one to test...
https://www-secure.symantec.com/connect/forums/pro...
Thanks!
If you find this is a solution, please mark it as such.
Statusupdate:
Case is opened with Symantec TechSupp - they are working on it - already escalated to the next level.
Just a little update:
The case is opened for several weeks now, but Symantec is totally ignoring the facts and sticks to the statement "rehydrating makes duplicate from dedup-devices slow".
But the case is not closed yet.
Would you like to reply?
Login or Register to post your comment.