Video Screencast Help

BE2012 - Duplicate from DeDup Device is VERY slow

Created: 21 Feb 2013 • Updated: 13 Jun 2013 | 19 comments
OP's picture
This issue has been solved. See solution.

Hopefully someone can help me - I'm running out of ideas:

Problem:

We're using DeDuplication for the firststage-backup, afterwards the data is copied to external media - this time USB-disks (USB3 with USB3-controller)
The first stage is running fine, but the copy to usb is awfully slow - and gets slower every day.

It started with about 1400 MB/min, now it's down to 150-250 Mb/min.

To have a comparison to a "normal" duplicate to usb-job, I've created a b2d-device on the same storage as the dedup-foleder, made a backup and duplicated that to usb-disk - everthing identical, except the b2d-device used as a source instead of the dedup-device.

The result is about 6.000-7.500 MB/min - so the diskchannel and the usb-disk-channel is fine.

I don't have a problem with the slower speed from the dedup-device (due to the rehydrating) - as long it stays above 1000 MB/min, but it mustn't drop that far.

Environment:

BE2012 with actual updates. The hardware is a HP DL180G6, CPU: E5606, 12 GB RAM. 5 SAS 1 TB Harddisks attached to an HP SmartArray P410 with 256 MB Cache and battery. CPU usage below 50%, RAM-usage ~ 50%.

 

Any idea ?

Thx

Olaf

Comments 19 CommentsJump to latest comment

CraigV's picture

Unless you're duplicating to another dedupe folder, you should actually be rehydrating your data...this is 1 reason for slower speeds. Add in a USB drive and you would have the right environment for slow duplication rates.

The server hardware itself looks to be fine, with no changes to be made!

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

OP's picture

The absolut speed of the (beginning) dedup rate isn't the problem.

THe quesion is - why is it getting slower every day?

Without any visible bottleneck regarding disk, cpu or ram ?

Imho that's not the way it should be.

Olaf

CraigV's picture

Have you used Perfmon.exe to see whether or not there is a bottleneck on that USB drive?

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

OP's picture

I'm pretty sure, that the usb-disk is not the botteneck due to 2 facts:

- Copy from B2D to USB works with ~ 7.000 MB/min

- Copy from DeDup to USB worked in the past with more than 1.500 MB/min.

I haven't used perfmon but the ressourcemonitor in the w2k8 task manager and couldn't see any problems.

 

Regards

Olaf

AshutoshTamhankar's picture

As a test what happens when you do a duplicate from your dedupe back to the dedupe folder?

Is that a GRT / non GRT enabled backup you are duplicating?

AshutoshTamhankar's picture

Is that a GRT / non GRT enabled backup you are duplicating?

CraigV's picture

...please edit your previous post if you post within minutes of the last one!

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

OP's picture

Its a duplicate of a backup from a w2k8-server - raws based.

It includes Exchange-Data/mailboxes - so the answer is yes and no.

Does that make any difference ?

AshutoshTamhankar's picture

When a backupset is duplicated from a dedupe device to a non dedupe device, data is rehydrated i.e reconstructed. The metadata(catalogs) also has to be rebuilt. GRT backups are catalog intensive than a non GRT backup. To isolate the issue, I would suggest you to create a backup without GRT and test a duplicate backup. I assume there are no other operations running on the dedupe folder during the duplicate backup. 

OP's picture

Hi all!

a few answers / comments to the suggestions made by you:

1. Yesterday (out of the blue sky) the dup-job ran with unbelievable 1700 MB/min - completly unexpected, because it was the same job / usb-disk / data that was duplicated.

The one and only difference was:
I started the duplicate-job manually with "run now" from the serverconsole itself - not via rdp.

So imho - there is no connection betwenn grt/non-grt data and the dup-speed.

2. Today - same dup job starts automatically at 9 am - and it's running with only 223 MB/min :-(

Changes since then:
I've installed last night the patch 201596 and restarted the server and today it's a different (identical type) usb-disc.

I'm gonna check if there is a difference when:

- The job is started manualy directly on the console without rdp (it shouldn't...)
- The disc from yesterday is used

OP's picture

New facts:

1. Starting the job manually from the serverconsole directly didn't change a thing - still veeeeeeeeery slow.

2. Using a brandnew empty freshly formatted usb-disc afterwords - even slower - 180 MB/min

3. Restarting the be-services before restarting the same job - juhu - speed is up to 1126 MB/min - still accelerating.

Some conclusions:

1. there seem's to be no connection between the usb-disc itself and the speed of the backup

2. the "BackupExec-setup" itself (hardware, software, configuration) seems to be ok, as the system is able to perform the duplicate-job with high-speed

Do you agree with ?

But why change a restart of the be-services speed up the duplicate job ??

Waiting for some ideas..

Olaf

CraigV's picture

...restarting the BE services is obviously releasing "something".

My suggestions:

1. Check that no AV is actively scanning the dedupe folder, and if so, put in an exclusion for that.

2. Check to see if there is a Known Issue in that section around this:

https://www-secure.symantec.com/connect/backup-and...

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

OP's picture

Craig,

I've got the same impression - it's pretty obvious.

The AV check exclusion is already done - and not modified between the diferrent tests / results, so that couldn't be the reason for different behaviour.

I couldn't find a similar problem / error in the list.

Regards

Olaf

OP's picture

I've opened a case with symantec.

I'll keep this post updated.

Olaf

 

CraigV's picture

Hi Olaf,

 

In connection with my post on the link below, if you updated BE 2012 which caused the issues you now face, see if you can uninstall those patches and make sure you can operate as-per-normal. Once done, install the patches one-by-one to test...

https://www-secure.symantec.com/connect/forums/pro...

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

OP's picture

Statusupdate:

Case is opened with Symantec TechSupp - they are working on it - already escalated to the next level.

OP's picture

Just a little update:

The case is opened for several weeks now, but Symantec is totally ignoring the facts and sticks to the statement "rehydrating makes duplicate from dedup-devices slow".

But the case is not closed yet.

 

OP's picture

Hi folks!

I just wanted to update and close this case.

To make a very long story short:
According to the 1st, 2nd and 3rd level supportengineers from Symantec  we do not have a problem - the dedup works like designed and they can NOT do anything to speed up the process of dplicating FROM the dedup to any external non-dedup-device. :-((

Due to this sad result and the amount of time we've invested in this case we've decided to discard dedup as a primary backup target at our and all customer site's.

From my point of view the product can't keep the expectations from the customers and promisses made by symantec.

 

CU

Olaf

SOLUTION