Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Verify job is very slow in Backup Exec 2012 deduplication storage

Created: 24 Mar 2013 • Updated: 08 May 2013 | 10 comments
sean lee's picture
This issue has been solved. See solution.
Hello.
We have build up new Backup Exec 2012 environemnt. 1 Media server and 20 Remote agents.
We are using Deduplication storage with client-side deduplication enabled for main backup.
While backup performance is OK (finished within 8 Hours), verify takes much more time comparing with backup (more than 24 hours)
 
Backup Exec Media Server
- Windows 2008R2 Enterprise with SP2
- Xeon E5620 CPU
- 24GB Memory
- IBM DS3500 for deduplication storage
- All BE updates installed except last one (HOTFIX 201596) 
 
We configured vefiry job as separate schedule, but still it takes too much time (all weekend)
And I could monitor that while Verify jobs are running, if some backup job started, the backup job is also extremly slow. (less than 100MB/min)
Is there any idea that we can improve verify performance?
 
Operating Systems:

Comments 10 CommentsJump to latest comment

Gurvinder Rait's picture

•Disable the option Allow this job to have direct access to this device if you use client-side deduplication with a verify template.

Ensure the above checkbox is not selected for verify job

sean lee's picture

Thanks for reply.

The option "Enable direct access" already deselected for all Verify jobs before I post this question.

Any other suggestion?

CraigV's picture

...if you have jobs that run while the veriy is running you could end up with a slow verify along with your slow backups. If can stagger this, do so.

That said, make sure that no antivirus on your media server is actively scanning the dedupe folder (exclude it). Your hardware spec looks right too!

THanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

sean lee's picture

This slow verify jobs can be seen even there are only verify jobs (no backup).

The reason why I made separate verify job schedule is to minimize overlap with backup window.

For antivirus, I already exclude Deduplication Storage Folder from antivirus software.

By the way, when I see the document "Best practices for Backup Exec 2012 Deduplication Option", the only thing I didn't meet is "Disable RAID caching", is it possible RAID caching can be the cause of slow Verify performance?

Thanks

pkh's picture

"Disable RAID caching" is unlikely to be the cause of your problem.  However, you should still disable the caching because this affects the integrity of your backup if there is an outage of the media server and/or the dedup folder.

teiva-boy's picture

Battery backed cache. Its been around for more than a decade.

There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."

pkh's picture

They do fail at the wrong time.  I had them failed twice when they are needed.  Once, this resulted in a server rebuild.  Imagine restoring your dedup folder, that is if you had taken a backup of it.

OP's picture

Hi sean lee,

I've got the same problem with every single task that is "read related" from the dedup storage - in my case duplicating to external media like tape or usb-disks.
Do you have similar experiences?

How is the speed during a restore? As slow like during the verify process ?

Thx

Olaf

teiva-boy's picture

Verify from the dedupe store will ALWAYS be slow, there is no way around it. BE needs to read the store, reassemble the data/file to its original state, read the data, verify the data and move on to the next file. Its a slow process designed by Symantec. Not an issue with other competing products...

Frankly verify from tape is more important than from disk. Or verify only on the replicated copy.

There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."

sean lee's picture

Thanks for lots of information

After I update all NICs and FC HBAs device driver and firmware to most recent version, the extremely slow performance when read/write operation at same time is getting better. So I don't need to run separate verify job.

I've monitored backup performance over a month and found that the overall backup performance was 3000MB/min ~ 4000MB/min with 8 concurrent operations on deduplication storage.

By the way when I increased the number of concurrent operations to 16, the overall backup performance was less than 1000MB/min.

I've varied the vaule of concurrent operations and found that, in my case, the optimum setting is 8 concurrent   job and overall backup performance is 3000MB/min ~ 4000MB/min. (even in read/write operation like backup jobs and verify jobs are running at same time)

If I found a better way to increase deduplication performance, I'll post it.

To Olaf

In my case, duplication performance to LTO6 tape library is almost same with backup performance (30000MB/min ~ 4000MB/min).

And I will intensive restore test until next week, so I'll let you know what the restore performance from deduplication storage is.

Thanks

Sean Lee

SOLUTION