Deduplication restore: The data being read from the media is inconsistent.
Our deduplication storage area was filling up so I carried out the following process:
- Disabled all BE services - manually via a script
- Rebooted the BE media server (so BE never started up)
- Used robocopy with all the right switches (esp. security) to copy the E:\ drive to a temporary iSCSI drive on the SAN
- Ran it again just to be sure (and nothing extra copied)
- Re-built the E: drive array with more disks (4TB RAID-10 to 8TB RAID-10)
- Copied the data back using robocopy again (no errors)
- Re-enabled all the services and rebooted the server
- Waited until BE came back up - which it did
The full backups ran last night and two of them failed during the verify stage. Also, when I do a test restore, it fails at exactly the same point.
As far as I was aware, the above procedure should have worked :-( As far as BE is concerned, nothing really changed (i.e. same folder path, drive letter etc).
It's looks like one or more of the 64k blocks in the deduplication database is corrupt/missing, despite being very careful to copy all the data. I'm guessing the full backup does this:
BEGIN
FOR each file DO
FOR each 64k in block DO
Calculate hash
IF hash not in database THEN
Write new 64k block
ENDIF
NEXT block
NEXT file
Now the key bit about this is that the backup doesn't actually touch the 64k block if it thinks it's already backed up.
However, the verify/restore will try and read the 64k block which as there appears to be damaged data, causes a fault.
I've since come across this article which I wasn't aware of:
http://www.symantec.com/business/support/index?page=content&id=TECH160832
Does anyone know what this "Deduplication Storage Folder recreation process" actually does? Does it simply re-attach the deduplication folder or does it carry out some kind of integrity check?
Or is there a way to force BE to carry out an integretity check of the deduplication database and remove any references to bad blocks/files so that a full backup would re-create them?
Thanks, Rob.
Comments
@robnicholson - I wrote this
@robnicholson -
I wrote this doc: http://www.symantec.com/docs/TECH160832
There is a rigorious acl process that runs when adding the dedupe folder after moving it from one drive to another.
If the failures you are seeing are "read/write" errors after moving the folder, it could be related.
Remove and re-add the folder. Allow a few hours for the acl modification process to complete and everything should be fine.
If not, post your job log and I will attempt to assist.
Regards.
Randey
Using this process, when the
Using this process, when the moved dedup folder is added to BE, are the catalogs re-created or do I have to run a catalog job?
@pkh -Since no media records
@pkh -
Since no media records are removed from BE database, there should not be any reason to recatalog.
The worst that will happen is maybe an inventory. Other than that, this process is similar to removing and readding a simple b2d folder, with a dozen extra moving parts in the background ;)
Regards.
Randey
Thanks for this Randey. I'll
Thanks for this Randey. I'll give it a go.
Okay, going through the
Okay, going through the process and it's currently "Gathering information" and behind it says "Dedupe (SERVER015) - Undiscovered". So guess time to leave it for the rest of the day.
One suggestion for the tech note. Each remote system that has been prepared for backup using client side dedupe has an entry as shown in the attached screenshot. It's not entirely clear in the technote whether these will need re-creating afterwards, i.e. delete them and re-prepare the systems for remote dedupe.
Cheers, Rob.
The service restart takes
The service restart takes care of sending the credentials for the dedupe folder return connection to the remotes.
At that point, its up to the remotes to connect back.
Regards.
Randey
Would you like to reply?
Login or Register to post your comment.