Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

SSR 2013: Host-level Hyper-V backups in Win Server 2012 R2

Created: 11 Apr 2014 • Updated: 22 Jul 2014 | 26 comments

Hi!

I am experimenting with Symantec System Recovery 2013 and host-level Hyper-V VM backups in Windows Server 2012 R2.

My observations are as follows:

1. SSR is capable of doing hot Hyper-V backups. In this case Hyper-V creates intermediate snapshots (checkpoints / avhdx) during the backup that are auto-merged into the VM being backed up after the backup. Unfortunately, the snapshots stay in the iv2i backup files and I am not sure how to handle them after the restore of the VM from the backup.

2. I tried to run a pre-backup script in SSR that shuts down the VM in order to perform a cold backup of the VM. After the backup I run a post-backup script in SSR to restart the VM. This works ok so far but Hyper-V still creates the snaptshots (checkpoints / avhdx) I described above although the VM is shutdown at the time of backup. Is this behavior expected?

3. I tried to turn off the "backup volume snapshot" integration service for the particular VM through the Hyper-V Manager. According to the docs this should force Hyper-V to move the VM into saved state and perform a cold backup instead of a hot backup. Unfortunately, this does not change anything. The VM continues to run (instead of stopping) and the execution flow is the same as described in 1. Does SSR somehow ignore the Hyper-V settings?

Thanks in advance.

Anguel

Operating Systems:

Comments 26 CommentsJump to latest comment

Markus Koestler's picture

Hm, good questions - let me escalate this to support for you.

*** Please mark thread as solved if you consider this to have answered your question(s) ***

Anguel's picture

Thanks Markus,

I also noticed that after the restore the VM has the temporary AVHDX hard disks attached instead of the original VHDX disks, i.e. if my VM has a virtual harddisk that was test.vhdx after restore it is test_SOME_LONG_ID.avhdx (although the original test.vhdx is still there. I expect that this is probably due to the fact that during backup Hyper-V redirects its writes to that temporary avhdx and after backup SSR does not "fix" this.

BTW: I am also running SSR inside the VM to get granular file-level backups. But that should not be important. Or does SSR make use of it's own VSS manager instead of Microsoft's VSS manager?

Anguel

Markus Koestler's picture

At least under W2K3 it has its own VSS provider. But vssadmin list writers should show in the VM.

*** Please mark thread as solved if you consider this to have answered your question(s) ***

Chris Riley's picture

Anguel,

1. SSR is capable of doing hot Hyper-V backups. In this case Hyper-V creates intermediate snapshots (checkpoints / avhdx) during the backup that are auto-merged into the VM being backed up after the backup. Unfortunately, the snapshots stay in the iv2i backup files and I am not sure how to handle them after the restore of the VM from the backup.

I'm not sure I understand the problem here. What exactly is the problem you mention regarding the incremental (iv2i) backups?

Anguel's picture

My question is actually the following:

When I perform a host-level backup of Hyper-V (it does not matter if my VM is shutdown or running at the time of backup) in the backup archive I always get the original virtual disk, e.g. MYDISK.vhdx and an additional MYDISK_SOME_LONG_ID.avhdx temporary snapshot.

Probably this is all related to:

http://www.aidanfinn.com/?p=15759

So on restore the VM is actually pointing to that *.avhdx snapshot instead of the original *.vhdx. So how should I proceed? Probably I should I just discard the *.avhdx because it may contain inconsistent data? This would mean that I would have to manually make the VM to point back to the *.vhdx files instead of the *.avhdx files. I don't see any explanations in the SSR docs so customers have to guess...

Anguel

Chris Riley's picture

Let me check on this and get back to you.

I'm going to be out of office for a few days so may not be able to get back to you until sometime next week.

Chris Riley's picture

Sorry for the delay on this.

I have received a response on this and need to verify one of two things. I should be able to provide an update within a day or two.

Anguel's picture

Thank you in advance Chris!

BTW: I have now ordered 2 server licenses and 15 workstation licenses.

Best regards,

Anguel

Markus Koestler's picture

Anything still open here? If no please mark this post as solved!

*** Please mark thread as solved if you consider this to have answered your question(s) ***

Anguel's picture

Still open. I am still waiting for an answer from Chris (see above).

Anguel's picture

Still no comment on this from Symantec support? It is important to know how to deal with those hot backups performed on Hyper-V in Server 2012 R2. This is a supported configuration, so someone should know how this actually works. We need this information in order to be able to restore from these backups properly. I don't like to "guess" when I pay for a professional backup solution...

Chris Riley's picture

Anguel,

My apologies - this somehow slipped off my radar.

I do now have this confirmed with our engineering team and I hope the following information helps to answer your questions.

Here’s a summary of our Hyper-V support:

Backup of boot / system volume of Hyper-v host from DR perspective:

a)    In order to backup boot / system volumes of host, install SSR on the host and backup boot/system volumes like regular (non-hyper-v) system. Restore of boot/system volume using SRD will restore these volumes. However, the VMs on the Hyper-V having VHDs on data volumes may be out-of-sync post restore.

Backup of Hyper-V VMs:

a)   Backup VMs from within the guest and restore VMs using SRD (from within guest)

b)    It is also possible to backup multiple VMs by installing SSR on Hyper-V host itself and then take volume level backup of the host volumes. (Note: SSR doesn’t support incremental backup of CSV volumes though). When applying such method to protect VM’s VHD(x) files, SSR recommends that the customer back up the related drives in the same backup job. i.e. drives containing all the VHDs of a VM to be protected and its boot volume (typically C:\). In case if customer has VM’s VHD(x) files located across multiple volumes and he is not really sure which related drives to be backed up together, SSR recommends that he backup all volumes of the Hyper-V host machine.

Restore of VMs from this host level backup needs to be done using granular restore of VHDs for VM. This can be done by mounting v2i files and copying VHD and AVHD files of VM from these V2is. After copying VHD and AVHD files the end user needs to point his VM to the latest avhdx.

Anguel's picture

Chris,

Thank you for the information. Well, I do backup the complete hyper-v host including any volumes containing the VMs and their VHDX (all in one job), so b) should apply to my case.

However, I still wonder that your engineer tells me to finally point my VM to the latest AVHDX, as this is some temporary file that is used while the main VHDX is locked during backup. See here http://www.aidanfinn.com/?p=15759

"Writes are temporarily redirected to the checkpoint’s AVHDX/AHVD file(s).  This gives the backup requestor a clean & crash consistent copy of the virtual machine’s VHD or VHDX files that are safe to read.    After the backup, the checkpoint is merged and the job is done."

Update: I also noticed that the AVHDX file inside the SSR backup file (after the backup is complete) is smaller than the AVHDX file that is on my volume towards the end of the backup. This means that the AVHDX contains some partial / temporary data during the backup that is IMHO inconsistent. Could you please check this with your engineer? Thanks.

Anguel

Chris Riley's picture

From what I have been told ....

The AVHDX file inside the SSR backup is small because it contains only a few writes that Hyper-V writer made. While the AVHDX file on the main volume contains writes for the all I/O happening in live VM after check-pointing and hence will be much bigger in size.

robbieD's picture

Anguel,

I have not used SSR 2013 to protect Hyper-V VM's as I use Microsoft DPM for that.  I do use ssr to protect all of my servers though.  You could manually merge the AVHDX file with the VHDX file within the Hyper-V Manager after the restore.  I have had the need to do this [merge files] a few times over the years.  Check out this link to see how.  http://social.technet.microsoft.com/wiki/contents/articles/6257.manually-merge-avhd-to-vhd-in-hyper-v.aspx

Rob

Anguel's picture

robbieD wrote:

You could manually merge the AVHDX file with the VHDX file within the Hyper-V Manager after the restore.

Rob,

Thank you for your feedback. I know that merging the AVHDX with the VHDX files is possible. However, as described above as far as I understand the AVHDX is auto-created at the beginning of backup only to make sure that we have a consistent VHDX in the backup archive that does not contain some partially written file. Any writes during backup are therefore redirected to the temporary AVHDX which is finally auto-merged into the VHDX after the backup is finished. According to http://www.aidanfinn.com/?p=15759 this is a new technique used by Microsoft in Server 2012 R2 instead of the old VSS method.

Now the Symantec engineer Chris talked to proposes to point a VM restored from backup to its restored AVHDX which is IMHO not a good idea as it contains data that is written by the VM during backup and may be therefore inconsistent. Therefore, I think that it is more safe to point the restored VM to the restored VHDX and not to the AVHDX. Any feedback form Symantec, specifically regarding Windows Server 2012 R2, is welcome.

Anguel

Anguel's picture

Chris,

Thank you for the feedback. If I understand what you are describing the SSR snapshot contains some "point in time" of a file, and if this file grows towards the end of the backup those latest changes to the file are not reflected in the backup image anymore. This is expected as an image-based backup is not an instant operation but takes some time.

But the more important question remains: Are these AVHDX files really consistent so that we can point our VM after restore to them (as your engineer proposed) or should we instead point the VM to the main VHDX file (my propsal). IMHO the VHDX should be chosen, as it was "closed" at the beginning of backup and during the backup all writes are redirected to the temporary AVHDX, which IMHO is not considered consistent.

Anguel

Chris Riley's picture

Anguel,

Maybe this information will help. Again, this has come direct from our engineering team who have in turn got this information directly from a Microsoft trainer.

This is the workflow for VM backups on Windows 2012 R2 Hyper-V:

a) Initiate a snapshot on host OS. This will initiate a snapshot on guest OS.
b) During snapshot process in the guest a new provider (instead of volsnap) is engaged which manages to create a crash consistent copy of VM disks (VHDXs).
c) Now, this copy of VHDX is crash consistent but not application consistent. This is called check-pointing.
d) A new hot disk (AVHDX) gets added to the VM if the VM has SCSI controller. This AVHDX becomes child of VHDX in step b) above.
e) Before completion of snapshot process, Hyper-V writer needs to do some changes to the snapshot volume. These changes now get redirected to AVHDX.
f) Snapshot process now completes. The hot-added disk contains only a few redirected writes from Hyper-V writer to make it consistent.
g) So, the AVHDX + VHDX together makes an application consistent snapshot.
h) Now while the hot disk is added there to capture only the Hyper-V writer initiated writes, other writes happening inside the VM continue to occur after step # c above. These writes get written to another AVHDX.

Anguel's picture

Chris,

This information is very helpful and interesting. So it looks like during a backup there is one VHDX file (main virtual disk) plus two AVHDX files (hot disks). When I look into my SSR backup archives they really seem to contain three files per virtual disk, e.g.:

1.) DISK.vhdx

2.) DISK_88278437-AEF5-47BB873847-34376A45.avhdx

3.) DISK-AutoRecovery.avhdx

So let's assume I need to restore such a VM from am SSR backup, which of these virtal disk files should I then point my restored VM to (for an application consistent restore)? As far as I understand this should be probably the DISK_88278437-AEF5-47BB873847-34376A45.avhdx. Is this correct?

Anguel

Chris Riley's picture

I have confirmed that you need to point the VM to DISK-AutoRecovery.avhdx

Anguel's picture

Chris,

Thank you. The answer is very interesting although unexpected :) The AutoRecovery.avhdx is NOT the file the VM is pointing to by default after restore. I am very busy at the moment but I will test if this works as soon as I have time.

I believe that such answers actually belong in the docs of any good backup program. It should not only do the backup but a user should also know how exactly to restore a VM he backed up with SSR in case of data loss. Thank you very much for the support once again.

Regards,

Anguel

Chris Riley's picture

Yes, I agree with you - this is something that I plan to document in a public article.

Roland R.'s picture

Chris, Anguel

after reading and confusion I did many own tests with Windows 2012R2 and last Version of SSR2013.

I have found the following way.

Example:
Name of VM: TestVM
Rootpath of the VM E:\TestVM
Backup running VM

1. Delete the VM in Hyper-V-Manager i want to restore.
2. Delete the Subfolders of the VM-Path (E:\TestVM\Snapshots, E:\Virtual Hard Disks, E:\Virtual Machines)
3. Restore the Subfolders with Recoverypoint-Browser
   There are three files like anguel described

   Mount the recoverypoint as Volume (I:) and copy the permissions with
   robocopy I:\TestVM E:\TestVM /E /COPY:ATSOU /DCOPY:DAT

   This way, because the combination of Recoverypoint-Browser + robocopy was much faster then
   copy Data and Permissions only with robocopy

4. Import the restored VM with Hyper-V-Manager
5. Stop the Hyper-V-Manager
6. Start the Hyper-V-Manager

After 6. Start the Hyper-V-Manager the snapshots were automatically merged and I have only the .vhdx file and no .avhdx.

I this the right way and an application consistent restore?

Please comment, thanks

Anguel's picture

Hi Roland,

If I remember I did not try to explicitly delete and then reimport the VM through Hyper-V Manager but just stopped the Hyper-V Manager Service and then copied the restored VM over from the recovery point. I.e. I mounted the recovery point to e.g. X: and used something like

xcopy X:\VMs\WS2012R2ESS E:\VMs\WS2012R2ESS /O /X /E /H /K

But your robocopy solution also sounds nice and probably does the same. I still don't understand why SSR is not able to set the permissions automatically. I expect this from such a program.

Back to the main problem: It is nice to hear that importing the VM with Hyper-V Manager merges the VHDX snapshots. In my case this was not the case as I restored the VMs directly as described above and in my case the VM was then pointing to the temporary AVHDX files instead. Probably you have found the correct way to do an app consistent restore? I have spent a lot of time figuring out how SSR works with Hyper-V and now I have a productive system and don't have more time to spend until something breaks. Of course, I am very disapointed that Symantec is not able to document such things. It is their product, not ours and we are running a supported configuration. So probably Chris can give us some more feedback on your solution and I hope that someone at Symantec will finally sit down, reproduce our problem and tell us what is the official way to do an app consistent VM restore on Hyper-V. Everything else is just guesswork!

UPDATE: I have now unmarked the question. Reading Roland's posting I don't think it is solved completely.

Anguel