Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

BE 2014 won't backup VMs on Hyper-V 2012 R2

Created: 18 Aug 2014 | 20 comments

Hi all.

The Backup Server is running Server 2008 R2 Standard with Backup Exec 2014 installed. All updates have been applied from live update.

I'm trying to backup VMs on a server running Hyper-V 2012 R2, but I continue to receive a "Final error: 0xa0009679 - A failure occurred querying the Writer status for a Hyper-V virtual machine" error. Running VSS Admin List results in a "Retryable Error" for the Microsoft Hyper-V VSS Writer. I have tried restarting the server already and that didn't help. Any tips?

Event Viewer also gives me this at the time of the backup failure:

1) Application and Service Logs > Microsoft > Windows > Hyper_V-VMMS > Admin:

Failed to fix-up absolute VHD paths in configuration of virtual machine 'VM Name': Account restrictions are preventing this user from signing in. For example: blank passwords aren't allowed, sign-in times are limited, or a policy restriction has been enforced. (0x8007052F). (Virtual machine ID AEB2B426-05DD-4C4F-AC8F-330B0D57B8D4)

2) Windows Logs > Application:

A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error.  If the backup process is retried,
the error may not reoccur.
. Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer.

 

Other details: The Hyper-V 2012 R2 OS has been separated from the VMs on the HD. OS is on C and the VMs are on D. VM backups continue to run perfectly fine from a Hyper-V 2008 R2 server. All servers mentioned are using the same storage device.

Any help will be much appreciated. Thanks everyone.

Operating Systems:

Comments 20 CommentsJump to latest comment

VJware's picture

Is the backup set to use the Microsoft VSS provider in the Advanced Open File options ?

If the Hyper-V host is running any AV, do add exclusions for the BE processes.

Is this a stand-alone host or a clustered one ?

Lastly, if the failure occurs on backup of specific VMs, try reinstalling the Hyper-V integration tools inside of that VM.

it_jccap's picture

The Advanced Open File option is set to "Automatic". Should I try changing to System or Hardware?

The Hyper-V host is not running any AV.

The host is stand-alone, and the only specific VMs that are affected are the ones on the Hyper-V 2012 R2 server. The Hyper-V 2008 R2 VM backups are running perfectly fine.

VJware's picture

See if this helps - http://www.symantec.com/business/support/index?page=content&id=TECH216944

Any errors in the event viewer of the VMs itself ?

And you can try setting the Advanced Open File option to "System".

PascalB's picture

If I remember, If you want to do a GRT backup of a windows 2012 R2 VM, you must to have a backup exec 2014 installed on a windows 2012 r2 server.

The OS of the backup server must be newer than the OS of the VM.

I had a lot of problems when I backup Win 2012 r2 VM on a ESX and my backup server is on win 2008r2.

it generate NTFS error 55 in the system events.

after a inplace upgrade to Windows server 2012 r2 and a repair of backup exec 2014, it works well.

There are informations in a symantec TID

it_jccap's picture

Do you have a link to confirm this information? I'm having a hard time finding anything.

it_jccap's picture

I'm having a hard time enabling the BERemote logging, but I can confirm that we are getting the same exact Event Viewer errors that are listed in that article.

The only errors in the Event Viewer of one of the VMs is this:

Volume Shadow Copy Service error: Unexpected error calling routine ConvertStringSidToSid(S-1-5-21-332934923-1340491104-1234779376-3418.bak).  hr = 0x80070539, The security ID structure is invalid.
.

Operation:
   OnIdentify event
   Gathering Writer Data

Context:
   Execution Context: Shadow Copy Optimization Writer
   Writer Class Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Name: Shadow Copy Optimization Writer
   Writer Instance ID: {a155e180-8e78-483d-9e4a-1fc5d030a5d3}

I'm going to investigate the Hyper-V integration services now.

PascalB's picture

in this TID : http://www.symantec.com/business/support/index?page=content&id=TECH214822

you can read :

The Backup Exec Media Server Operating System should be at least the same version as the Operating System of the VM being backed up via GRT to avoid this alert being created.

in another TID, if you want to do a backup GRT of an AD on a windows server 2012, you need to have a media server on a windows server 2012.

veritas support recommand me to upgrade the OS of the media server to Win server 2012r2

it doesn't solve all bugs

we can't do a GRT backup of an AD using a grt backup of the VM, we must to do a backup of the server using the windows agent

we have lot of errors on diff backups of a sharepoint 2010 on a real server.

 

JonathanK's picture

I doubt that running BE from a 2012 R2 host will make any difference for the problem you are experiencing, as I am doing that also with BE 2014, and it has never worked.  I've had a case open with Symantec since July 30th, and after a number of calls, emails, log files, etc., it was escalated to advanced tech support.  We worked on it for a few hours today, and he is going to involve some additional support on his end, so it might be a few days before I hear back.

Anyway, I have the exact same details you list in #1 and #2.  My client has 2 identical-hardware servers both running Server 2012 R2 as the host, with 2003 R2, 2008 R2, and 2012 R2 virtual machines between the 2 hosts, none of which will back up when selecting them as entire virtual machines to backup.  There is a matching 10178 error (your #1) for each machine, and with your indication of no issues with a 2008 R2 host, this leaves the 2012 R2 host as the common denominator.

From Server 2008 on, you cannot actually uninstall the integration services.  You can attempt it, with Setup/ uninstall via the integration iso, but unless you've done something really weird in getting that virtual machine on that host, it's just going to tell you that it's up to date.  If you upgrade a host server, you can update the integration services, but that should be fairly automatic, with inserting the integration disk being the most you might have to do to initiate it.

BE 2014 will run fine on Server 2008 R2, but, as previously mentioned, you cannot do GRT backups of anything above that level of OS.

The Symantec tech did grab this link to add in to my support case, so hopefully they'll check this link now and then for any updates that might help solve this.  I will try to remember to post back if and when we have a solution.

it_jccap's picture

Thanks for post, man. Glad to hear we aren't the only ones having this problem.

We have 3 Hyper Visors set up. The VM backups didn't start failing until we upgraded two of the Hyper Visors to Hyper-V 2012 R2. The VM jobs on the last Hyper-V 2008 R2 machine still run fine, so I think it's safe to say that 2012 R2 is the problem.

I'd really appreciate it if you updated when you get the chance. I'm playing with the Integration Services on the VMs now but I'm not making much headway.

SBESSEatCGLOBAL's picture

Hi all.

I have just installed a brand new server with Hyper-V 2012 R2 with 4 VMs all running 2008 R2.

I have launched my first complete backup.

I am getting errors :

event id 10172 : VSS writers inside virtual machine 'XXXXXXXXX' failed to perform BackupComplete to its shadow copy (VSS snapshot) set: Catastrophic failure (0x8000FFFF). (Virtual machine ID 0E06A47D-69A6-44A1-B130-C0D13912DC3A).

Problems with VSS / Hyper-V exist in Backup Exec for years : I am really wondering if competition has so many issues ...

Best regards.

TheSMB

SBESSEatCGLOBAL's picture

OK, I have tried the following changes :

- force the Advance Open File method to "System" and activate "Checkpoint recovery" (translation from my language, do not know the name in English)

- enter my VMs and force the creation of the first snapshot (since we do not use them).

The backup appears to run (several Gb parsed).

Wait & See

it_jccap's picture

Okay, we are making progress now.  Today, I attempted to install Integration Services on 2008 R2 Guest VMs that are hosted on the Hyper-V 2012 R2 Hypervisor.  I was getting an error when attempting to insert the CD.  I have in the past realized that the new 2012 R2 hypervisors have some legacy GPO applied that breaks account access rights when transferring/managing VMs.  At times, a domain admin cannot even access the Hyper-V service because of permissions issues.  GPUPDATE appears to fix this for us, but we do have work cut out in trying to track down what is causing this problem.

In any case, after the GPUPDATE, I attempted to insert Integration Services CD.  I found that I still could not complete, so I researched.  The solution was to access the server via AD properties, go to Delegation tab, enable delegation for all authentication methods.  Add the hypervisor when prompted and select the CIFS type.  This seemed to open communication between the VM and the Hyper-V host.  I updated integration services for one Server 2008 R2 VM and started the VM job.  It has been running successfully.

However, integration services for 2012 R2 VMs are not required, and I am wondering why the jobs failed when both the hypervisor and VM were essentially the same operating system.  It seems to boil down to permissions issues.  I am also enabling every single Integration service listed for my VMs, including Guest services.  I will let you know of any progress.

-A.

JonathanK's picture

it_jccap, please check on something if you would.  I had one virtual machine on each host that did not have a SCSI Controller in its settings.  After adding this to any virtual machine that did not have one, even if nothing was using the SCSI Controller, that did appear (so far) to get rid of the 10178 errors (your #1).  The backup is running, and getting most of the virtual machines so far.  It's still running, as I had to deal with some disk space issues on the target NAS device, so we'll see what the final outcome is later today.

So, my question is, did you before, and do you now, have a SCSI Controller in the settings for all of your virtual machines being backed up?  Odd that it previously threw an error on all of the virtual machines, even the ones that already had a SCSI Controller, but so far this seems to help.

I still received at least one error similar to your #2:  A VSS writer has rejected an event with error 0x800423f4, The writer experienced a non-transient error. If the backup process is retried,the error is likely to reoccur.

Slightly different number, and now it says it's likely to reoccur, so I'm still digging into that issue, but at least I have some progress also.

it_jccap's picture

JonathanK,

At this time, we do not have any iSCSI Controllers installed for the VMs.  I also applied the CIFS settings to another problematic hypervisor, and a backup of one of our virtual domain controllers appears to be working correctly.

It seems like it will only be a matter of time until this account issue occurs again.  But I think that is our main problem at this point.

Also, I think updating Integration Services seemed to help the issue a lot.

JonathanK's picture

Not iSCSI, the SCSI Controllers that are an option in each virtual machine's settings.  If it's a Gen 2 virtual machine, a SCSI Controller is the only controller option.

it_jccap's picture

Oops, but no, we do not have any SCSI controllers enabled for the VMs.

JonathanK's picture

Some progress, but some original errors still remain.  I did still get a 10178 error (your #1) before it was all done, and some virtual machines backed up, but others did not.

It complained about the Integration Services on one machine, and I ran them again, but it just says they are up to date.  I tried running them on a couple other VMs and I get the same result.

I'll let it run again tonight and see if anything changes tomorrow.

it_jccap's picture

I have made some extensive progress with this.  The VM jobs are running successfully.  It does warn me about GRT not being available, but that at this point is blinded by the fact VMs are actually being backed up.  What a relief.

I still need to restart my 2008 R2 servers that have updated Integration Services.  I also enabled Guest features, which I am not sure if that helped or not.

 

-A.

nickrich2412's picture

Hi,

Did anyone get any further with this? I've had the same problems as described, host is 2012R2 3 VM's all 2012R2, get the VSS errors, if i restart the server i can maybe 2 complete backups then it fails again, all very frustrating. SP! for BE2014 made no difference.

Previously BE have told me this is a microsoft problem and not BE.

MAkes BE completely worthless however!!!

Cheers for any help

Nick

JonathanK's picture

Yes, I finally got everything working about a week ago after working with Symantec Tech Support for well over a month.

I had already checked or addressed all of the issues that are fairly well documented, including these, before starting with tech support, but the GRT-enabled backups still did not work:

  • Integration Services up to date with host (my VMs were built on the host)
  • No duplicate drive IDs (an issue on some of my Server 2008 R2 VMs)
  • Add a SCSI Controller to the VM, even if nothing will be using it

What we finally found was touched on in a reference in a post above: http://www.symantec.com/business/support/index?page=content&id=TECH216944

See Issue #2, and run that 3-line PS script as Admin.  Run it from the host server, and put each VM name that is giving you issues in the first line.  I was getting the {No Contact} issue on my Exchange VM that I was attempting to do a GRT-enabled backup on, and also on another server that had SQL Server on it, but I was not attempting a GRT-enabled backup on that server(I purposedly did not have the SQL Server GRT option on for that host, but BE still recognized that there was a GRT-enabled application on the VM).  Both VMs would throw exceptions in BE every day, with the Exchange VM being my biggest concern.

The causes listed at the beginning of Issue #2 were not relevant, and none of the suggested issues at the end were relevant.  In my case, rebooting the VM actually causes the issue, contrary to the suggestion of rebooting the VM to solve it.

Re-start the Hyper-V Volume Shadow Copy Requestor service on each server reporting {No Contact}, and it should change to {OK}.  I did that, and the BE Exceptions on both VMs went away, and I verified that I have a full GRT-enabled backup of my Exchange VM.

I just verified last night that rebooting puts these two VMs back in {No Contact} mode and the BE Exceptions return.  I attempted (just now) to set the Hyper-V Volume Shadow Copy Requestor service to Delayed Start, but that's not allowed for that service (apparently because it's part of a group), so I'm not sure of an automated was to resolve the issue after a reboot.  That does also indicate that the issue really is an OS issue, as it shouldn't be switching to {No Contact} on boot.  FYI, both of my VMs with this issue are Windows Server 2008 R2 running on Windows Server 2012 R2 hosts (two different hosts).

Hope that helps.

Jonathan