Video Screencast Help

Backup Exec hangs while backing up VMware VMs (snapshot)

Created: 27 Aug 2012 • Updated: 06 Sep 2012 | 15 comments
This issue has been solved. See solution.

We are having this issue for weeks/months now : while processing a backup-to-disk job using Backup Exec 2012 SP1a VMware infrastructure agent the backup process hangs at some point. We are running a VMware 5 (Build 768111) environment with a fibre channel SAN which is being backed up by a Windows 2008 R2 BE2012 SP1a (Verion 14.0 Rev. 1798 64 Bit) VM.

At some point in the backup process - and different VMs - I can see that there is no progress anymore for hours or days in my vSphere client (percentage of the job freezes) and in BE the MB/min indicator decreases and no changes to the number of backed up Bytes. This is only when using backup-to-disk, which actually is a (VMFS 5) partition within the BE virtual machine (drive letter V: size 1,5TB). Last time it stopped at 95% of our backend Exchange 2010 server (running, approx. 300GB), last time before it stopped while backing up a powered off Windows Server 2003 test system (approx. 30GB).

The only thing to do is hit cancel at the BE machine (which actually wouldn't cancel or stop the job), then reboot the BE machine, cancel the VMware job in vSphere client and then "delete" the VMware snapshot to return to merge the changes on the disks.

Gladly we back up all data (non-VMs) separately directly to tape (VMware Direct I/O connected SCSI LTO juke box), so I have at least my data at hand, but I don't understand why even a snapshot of a powered off system halts the Symantec VMware job. Snapshots manually initiated in vSphere client have not been an issue yet, disk space is available, too. Have not found any similar problem desciption in Google...

Any help really appreaciated!

Comments 15 CommentsJump to latest comment

CraigV's picture

Hi,

 

Are you using AOFO in your job?

If so, turn it off and see what the outcome is. If not, turn it on and run the backup again.

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

Bembel's picture

Indeed our B2D jobs had AOFO enabled by default, which I deactivated yesterday before starting a test backup. This test backup finished with a lot of error messages, mostly GRT and VSS provider related.

For GRT, which is not needed in our full VM B2D configuration, I simply deactivated the setting for the whole job. Concerning the VSS provider issue I uninstalled VMware VSS providers on all VMs which had BE agents installed to only use Symantec's VSS provider for GRT enabled backups (non-B2D). Waiting for the reboot of these VMs tonight and will start another B2D test backup tomorrow morning.

What I don't get here is that I thought I would not need a VSS provider within each VM since I backup the whole machine from outside via vCenter server, so a VMware snapshot of the machine will be created and backed up against...

Will keep you guys updated!

Cheers,

Bembel

jlax1124's picture

We are seeing the same problem backing up exchange databases. Randomly one of them will just hang - I've just let one run for 3 days to see if it would ever end - it doesn't.

We are BE 2012 Sp1a, on Server 2008 R2 x64, Exchange is 2007.

Our exchange is on a dedicated system (not VM'ed at this stage), and we are backing up to a Riverbed Whitewater device using backup to disk.

This was working flawlessly until we went to 2012.

CraigV's picture

...just for information: that Riverbed device isn't listed on the BE 2012 HCL. As such, getting it to work doesn't mean you'll get any support unfortunately.

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

jlax1124's picture

So is that Symantec's answer? Despite the original post with the same symptoms?

CraigV's picture

1. I am not a Symantec employee so I don't know, and I am, not privileged to know either.

2. If your hardware isn't on the HCL, it means it hasn't been tested. It might work, but if you have a support call and your unsupported hardware comes up, chances aren't good you will get support. Or it will be very limited.

Either way, refer to the BE SCL/HCL for further information. Or download & run the b2dtest.exe application against the B2D drive and see if it is supported or shows any errors.

Thanks!

Alternative ways to access Backup Exec Technical Support:

https://www-secure.symantec.com/connect/blogs/alte...

PB Fastener IT's picture

We are currently having the EXACT same issue. Attempting to turn of AOFO settings and running backup overnight. Will report back as soon as tomorrow.

samueljn's picture

Hi,

I would like to share some more info

  • Does it seems to happen specifically with Exchange 2010?
  • GRT is not supported or Exchange if DAG is used using VMware agent 
  • If it happens specifically in Exchange VM, disabled application GRT for exchange in the job
  • Create a seperate for Exchange to Back it using the Remote agent for windows.

If it is not specific to exchange:

  • Is BE server and Vcenter server also VM's?
  • Is both the ESX and Vcenter are version 5.0?
  • What is the default transport mode used for Vmware Backup?

Regards,

John

PB Fastener IT's picture

Hey John,

Yes our BE is a VM server on its own seperate esxi 5.1 server.  attempting to backup other VMs on another esxi server yet always fails or hangs as stated above. Hotadd and NBD are only checked for transport mode. We do have a SAN yet nothing ever transfers when we attempted to backup before with SAN checked. Attempting another backup tonight with just Hotadd and NBD checked. Will report back tomorrow morning and if backup still fails or hangs , I will be calling in.

 

 

samueljn's picture

- If BE is on a Virtual machine, SAN transport mode is not possible

- BE 2012 supports Backing up only upto ESX 5.0 Update 1. Check the SCL: http://www.symantec.com/docs/TECH175581

Regards,

John

Bembel's picture

Yesterday I have changed settings on my jobs and had to wait until the VMs rebooted on which I uninstalled VMware's VSS provider. Meanwhile I used the time to upgrade our vCenter infrastructure to the most current level (5.0 Update 1b, Build 804277) which today seems to have broken all my VM backup jobs - in VMware event viewer I can see that when triggering a backup job this message appears:

Cannot login @<IP address of BE2012 server>

On my BE log I get this error message : V-79-57344-38209

Even after deleting the vCenter server in BE, adding it and successfully validating to every single VM and VM infrastructure folder the only machine I was able to backup is our VM template.

While 5.0 Update 1 is supported in Symantec's BE2012 SCL (http://www.symantec.com/docs/TECH175581) most probably 5.0 Update 1b (released 2012-08-16) is not.

Once I can back up our VMs again, I will test again.

unitofone's picture

I'm having this exacty same issue as well.

We are trying to backup two file servers, around 1.5TB in size as virtual machines to deduplicated storage with GRT.

Physical Backup Exec server with SAN Transport enabled.

Backup starts at around 3.5GB per minute but after about 15-20 minutes just seems to hang and the data rate doesn't change.
 

There doesn't appear to be any logging of what is or isn't happening during this either, I will turn off AOFO and see if this helps.

 

 

PB Fastener IT's picture

Hello Everyone.

I spoke with a Symantec Tech as well as a VMWARE tech and they both stated the free esxi versions do not support vstorage API license which is needed in order for any vmware backup to work at all. If you have a standard, enterprise or enterprise plus vmware software then vstorage comes with it and you may have to confirm with symantec why it is not working, BUT if you do have the free ESXi versions then this is why no VMWARE backup works and just hangs or fails.

regards,

PB Fasteners IT Support

Bembel's picture

Hi everyone,

finally everything seems to be OK now. For me the solution was

1.) uninstalling VMware VSS providers where BE VSS providers (agents) are installed,

2.) deactivating GRT within these backups (for me not needed anyway) and

3.) specifically choose Microsoft VSS provider under "Advanced Open File"

For 3.) I got this hint from the Symantec support technician who helped me with another problem; he basically said that for VM AVVI backup taking snapshots under "Advanced Open File" must be activated, but the default to automatically chosse the snapshot provider is not the best setting because this could result in the problems described in this thread. The best choice is "System - Microsoft VSS provider".

That was my choice yesterday and for the first time the job finished without any problem :-)

Hope this helps!

SOLUTION
JShetler's picture

We had a similar issue with our virutal environment and backups.  Thanks to this thread we finally got some working backups, thanks!  However, we need to use GRT.  To fix the GRT issue we made three registry changes as directed by Symantec support:

HKLM\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Engine\VMware Agent

Change "Disable NTFS Used Sector Tracking" from 0 to 1
Change "Disable NTFS Used Sector Tracking Dynamic Disk Support" from 0 to 1
Change "Disable NTFS Used Sector Tracking GPT Disk Support" from 0 to 1

When we asked what exactly this does differently, all we were told was that it "disables features that are no longer needed."

Afterwards our backups were able to run with GRT enabled, using AOFO and the Microsoft VSS provider.  Hopefully this will help save somebody the headache we've been dealing with.

Edit:  Forgot to specify, registry changes were made to our vCenter/Backup Exec server.