Video Screencast Help

Backup Exec 2012 error- Remote Agent crash during VMware backup

Created: 18 Dec 2012 | 15 comments

When I backup VM using Backup Exec, the following errors was found in Event Viewer. The Backup Exec Remote Agent service was stopped.

Event ID 1000

 

Faulting application name: beremote.exe, version: 14.0.1798.0, time stamp: 0x4f1e02d7
Faulting module name: vddkWrapper.dll, version: 14.0.1798.111, time stamp: 0x50be7e05
Exception code: 0xc0000005
Fault offset: 0x0000000000019f7a
Faulting process id: 0x1704
Faulting application start time: 0x01cddd446eb13e76
Faulting application path: C:\Program Files\Symantec\Backup Exec\beremote.exe
Faulting module path: C:\Program Files\Symantec\Backup Exec\vddkWrapper.dll
Report Id: fac63bb8-4942-11e2-8cdb-001b7892bb88
 
I found one KB about Backup Exec 2010 Remote Agent crashes during VMware backups.
http://www.symantec.com/business/support//index?page=content&pmv=print&impressions=&viewlocale=&id=TECH200972
 
Can I follow the above KB to solve Backup Exec 2012?
 

 

Comments 15 CommentsJump to latest comment

Backup_Exec's picture

Hi

 

Yes you can try the keys mention in the article only you may expereience a bit performance issue

Thanks

 

Sameer

Don't forget to give a "Thumbs Up" or Mark as "Solution" if someones advice has helped you.

Nitin_P's picture

Hi steven_man

This issue is resolved in the latest hotfix no : 199866.

Please run a live update on BE media server and update respective remote agents as well. Please reboot if notified [ generally it shouldnt request for a reboot ] .

 

Ref : http://www.symantec.com/docs/TECH199866

 

After this hotfix installation and the updates , please disable following keys if available. 

HKey_Local_Machine\Software\Symantec\Backup Exec for Windows\Backup Exec\Engine\VMware Agent 

 

Key1  : Disable NTFS Used Sector Tracking 

Key 2 : Disable NTFS Used Sector Tracking Dynamic Disk Support

Please change the Value to 1 .

Note : Do not create keys if not available , disable only those which are available.

 

 

 
steven_man's picture

Nitin_P,

I read the URL you mention. But I cannot find the change about registry.

http://www.symantec.com/docs/TECH199866

 

Can I provide the document about this?

 

Copy to Backup_Exec,

After I install hotfix 199866, do I need to change the registry accordin to the following URL.

Set the following registry keys to a value of "0"

  1. HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Backup Exec for Windows\Backup Exec\Engine\Vmware Agent\Enable Buffered Reads
  2. HKEY_LOCAL_MACHIEN\SOFTWARE\Symantec\Backup Exec for Windows\Backup Exec\Engine\Vmware Agent\Enable Buffered Writes

 

http://www.symantec.com/business/support//index?pa...

 

 

Nitin_P's picture

 

steven_man , 
 
I have referred the document you provided.
 
This change in the registry was involved only as a work around. But Now , since the hotfix is released , you can enable the keys back again. However , the keys mentioned below might conflict. Therefore , please disable the keys mentioned below : 
 
HKey_Local_Machine\Software\Symantec\Backup Exec for Windows\Backup Exec\Engine\VMware Agent 
 
 
 
Key1  : Disable NTFS Used Sector Tracking 
 
Key 2 : Disable NTFS Used Sector Tracking Dynamic Disk Support
 
Please change the Value to 1 .
 
Note : Do not create keys if not available , disable only those which are available.
 
 
 
 
Nabil polska's picture
  1. Stop the Backup Exec services on the Backup Exec sever
  2. Set the following registry keys to a value of "0"
    1. HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Backup Exec for Windows\Backup Exec\Engine\Vmware Agent\Enable Buffered Reads
    2. HKEY_LOCAL_MACHIEN\SOFTWARE\Symantec\Backup Exec for Windows\Backup Exec\Engine\Vmware Agent\Enable Buffered Writes
  3. Start the Backup Exec Services

 

Nitin_P's picture

Nabil polska , 

 

Since Steven already have installed the hotfix. " Buffer read / writes "  are not required to be disabled , they can be enabled back again using the value 1. 

 

Note : Disabling buffer read / writes reduces performance of VMWARE backups. Therefore we need to enable these keys back again .

 

However , due to this Hotfix installation , following keys needs to be disabled - 

 

HKey_Local_Machine\Software\Symantec\Backup Exec for Windows\Backup Exec\Engine\VMware Agent 
 
 
Key1  : Disable NTFS Used Sector Tracking 
 
Key 2 : Disable NTFS Used Sector Tracking Dynamic Disk Support
 
Please change the Value to 1 .
 
Note : Do not create keys if not available , disable only those which are available.
 
Steven , please provide us the update after disabling these keys.
steven_man's picture

In our environment, I have 3 Backup Exec 2012 Server (Windows 2008 R2 with SP1).  And I ran Symantec LiveUpdate to install all hotfixs for Backup Exec 2012.

One Backup Exec 2012 (Server 1) encounter the issue of Remote Agent service stopped when backup up VMware VM. Other two Backup Exec servers (Server 2 and Server 3) are normal.

After change the following registry, the service still failed when backup VM for a while (about 2 hrs).

 

Key_Local_Machine\Software\Symantec\Backup Exec for Windows\Backup Exec\Engine\VMware Agent 
 
Key1  : Disable NTFS Used Sector Tracking 
 
Key 2 : Disable NTFS Used Sector Tracking Dynamic Disk Support
 
Please change the Value to 1 .
 
When I run Window Update, I found that Server 1 have not installed any Window hotfixs. After I install all hotfix from Windows Update, the Remote Agent service seem become normal. 
 
Do anyone know any Symantec document mention about Microsoft hotfix is neccessr for Backup Exec 2012?
 
Also, after I modifed the two registry above, the backup time and backup size is larger than before in Server 2 and Server 3. Is it normal?
 
And I found that the backup size of incremental backup in backup job often larger than the backup size of Full backup in backup job. So strange! The incremental backup size should be <=  The Full backup size.
Can anyone know explain what the backup size means in backup log?
Nitin_P's picture

Steven_man, 

While taking backup of VMs , if VMWARE API's have any issues to collect change block tracking information , it will backup the entire VMDK and not just the used space.  This may lead to find increase in the time and the size of backup

To get this corrected, we need to reset Change Block Tracking [ CBT ] for VMs.

Please refer : http://www.symantec.com/docs/TECH197311

 

steven_man's picture

Nitin_P,

 

 

Key_Local_Machine\Software\Symantec\Backup Exec for Windows\Backup Exec\Engine\VMware Agent 
 
Key1  : Disable NTFS Used Sector Tracking 
 
Key 2 : Disable NTFS Used Sector Tracking Dynamic Disk Support
 
Please change the Value to 1 .
 
Do I still need to reset Change Block Tracking [ CBT ] for VMs if I disable to use CBT in registry above?
 

 

Nitin_P's picture

Hi Steven_Man ,

Yes , we need do do this. The issue that we have about backup size increase , it should be taken care due to reset operation for CBT.

Please update me about the progress after resetting CBT for VM.

 

Colin Weaver's picture

Please BE aware

"NTFS Used Sector tracking" and VMWare "Change Block Tracking" (CBT) are two completely different concepts/processes. Disabling the used sector tracking reverts the system to using CBT, this however can still affect performance and amount of data backed up.

Also we are aware of a condition where the latest hotfixes for both BE 2010 R3 and BE 2012 can still see problems with the stability of the beremote.exe process during VMware backups unless either the buffers are disabled or the used sector tracking is disabled. These new, post hotfix, conditions are currently being investigated.

 

 

 

Nitin_P's picture

Hi Colin Weaver,

So , in this situation , will it be ok for us to go back to the position to disable buffer reads / writers for now  , it will stop crashing beremote.

 

steven_man's picture

VM version of some VMs in my environment are 4, some are version 7 and 8. After upgrade some VMs, the issue of Remote Agent service seem to be solved after upgrade of VM to version 8. The service never stop after upgrade some VMs to version 8.

Because my environment have many VMs, I need some time to upgrade VM. Keep monitoring the backup result.

steven_man's picture

After upgrade all VMs to version 8, the issue of remote agent service does not happen again.

 

No change on the following registry.

 

 

HKey_Local_Machine\Software\Symantec\Backup Exec for Windows\Backup Exec\Engine\VMware Agent 

 

Key1  : Disable NTFS Used Sector Tracking 

Key 2 : Disable NTFS Used Sector Tracking Dynamic Disk Support

The value is still 0.

steven_man's picture

But the followng three errors sometimes found.

 

V-79-57344-38725 - One or more virtual disks contain volumes that do not support Granular Recovery Technology (GRT). Granular restores cannot be performed for these volumes.

 

V-79-57344-38724 - Directory

 C:\XXX does not support Granular Recovery Technology (GRT).  Granular restores cannot be performed for items in this directory.

 

 

The job failed with the following error: Incremental\\Differential backup methods are not supported for this VM. The backup job failed, since the option 'to fall back to a full backup' was not selected. 

 

Why?