Parallel restore jobs of VMware virtual machines cause beremote service to crash and jobs to fail with error "Connection to the restore target has been lost."

Article:TECH141713  |  Created: 2010-10-12  |  Updated: 2013-03-24  |  Article URL http://www.symantec.com/docs/TECH141713
Article Type
Technical Solution


Issue



 

When running multiple restore jobs of an entire VMware virtual machines to the original datastore at the same time, the Backup Exec Remote Agent (beremote) service will crash on the Backup Exec server and the restore jobs will fail with error "Connection to the restore target has been lost.”


Currently this issue is seen with virtual machine restores being hosted on the same ESX server. 

Parallel restores have not been tested going to different ESX hosts.

 


Error



 

Final error: 0xe000fe30 - A communications failure has occurred.
V-79-57344-65072 - Connection to the restore target has been lost. Operation canceled.

Event Viewer Application log error:
Event Type:     Error
Event Source: Application Error
Event ID:          1000
Description:    Faulting application beremote.exe, version 13.0.4164.0, faulting module bedsvmvcb.dll, version 13.0.4164.0, fault address 0x0003dfd0.

Beremote debug on Backup Exec server:
[7292] 09/21/10 10:12:14 [fsys\vmvcb]         - SymVmTools: RetrieveVmInfo: OK
[7292] 09/21/10 10:12:14 [fsys\vmvcb]         - VM_VCBPROXY_FS::createSnapshotForRestore() specified Transport mode priority list: 'nbd:san:hotadd:nbdssl' for the vm '/vmmor/vm-1996'
[7292] 09/21/10 10:12:14 [fsys\vmvcb]         - SymVmTools: GetAllVmDiskDetails: vmPath=/vmmor/vm-1996
[7292] 09/21/10 10:12:14 [fsys\vmvcb]         - SymVmTools: RetrieveDcMorAndObjMorFromPath: OK
[7292] 09/21/10 10:12:14 [fsys\vmvcb]         - SymVmTools: GetAllVmDiskDetails: SYM_VMC_ERROR:  FAILED_TO_GET_MOR_BY_NAME
[7292] 09/21/10 10:12:14 [fsys\vmvcb]         - VM_VCBPROXY_FS::GetAllVmDiskDetails: failed, result = 0XE
[7292] 09/21/10 10:12:14 [fsys\vmvcb]         - VM_VCBPROXY_FS::createVM() Could not get the disk details for the the newly created VM 'VIRTUAL_MACHINE_NAME' CODE (E000958F)
[7292] 09/21/10 10:12:14 [fsys\vmvcb]         - VM_VCBPROXY_FS::DisconnectFromVCServer: vmcDisconnect session = 01A50988
[7292] 09/21/10 10:12:14 [fsys\vmvcb]         - VM_VCBPROXY_FS::DisconnectFromVCServer: status = 0X0
[7292] 09/21/10 10:12:14 [fsys\vmvcb]         - VM_VCBPROXY_FS::handleNonStagedRestoreMetaData() Could not create the VIRTUAL_MACHINE_NAME.
[7292] 09/21/10 10:12:14 [ndmp\loops]         - LP_ENV::MsgError: error 0xe000958f processing object VIRTUAL_MACHINE_NAME

Bengine debug on Backup Exec server:
[8172] 09/21/10 10:10:49 [server]             - Status for: 'VM test restore' updated
[2496] 09/21/10 10:12:17 [ndmp\ndmpcomm]      - ndmp_readit: Caught message on closed connection. Socket 0x618 len 0xffffffff
[2496] 09/21/10 10:12:17 [ndmp\ndmpcomm]      - ndmp_readit: ErrorCode :: 10054 : An existing connection was forcibly closed by the remote host.
[2496] 09/21/10 10:12:17 [loops]              - BaseBSDProcessor: NDMP control connection lost.
[2496] 09/21/10 10:12:17 [tpfmt]              - TF_NDMPGetResult(): MediaServer thread done, channel->dsError is 0x00002746, returning FS_COMM_FAILURE
[2496] 09/21/10 10:12:17 [loops]              - RARestoreBSDProcessor: MessagePumpAndWiatForResults() returned -536805635 (0xe000fefd)

 


Environment



ESX server


Solution



 

Workaround

 

Wait for VMware to create the virtual machine container before starting another restore:

1.    Start the first virtual machine restore job

2.    Log into vCenter console and wait for VMware to complete the creation of the container

3.    Start the next virtual machine restore job

 

or

 

Create a single restore job and select all the virtual machines needed to be restored

Note:  This will run each virtual machine restore one after another (sequentially), allowing VMware plenty of time to create the virtual container and restore the virtual machine before starting the next in the restore list.

 


Supplemental Materials

SourceError Code
Value0xe000fe30
Description

Final error: 0xe000fe30 - A communications failure has occurred.


SourceUMI
ValueV-79-57344-65072
Description

Connection to the restore target has been lost. Operation canceled.


SourceEvent ID
Value1000
Description

Faulting application beremote.exe, version 13.0.4164.0, faulting module bedsvmvcb.dll, version 13.0.4164.0, fault address 0x0003dfd0.



Article URL http://www.symantec.com/docs/TECH141713


Terms of use for this information are found in Legal Notices