Video Screencast Help
Protect Your POS Environment Against Retail Data Breaches. Learn More.

VMware hotadd restore fails with "NBU VMware policy restore error (2820)"

Created: 04 Sep 2013 | 7 comments

Hello,

I try to make a VM restore using the HOTADD transport mode, and it fails, in the way that I can see the VM in the datastore, but I can't start it. 

The Environment is the following:

1xVM - NetBackup (MasterServer,MediaServer,VMBackupHost) - 7.5.0.6

vCenter- V5.1

Transport mode for Backup&Restore: hotadd

In vCenter I see the message:

"An error was received from the ESX host while powering on VM ....

Cannot open the disk '/vmfs/volumes/...vmdk' or one of the snapshot disks it depends on. 
Failed to lock the file"
 
If I make VM Consolidate than I can start the VM. 
 
In the bprd.log I have found this
 
11:14:18.727 [25019] <2> bpcr_disconnect_rqst: bpcr received the following message:
11:14:18.736 [25019] <16> bprd.sfr: do_VMware_virtual_snapshot_delete: delete virtual machine snapshot failed on x.x.x with 156
11:14:18.736 [25019] <32> bprd.sfr: sfr: do_VMware_virtual_snapshot_delete: returned 156
11:14:18.736 [25019] <16> bprd.sfr: sfr: aborting entire restore:  error 2849
11:14:18.736 [25019] <4> bprd.sfr: sfr: Reached done, error: 2849
11:14:18.741 [25019] <2> restorefiles: Last attempt to restore from image = a.a.a_1378280791 failed (browse_client = a.a.a destination_client = x.x.x
requesting_client = x.x.x user = root): Restore failed due to failure to delete the virtual machine snapshot
11:14:18.741 [25019] <2> mail_msg_and_set_exit_status: entered; status = 0
11:14:18.741 [25019] <2> mail_msg_and_set_exit_status: client_type = 40
11:14:18.741 [25019] <2> mail_msg_and_set_exit_status: Attempting to send mail to root on a.a.a
11:14:18.742 [25019] <2> logconnections: BPCD CONNECT FROM i.i.i.i.59557 TO i.i.i.i.13782 fd = 6
11:14:18.748 [25019] <2> mail_msg_and_set_exit_status: CLIENT_CMD_SOCK from bpcr = 6
11:14:18.748 [25019] <2> mail_msg_and_set_exit_status: CLIENT_STAT_SOCK from bpcr = 16
11:14:18.748 [25019] <2> bpcr_get_version_rqst: bpcd version: 07500004
11:14:18.748 [25019] <2> bpcr_get_version_rqst: bpcd version: 07500004
11:14:18.755 [25019] <2> bpcr_get_version_rqst: bpcd version: 07500004
11:14:18.760 [25019] <2> mail_msg_and_set_exit_status: RESTORE EXIT STATUS = 2820
11:14:18.760 [25019] <2> logconnections: BPCD CONNECT FROM i.i.i.i.37736 TO i.i.i.i.13782 fd = 6
11:14:18.783 [25019] <2> job_monitoring_exex: ACK disconnect
11:14:18.783 [25019] <2> job_disconnect: Disconnected
11:14:18.784 [25019] <2> logconnections: BPCD CONNECT FROM i.i.i.i.56430 TO i.i.i.i.13782 fd = 5
11:14:18.790 [25019] <2> terminate_restore: EXIT STATUS 2820
 
In vCenter I can see that there there is no snapshot available, and that the "Remove Snapshot" task was Completed sucesfully.
 
Another info, if I make the restore using nbd as transport mode than it works. 
Please helppp :) !

Thanks

Regards

Bogdan

Operating Systems:

Comments 7 CommentsJump to latest comment

Mark_Solutions's picture

Is the hotadd client a Windows 2008 based one?

If so take a look in Disk Manager (Storage) and see if the datastore volumes it has access to are shown as online - if not set them as online and try again

The same happens with any sort of VMWare restore if the VMWare backup host does not have the disks marked as online - they backup fine but do not restore (but nbd restores do work as you are not trying to write to the disks over the fibre)

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

carsteab's picture

Thanks for the quick reply. Sincerly I don't exactly understand what you mean. My configuration is the following:

NetBackup (Master and Media Server) is installed on a Virtual Machine in the same Data Center with the vm I want to backup and restore, a SLES 11 SP2 OS. In NetBackup I have configured a VMware backup host, fqdn same machine where the NetBackup is installed. Both machines are managed by the same ESXi host, and both are located on the same DataStore connected via iSCSI. The machine I want to restore is a Win2008.

During the Backup process I can see the vmdk is mounted in the NetBackup machine using the HotAdd transport mode. By restore I can see that the VM is erased, than is created again, than I can see a Snapshot is made, than the restore rappens, and than the snaphot is deleted, but the machine does not start and has an exclamation mark on it, like the snaphost delete did not complete succesfuly, and a consolidation has to be perfomed.

I should have given this info earliear :)...srry. 

Thanks

Regards

Bogdan

Mark_Solutions's picture

So what O/S is your VMWare backup host (the virtual machine that you use to do the backups)

If it is Windows what do you see in disk management?

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Mark_Solutions's picture

OK - that should be OK then.

There are a number of restrictions with hotadd restores but it doesn't sound like you have any of those

It may be worth logging a case in that case (unless the SAN has not been mapped with full rights to your backup host) - although I do love the get-out clause in the VMWare Admin Guide:

"If the attempt to restore a full virtual machine fails while using the SAN transport type, try the NBD transport type instead."

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

carsteab's picture

I thought if the VMware backup host is installed in a VM, I don't have to configure SAN access in the VMware backup host for the HOTADD transport. Or I got it wrong?

Mark_Solutions's picture

Apologies - it uses a linked clone for its purposes

There are some limitations with hotadd restores (lots of VMWare KB's) such as a limit on the number of scsi controllers available (60 oer ESX server i believe)

I also saw a note that any existing snapshots for the client must be removed before you run a restore using the hotadd method - it is not always easy to see these, even via vCenter

Hotadd restores also need additional rights to nbd restores:

■  Datastore->Allocate space
 
■  VirtualMachine->Configuration->Add existing disk
 
■  VirtualMachine->Configuration->Add new disk
 
■  VirtualMachine->Configuration->Add or remove device
 
■  VirtualMachine->Configuration->Change resource
 
■  VirtualMachine->Configuration->Remove disk
 
■  VirtualMachine->Configuration->Settings

These taken from this link: http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.datarecovery.admin.doc_20%2FGUID-9550127D-95CF-4302-AEA4-7DAC93DB4BA1.html

As this is such a complex area with so many requirements it may be worth loggin a call so that every possibility of failure can be worked through

The bpfis, bpcd, and tar logs may be useful to get more information on exactly why it fails but you may need to up the logging levels to get enough information

 

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.