VMware Backup Failure with NetBackup 7.5, Transport Mode Error 23, Status 6
I thought I would post a solution here to a Status 23 error from bpbrm which results in a Status 6 on the parent VMware backup job (child job never spawns).
NBU version: 184.108.40.206 / 220.127.116.11
Master Server: Solaris 10 (SPARC)
Media / Backup Host: Windows 2008 R2
ESXi 4.1.0 / vCenter 4.1.0
Storage: NetApp 3240
Policy Type: VMware
Storage Policy: Local DSSU
Transport mode: NBD/NBDSSL (this issue can also apply to SAN)
If you're like me, you've already looked into the possible solutions below...
Ensure hostname resolution works for all ESXi hosts / vCenter to/from the Master Server
Are the datastores presented/accessible to the VMware Backup Host? (not applicable for NBD)
...but to no avail. Engaged tech support, turn VxMS logging on etc.
Logs produced look similar to this:
Detailed Status (from Activity Monitor):
10/23/2012 10:45:07 PM - Error bpbrm(pid=8764) from client <hostname>: ERR - Error opening the snapshot disks using given transport mode: Status 23
10/23/2012 10:45:08 PM - Info bptm(pid=4724) waited for full buffer 1 times, delayed 1619 times
10/23/2012 10:45:08 PM - Info bptm(pid=4724) EXITING with status 90 <----------
10/23/2012 10:45:08 PM - Info bpbkar32(pid=5064) bpbkar waited 0 times for empty buffer, delayed 0 times.
10/23/2012 10:45:11 PM - Error bpbrm(pid=8764) could not send server status message
10/23/2012 10:45:11 PM - Critical bpbrm(pid=8764) unexpected termination of client <hostname>
the backup failed to back up the requested files(6)
09:53:23.402 [6844.7840] <2> onlfi_vfms_logf: INF - vfm_fi_get_credentials: No disk array credentials found on server
09:53:23.621 [6844.7840] <2> onlfi_vfms_logf: INF - vfm_fi_get_credentials: No storage server credentials found on server
09:53:37.521 [6844.7840] <2> onlfi_vfms_logf: INF - vmwareLogger: RetrieveHostContents: Datastore: name = vmbackup is NOT accessible
09:53:37.521 [6844.7840] <2> onlfi_vfms_logf: INF - vmwareLogger: RetrieveDatastoreContents: datastore vmbackup is NOT accessible
18:57:28.0408 : connectToHost:xGuest.cpp:1017 <INFO> : TransportMode requested: nbd
18:57:28.0408 : connectToHost:xGuest.cpp:1022 <INFO> : Transport Modes available on this host: file:san:hotadd:nbdssl:nbd
18:57:30.0265 : vdConnect:xInterface.cpp:869 <DEBUG> : Done with VixDiskLib_Connect(): 214236752
18:57:30.0265 : vdOpen:xInterface.cpp:367 <INFO> : Calling VixDiskLib_Open()
18:57:30.0265 : g_vixInterfaceLogger:bvix.cpp:1833 <INFO> : VixDiskLibVimResolveHostName: Resolving IP address for hostname server.domain.com.
18:57:30.0265 : g_vixInterfaceLogger:bvix.cpp:1833 <INFO> : VixDiskLibVimResolveHostName: Resolved to 10.10.10.10.
18:57:30.0280 : g_vixInterfaceLogger:bvix.cpp:1833 <INFO> : VixDiskLibVim: VixDiskLibVimLogin
18:57:31.0185 : g_vixInterfaceLogger:bvix.cpp:1837 <WARN> : VixDiskLibVim: Not licensed to use this function.
This error will repeat for all available transport modes.
Steps we tried that did not work:
- Removed and re-added client to policy
- Upgraded from 18.104.22.168 to 22.214.171.124 (a suggestion from Symantec)
- vMotion VM to another ESXi host
- Storage vMotion VM to another datastore
We cloned the VM and the clone successfully backed up. Which led us to wonder if it was a config issue on the VMware side (VMX file?).
After taking an outage for the client we did the following:
- Remove the offending VM from vCenter inventory.
- Rename the VM folder using the datastore browser. (This step is to avoid the same name folder that will be created in step 3)
- Create a new VM with the same name and specs. But no disks attached.
- Move the vmdk files from the old file folder to new folder.
- "Edit settings" on the newly created VM and attach "existing disk" by locating the VMDK files.
- Power on the new VM.
- Remove the old (renamed) file folder which should contain only the vmx, nvram and log files [house cleaning]
The client was then able to backup successfully.