Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

VMWare backup failed with RC 156

Created: 09 Oct 2012 | 6 comments
tarangdce's picture
All,
 
I have created a policy with 22 clients and 16 failed with below error and rest 6 successful at 1st attempt. 
Please share your suggession!!!
 
Error log as below.
 
10/9/2012 6:16:50 AM - Info nbjm(pid=15412) starting backup job (jobid=1420847) for client FDYTS407, policy VMBACKUP_BL465_1, schedule FULL  
10/9/2012 6:16:50 AM - Info nbjm(pid=15412) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=1420847, request id:{2CE7B461-FDFA-4F34-AE08-317C9DAD68E8})  
10/9/2012 6:16:50 AM - requesting resource Findlay_Backup
10/9/2012 6:16:50 AM - requesting resource mps6645.NBU_CLIENT.MAXJOBS.FDYTS407
10/9/2012 6:16:50 AM - requesting resource mps6645.NBU_POLICY.MAXJOBS.VMBACKUP_BL465_1
10/9/2012 6:16:50 AM - granted resource mps6645.NBU_CLIENT.MAXJOBS.FDYTS407
10/9/2012 6:16:50 AM - granted resource mps6645.NBU_POLICY.MAXJOBS.VMBACKUP_BL465_1
10/9/2012 6:16:50 AM - granted resource MediaID=@aaaao;DiskVolume=fdyddsvr001-bkp-lsu;DiskPool=fdyddsvr001-bkp-lsu;Path=fdyddsvr001-bkp-lsu;StorageServer=fdyddsvr001-10g;MediaServer=MPS6647
10/9/2012 6:16:50 AM - granted resource Findlay_Backup
10/9/2012 6:16:55 AM - estimated 0 Kbytes needed
10/9/2012 6:16:55 AM - begin Parent Job
10/9/2012 6:16:55 AM - begin VMware, Start Notify Script
10/9/2012 6:16:55 AM - Info RUNCMD(pid=18744) started            
10/9/2012 6:16:55 AM - started
10/9/2012 6:16:56 AM - Info RUNCMD(pid=18744) exiting with status: 0         
Status 0
10/9/2012 6:16:56 AM - end VMware, Start Notify Script; elapsed time: 00:00:01
10/9/2012 6:16:56 AM - begin VMware, Step By Condition
Status 0
10/9/2012 6:16:56 AM - end VMware, Step By Condition; elapsed time: 00:00:00
10/9/2012 6:16:56 AM - begin VMware, Read File List
Status 0
10/9/2012 6:16:56 AM - end VMware, Read File List; elapsed time: 00:00:00
10/9/2012 6:16:56 AM - begin VMware, Create Snapshot
10/9/2012 6:16:57 AM - started process bpbrm (8892)
10/9/2012 6:16:58 AM - Info bpbrm(pid=8892) FDYTS407 is the host to backup data from     
10/9/2012 6:16:58 AM - Info bpbrm(pid=8892) reading file list from client        
10/9/2012 6:16:58 AM - Info bpbrm(pid=8892) start bpfis on client         
10/9/2012 6:16:58 AM - Info bpbrm(pid=8892) Starting create snapshot processing         
10/9/2012 6:16:59 AM - Info bpfis(pid=9856) Backup started           
10/9/2012 6:17:20 AM - snapshot backup of client FDYTS407 using method VMware_v2
10/9/2012 6:18:49 AM - end writing
Status 156
10/9/2012 6:18:49 AM - end VMware, Create Snapshot; elapsed time: 00:01:53
10/9/2012 6:18:49 AM - begin VMware, Stop On Error
Status 0
10/9/2012 6:18:49 AM - end VMware, Stop On Error; elapsed time: 00:00:00
10/9/2012 6:18:49 AM - begin VMware, Delete Snapshot
10/9/2012 6:18:50 AM - started process bpbrm (6908)
10/9/2012 6:18:51 AM - Info bpbrm(pid=6908) Starting delete snapshot processing         
10/9/2012 6:18:51 AM - Info bpfis(pid=0) Snapshot will not be deleted        
10/9/2012 6:18:53 AM - end writing
Status 1542
10/9/2012 6:18:53 AM - end VMware, Delete Snapshot; elapsed time: 00:00:04
Status 156
10/9/2012 6:18:53 AM - end Parent Job; elapsed time: 00:01:58
snapshot error encountered(156)

Comments 6 CommentsJump to latest comment

Marianne's picture

Are you trying to backup 22 VM's simultaneously?

If you are using automatic selection of virtual machines (Query Builder) have a look a Resource Limits.

You can limit number of simultaneous backups per ESX server, per Datastore, maximum number of simultaneous snapshots, etc... See NetBackup for VMware Administrator's Guide .

The other option is to limit the number of simultaneous backups per policy. Use the Limit jobs per policy setting on the policy Attributes tab.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

tarangdce's picture

I am not using automatic selection. All the vms are selected from ESX cluster and writing to same vmware backup host which has 16CPU and 12GB RAM.

Please suggest.

Stuart Green's picture

I would look to maybe slice and dice your VM's into more than 1 policy to troubleshoot. If this is the first time run...to attempt to backup these VM's.

Status 156 is a deep chasm of possible problems/root causes.

Unless these are ALL deployed from a same template, I would doubt if all these 22 VM's are the same.

Check the hardware level compatibility. NBU 7.5 supports no greater than vmx-8. vmx-9 is available via vsphere 5.1 (not supported by NBU yet - 7.6 maybe)

Are any of the VM's using IDE controllers for their VMDK's. Need to be SCSI.

Are all the datastores these 22 VM's are stored presented to the Vmware backup host.?

Can you snapshot all 22 VM's from vsphere client without any problems.?

Are the correct Vmware credentials given with the right amount of privileges in vsphere applied from a top level hierarchy.?

I would maybe think about using automatic selection but based on a folder level to begin with.

Are VMware Tools installed. Running a recent/current version/status.

Also, update to the latest maintenance/update pack for NetBackup.

Then start to look at logs on the vmware backup host.

Lots of things to check out.  Also...

Resources:

Netbackup 7.1 for Vmware best practices
http://www.symantec.com/content/en/us/enterprise/w...

Datasheet: NetBackup 7.5 for VMware
http://eval.symantec.com/mktginfo/enterprise/fact_...

Tip: Get overview/document your NBU environment. Run 'nbsu' and review the output.

• If this provides help, please vote or mark appropriate solution.

tarangdce's picture

Unless these are ALL deployed from a same template, I would doubt if all these 22 VM's are the same.

They are different Operating system.

Check the hardware level compatibility. NBU 7.5 supports no greater than vmx-8. vmx-9 is available via vsphere 5.1 (not supported by NBU yet - 7.6 maybe)

All are hardware level vmx-8

Are any of the VM's using IDE controllers for their VMDK's. Need to be SCSI.

All are SCSI controllers

Are all the datastores these 22 VM's are stored presented to the Vmware backup host.?

YES

Can you snapshot all 22 VM's from vsphere client without any problems.?

Need to check out

Are the correct Vmware credentials given with the right amount of privileges in vsphere applied from a top level hierarchy.?

YES. I can take successful backup of 6 host out of those

I would maybe think about using automatic selection but based on a folder level to begin with.

As of now i have selected "ALL LOCAL DRIVES". I can try with C:\ only.

Are VMware Tools installed. Running a recent/current version/status.

YES

NetBackup Master & Media servers are 7.5.0.3

I will check out logs and try with just C:\ and see if that make difference.

Stuart Green's picture

Just to clarify:

By folder level, I meant within the vsphere infrastructure. not at an OS level.

Please leave at ALL_LOCAL_DRIVES

Also lots of good reasons for 156 status discussed in this forum post

https://www-secure.symantec.com/connect/forums/status-156-vmware-vstorage-backup

Tip: Get overview/document your NBU environment. Run 'nbsu' and review the output.

• If this provides help, please vote or mark appropriate solution.

tarangdce's picture

Thanks for different inputs. Actual issue was selecting VM DISPLAY NAME instead of VM HOSTNAME.

Issue is solved now.