Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

156) snapshot error encountered (VM)

Created: 19 Sep 2013 | 8 comments

I am trying to make backups of about 20-25 VM clients from them about 10 does not get backup, it gives error.

(156) snapshot error encountered

 

netbackup 7.1

I am not so good in VM at this moment, waiting to go on a course for it.

I do not know how to fixed this, here are the detail report of the error.

 

09/19/2013 22:36:27 - Info nbjm (pid=3964) starting backup job (jobid=41097) for client PRASAGPPMAR01.mrail.co.za, policy VM_backups_VMW02, schedule Daily_Diff
09/19/2013 22:36:27 - Info nbjm (pid=3964) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=41097, request id:{278AA80F-FFEF-4CF8-B553-6B4F8A4BA2BA})
09/19/2013 22:36:27 - requesting resource prasagppbck02-hcart2-robot-tld-0
09/19/2013 22:36:27 - requesting resource prasagppbck01.prasa.com.NBU_CLIENT.MAXJOBS.PRASAGPPMAR01.mrail.co.za
09/19/2013 22:36:27 - requesting resource prasagppbck01.prasa.com.NBU_POLICY.MAXJOBS.VM_backups_VMW02
09/19/2013 22:36:27 - awaiting resource prasagppbck02-hcart2-robot-tld-0. No drives are available.
09/19/2013 22:36:45 - granted resource  prasagppbck01.prasa.com.NBU_CLIENT.MAXJOBS.PRASAGPPMAR01.mrail.co.za
09/19/2013 22:36:45 - granted resource  prasagppbck01.prasa.com.NBU_POLICY.MAXJOBS.VM_backups_VMW02
09/19/2013 22:36:45 - granted resource  CTB389
09/19/2013 22:36:45 - granted resource  HP.ULTRIUM5-SCSI.001
09/19/2013 22:36:45 - granted resource  prasagppbck02-hcart2-robot-tld-0
09/19/2013 22:36:45 - estimated 610014 kbytes needed
09/19/2013 22:36:45 - begin Parent Job
09/19/2013 22:36:45 - begin Flash Backup Windows: Start Notify Script
09/19/2013 22:36:45 - Info RUNCMD (pid=3956) started
09/19/2013 22:36:45 - Info RUNCMD (pid=3956) exiting with status: 0
Operation Status: 0
09/19/2013 22:36:45 - end Flash Backup Windows: Start Notify Script; elapsed time 0:00:00
09/19/2013 22:36:45 - begin Flash Backup Windows: Step By Condition
Operation Status: 0
09/19/2013 22:36:45 - end Flash Backup Windows: Step By Condition; elapsed time 0:00:00
09/19/2013 22:36:45 - begin Flash Backup Windows: Read File List
Operation Status: 0
09/19/2013 22:36:45 - end Flash Backup Windows: Read File List; elapsed time 0:00:00
09/19/2013 22:36:45 - begin Flash Backup Windows: Create Snapshot
09/19/2013 22:36:47 - started process bpbrm (pid=1016)
09/19/2013 22:36:48 - Info bpbrm (pid=1016) PRASAGPPMAR01.mrail.co.za is the host to backup data from
09/19/2013 22:36:48 - Info bpbrm (pid=1016) reading file list from client
09/19/2013 22:36:48 - Info bpbrm (pid=1016) start bpfis on client
09/19/2013 22:36:48 - begin Create Snapshot
09/19/2013 22:36:48 - Info bpfis (pid=7048) Backup started
09/19/2013 22:36:48 - snapshot backup of client PRASAGPPMAR01.mrail.co.za using method VMware
09/19/2013 22:37:22 - Critical bpbrm (pid=1016) from client PRASAGPPMAR01.mrail.co.za: FTL - snapshot creation failed, status 156
09/19/2013 22:37:22 - Warning bpbrm (pid=1016) from client PRASAGPPMAR01.mrail.co.za: WRN - ALL_LOCAL_DRIVES is not frozen
09/19/2013 22:37:22 - Info bpfis (pid=7048) done. status: 156
09/19/2013 22:37:22 - end Create Snapshot; elapsed time 0:00:34
09/19/2013 22:37:22 - Error bpbrm (pid=1016) cannot send mail to hendrik@apx.co.za
09/19/2013 22:37:23 - end writing
Operation Status: 156
09/19/2013 22:37:23 - end Flash Backup Windows: Create Snapshot; elapsed time 0:00:38
09/19/2013 22:37:23 - begin Flash Backup Windows: Stop On Error
Operation Status: 0
09/19/2013 22:37:23 - end Flash Backup Windows: Stop On Error; elapsed time 0:00:00
09/19/2013 22:37:23 - begin Flash Backup Windows: Delete Snapshot On Exit
09/19/2013 22:37:23 - begin Flash Backup Windows: Delete Snapshot On Exit
09/19/2013 22:37:25 - started process bpbrm (pid=1984)
09/19/2013 22:37:28 - end writing
Operation Status: 0
09/19/2013 22:37:28 - end Flash Backup Windows: Delete Snapshot On Exit; elapsed time 0:00:05
09/19/2013 22:37:28 - begin Flash Backup Windows: End Notify Script
09/19/2013 22:37:29 - Info RUNCMD (pid=1196) started
09/19/2013 22:37:29 - Info RUNCMD (pid=1196) exiting with status: 0
Operation Status: 0
09/19/2013 22:37:29 - end Flash Backup Windows: End Notify Script; elapsed time 0:00:01
Operation Status: 156
09/19/2013 22:37:29 - end Flash Backup Windows: Delete Snapshot On Exit; elapsed time 0:00:06
snapshot error encountered  (156)
 

Operating Systems:

Comments 8 CommentsJump to latest comment

Marianne's picture

Have you checked and verified basic requirements such as vmtools installed on all VMs? And that there are no RDM disks on these VMs?

Work with you VM administrator and see if you can create snapshots for these VMs using vCenter.

Couple of TNs:

http://www.symantec.com/docs/TECH160314

http://www.symantec.com/docs/HOWTO44492

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Mark_Solutions's picture

Lots of causes for this - best to setup bpfis logging first on the VMware backup host (media server?) and see if that gives more information

The vCenter console should also have given an alert so take a look in there to see what error that gave

One other thing i have seen is when people kick off too many VMware backups at the same time which overloads the VMWare backup host and ends in multiple bpfis processes crashing

The word APP in the client name also suggests that the clint is running something (SQL, Exchange etc) which could cause the file system quiesce to timeout or fail

Hope this helps - we would need more info to get to the bottom of this

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

watsons's picture

As Mark & Marianne mentioned, error 156 is generic and many things could cause that.

In one recent environment I saw, the diskspace is getting low on the VM backup host.

When we try to change the VM policies to run on different backup windows, they will all be working. Creating snapshot usually requires certain amount of diskspace, when  you have more and more VM clients setup and when they run backup at the same time, this may happen.

RJGuinn's picture

Check this link from VMWare. I had the same issue at one of my clients, and this was the resolution.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1018194

 

SanjayPM's picture

VMware snapshot errors can come from the following sources : a) too many snapshot requests on the datastore or ESX server b) SAN connectivity and disk performance c) Load on the vCenter d) applications/usage on the vm itself and e) in some instances the OS of the VM. It's best to use resource limits from Advanced Options of VIP policy configuration to limit 1 snapshot per datastore and 1 per ESX. Using policy job limits then spread the load of the job across as many ESX servers you have in the environment. If you are finding that one particular vm is always giving the snapshot error I would isolate that into a seperate policy to determine the root cause of the issue. Using bpfis logs with verbosity level of 3 will provide additional details. This will assist you in finding the right area to focus your troubleshooting efforts in.

MOHAMED PATEL's picture

Following from SanjayPM:

Limit the amount of backups to say 6 and test -

In the VM Policy - use "Limit jobs per Policy" start using 6 jobs and check progress.

Resources (memory) on your backup host is also relevant - make sure it can handle the amount of snapshots.

 

If this post is helpful in any way, please vote or mark as solution.
A thumbs UP will do as well....
Regards
Mo

Hendrikbez's picture

We are going to move all the VM stuff to a new storage and fixed this storage, hopefully then the problems will be fixed.

 

Marianne's picture

WOW! This is a long time to be without successful backups!

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links