Video Screencast Help

VMWare Host Backup with status 196

Created: 08 Mar 2013 | 4 comments

Unix Netbackup v7.1.0.4 running on RHEL 6.2.  Two Windows 2008 R2 Standard Media Servers v7.1.0.4.  Backup target is DataDomain dd670

I seem to be plagued with the oddest of problems.  This one is a VMWare backup policy that starts at the beginning of it's backup window and immediately dies with a status 196 then ~12 minutes later starts again and runs the entire VMWare backup status 0 around 264 VMs.  After backing up for a prolonged time without any such error it started happening within the last six weeks and occurs frequently.  Most recently with this mornings (3-8-2013) backup just after midnight.

Below is the status 196 and below that is the status 0

03/08/2013 00:00:00 - Info nbjm (pid=11371) starting backup job (jobid=1383631) for client utlp5178.ad.ama-assn.org, policy VMWare-5178, schedule DailyIncremental
03/08/2013 00:00:00 - Info nbjm (pid=11371) requesting MEDIA_SERVER_ONLY resources from RB for backup job (jobid=1383631, request id:{69727E98-87B5-11E2-B0C8-02652D090016})
03/08/2013 00:00:00 - requesting resource su_dtc_dd670dtc
03/08/2013 00:00:00 - requesting resource utlp1227.NBU_CLIENT.MAXJOBS.utlp5178.ad.ama-assn.org
03/08/2013 00:00:00 - requesting resource utlp1227.NBU_POLICY.MAXJOBS.VMWare-5178
03/08/2013 00:00:00 - granted resource  utlp1227.NBU_CLIENT.MAXJOBS.utlp5178.ad.ama-assn.org
03/08/2013 00:00:00 - granted resource  utlp1227.NBU_POLICY.MAXJOBS.VMWare-5178
03/08/2013 00:00:06 - estimated 143744 kbytes needed
03/08/2013 00:00:06 - begin Parent Job
03/08/2013 00:00:06 - begin Application Resolver: Start Notify Script
03/08/2013 00:00:06 - Info RUNCMD (pid=31661) started
03/08/2013 00:00:06 - Info RUNCMD (pid=31661) exiting with status: 0
Operation Status: 0
03/08/2013 00:00:06 - end Application Resolver: Start Notify Script; elapsed time 0:00:00
03/08/2013 00:00:06 - begin Application Resolver: Step By Condition
Operation Status: 0
03/08/2013 00:00:06 - end Application Resolver: Step By Condition; elapsed time 0:00:00
03/08/2013 00:00:06 - begin Application Resolver: Resolver Discovery
Operation Status: 0
03/08/2013 00:01:13 - end Application Resolver: Resolver Discovery; elapsed time 0:01:07
03/08/2013 00:01:13 - begin Application Resolver: Policy Execution Manager Preprocessed
Operation Status: 196
03/08/2013 00:02:49 - end Application Resolver: Policy Execution Manager Preprocessed; elapsed time 0:01:36
03/08/2013 00:02:49 - begin Application Resolver: Stop On Error
Operation Status: 0
03/08/2013 00:02:49 - end Application Resolver: Stop On Error; elapsed time 0:00:00
03/08/2013 00:02:49 - begin Application Resolver: End Notify Script
03/08/2013 00:02:49 - Info RUNCMD (pid=5491) started
03/08/2013 00:02:49 - Info RUNCMD (pid=5491) exiting with status: 0
Operation Status: 0
03/08/2013 00:02:49 - end Application Resolver: End Notify Script; elapsed time 0:00:00
Operation Status: 196
03/08/2013 00:02:49 - end Parent Job; elapsed time 0:02:43
client backup was not attempted because backup window closed  (196)

Status 0

03/08/2013 00:12:49 - Info nbjm (pid=11371) starting backup job (jobid=1384083) for client utlp5178.ad.ama-assn.org, policy VMWare-5178, schedule DailyIncremental
03/08/2013 00:12:49 - Info nbjm (pid=11371) requesting MEDIA_SERVER_ONLY resources from RB for backup job (jobid=1384083, request id:{33E8CF82-87B7-11E2-A452-B3F10C46F4AC})
03/08/2013 00:12:49 - requesting resource su_dtc_dd670dtc
03/08/2013 00:12:49 - requesting resource utlp1227.NBU_CLIENT.MAXJOBS.utlp5178.ad.ama-assn.org
03/08/2013 00:12:49 - requesting resource utlp1227.NBU_POLICY.MAXJOBS.VMWare-5178
03/08/2013 00:12:49 - granted resource  utlp1227.NBU_CLIENT.MAXJOBS.utlp5178.ad.ama-assn.org
03/08/2013 00:12:49 - granted resource  utlp1227.NBU_POLICY.MAXJOBS.VMWare-5178
03/08/2013 00:12:49 - estimated 143744 kbytes needed
03/08/2013 00:12:49 - begin Parent Job
03/08/2013 00:12:49 - begin Application Resolver: Start Notify Script
03/08/2013 00:12:49 - Info RUNCMD (pid=22200) started
03/08/2013 00:12:49 - Info RUNCMD (pid=22200) exiting with status: 0
Operation Status: 0
03/08/2013 00:12:49 - end Application Resolver: Start Notify Script; elapsed time 0:00:00
03/08/2013 00:12:49 - begin Application Resolver: Step By Condition
Operation Status: 0
03/08/2013 00:12:49 - end Application Resolver: Step By Condition; elapsed time 0:00:00
03/08/2013 00:12:49 - begin Application Resolver: Resolver Discovery
Operation Status: 0
03/08/2013 00:12:49 - end Application Resolver: Resolver Discovery; elapsed time 0:00:00
03/08/2013 00:12:49 - begin Application Resolver: Policy Execution Manager Preprocessed
Operation Status: 0
03/08/2013 03:31:05 - end Application Resolver: Policy Execution Manager Preprocessed; elapsed time 3:18:16
03/08/2013 03:31:05 - begin Application Resolver: End Notify Script
03/08/2013 03:31:05 - Info RUNCMD (pid=31260) started
03/08/2013 03:31:05 - Info RUNCMD (pid=31260) exiting with status: 0
Operation Status: 0
03/08/2013 03:31:05 - end Application Resolver: End Notify Script; elapsed time 0:00:00
Operation Status: 0
03/08/2013 03:31:05 - end Parent Job; elapsed time 3:18:16
the requested operation was successfully completed  (0)

Operating Systems:

Comments 4 CommentsJump to latest comment

Nagalla's picture

failing the job with  EC 196 in less then 3 min is big  strange issue.. unless you have backup window is less then that( i dont think so this is the case)

may be some EMM DB update issues

does your server loaded heavlily?

I would suggest you to go with symantec support case...

MilesVScott's picture

What version of ESX are you running? Both 5.0U1 and 5.1 can have issues because 7.1 uses the VDDK from 5.0. You may want to check the VM datastore for phantom snapshot's as well, just as a heads up!

If my answer was helpful, please mark as a solution so others can benefit as well.

NBU 7.1.0.3,Win 2k3 Enterprise x64 SP2

Miles Scott | Holland & Knight
IT Infrastructure Specialist

trs06's picture

Thx, we are running 5.0U1 so I will research from that angle and see what I can find.

Worst time is when Fridays weekly fulls kick off.  Our system is indeed heavily and universally taxed.  We are looking at this too and what we can / need to do about it.

MilesVScott's picture

I actually just reread the latest update on this, (http://www.symantec.com/docs/TECH127089 pg12). It looks like there are only known issues for vmx-09. I have also heard about issues with the new distributed authentication(SSO) features, but it dosent look like this is similar to your issue. Sorry about the false lead.

If my answer was helpful, please mark as a solution so others can benefit as well.

NBU 7.1.0.3,Win 2k3 Enterprise x64 SP2

Miles Scott | Holland & Knight
IT Infrastructure Specialist