Video Screencast Help

VM Backups intermittetly fail with 156

Created: 04 Oct 2012 | 4 comments

I have Netbackup 7.1.04 master server running on Linux and 4 Windows media servers (also acting as VMware backup host). We have configured VM backups from two different VC's. Backups from one of the VC's intermittenly fail with error 156 (snapshot creation failed). Most of the time backup completes by itself in the second retry and when I check the bpfis logs for the first attemp failure, it is giving the following error. It has problem connecting to VC during that time, but some of the backups are completing successfully during that time, so I guess there is no network issue.

 

vSphereConnect: connecting to https://FRAKBPOVC01/sdk
01:04:42.361 [3900.5156] <2> onlfi_vfms_logf: INF - dovSphereDisconnect: disconnect for FRAKBPOVC01 completed with flags 0 and status 0
01:04:42.814 [2212.1508] <2> onlfi_vfms_logf: INF - vSphereConnect: connect returned 6
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - vSphereConnect: connect returned 6
6
01:04:42.814 [2212.1508] <2> onlfi_vfms_logf: INF - VMware_thaw: dovSphereLogin returned 25 for FRAKBPOVC01
01:04:42.814 [2212.1508] <2> onlfi_vfms_logf: INF - Login failed for FRAKBPOVC01 with 25
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - Connect failed for FRAKBPOVC01 with 6
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - vSphereconnect returned 25
ect for FRAKBPOVC01 completed with flags 0 and status 0
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - VMware_freeze: hostname: 10.40.21.125
01:04:42.814 [2212.1508] <4> onlfi_thaw: INF - do_thaw return value: 11
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - VMware_freeze: displayname: FRAKBPOEXXTW01S
01:04:42.814 [2212.1508] <32> onlfi_thaw: FTL - VfMS error 11; see following messages:
01:04:42.814 [2212.1508] <32> onlfi_thaw: FTL - Fatal method error was reported
: vmx-07
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - VMware_freeze: version number: 7
01:04:42.814 [2212.1508] <32> onlfi_thaw: FTL - vfm_thaw: method: VMware, type: FIM, function: VMware_thaw (invalid method return 25; must return VFM_E_METHOD or VFM_E_METHOD_FTL or VFM_E_OK)
01:04:42.814 [2212.1508] <32> onlfi_thaw: FTL - VfMS method error 7; see following message:
01:04:42.814 [2212.1508] <32> onlfi_thaw: FTL - VMware_ctl: Invalid parameter value: cmd=13
01:04:42.814 [2212.1508] <16> bpfis Exit: FTL - snapshot delete returned status 156
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - RegOpenKey failed with 2

01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - RegOpenKey failed with 2

01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - findVCB: C:\Program Files\VMware\VMware Consolidated Backup Framework\ not executable, errno 2
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - findVCB: D:\Program Files\VMware\VMware Consolidated Backup Framework\ not executable, errno 13
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - findVCB: E:\Program Files\VMware\VMware Consolidated Backup Framework\ not executable, errno 2
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - findVCB: F:\Program Files\VMware\VMware Consolidated Backup Framework\ not executable, errno 2
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - findVCB: G:\Program Files\VMware\VMware Consolidated Backup Framework\ not executable, errno 2
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - findVCB: H:\Program Files\VMware\VMware Consolidated Backup Framework\ not executable, errno 2
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - findVCB: I:\Program Files\VMware\VMware Consolidated Backup Framework\ not executable, errno 2
01:04:42.814 [2212.1508] <2> ol_cleanup: INF - removing C:\Program Files\Veritas\NetBackup\temp\VMWARE_Krupp4+d6600+1.std_filelist
table, errno 2
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - findVCB: K:\Program Files\VMware\VMware Consolidated Backup Framework\ not executable, errno 2
01:04:42.814 [3900.5156] <2> onlfi_vfms_logf: INF - findVCB: L:\Program Files\VMware\VMware Consolidated Backup Framework\ not executable, errno 2
01:04:42.814 [2212.1508] <4> bpfis Exit: INF - EXIT STATUS 156: snapshot error encountered
01:04:42.814 [2212.1508] <2> stop_keep_alive_thread: INF - Stop keep_alive thread
01:04:42.829 [3900.5156] <4> delete_mount_point: INF - Deleted mount point C:\Program Files\Veritas\NetBackup\temp\_vrts_frzn_img_ALL_LOCAL_DRIVES_3900
01:04:42.829 [3900.5156] <32> onlfi_freeze_fim_fs: FTL - VfMS error 11; see following messages:
01:04:42.829 [3900.5156] <32> onlfi_freeze_fim_fs: FTL - Fatal method error was reported
01:04:42.829 [3900.5156] <32> onlfi_freeze_fim_fs: FTL - vfm_freeze: method: VMware, type: FIM, function: VMware_freeze
01:04:42.829 [3900.5156] <32> onlfi_freeze_fim_fs: FTL - VfMS method error 7; see following message:
01:04:42.829 [3900.5156] <32> onlfi_freeze_fim_fs: FTL - VMware_freeze: Unable to find VCB path

 

Can someone help on this.

 

Thanks

Sreekanth

 

 

Comments 4 CommentsJump to latest comment

Marianne's picture

You don't say how many simultaneous backups you are running, so, my theory may be completely wrong. 

Status 156 could be because of overload.
Resource Limits were introduced in NBU 7.1 where you can limit number of simultaneous backups per ESX, per data store, maximum # of simultaneous snapshots, etc...

See Setting global limits on the use of VMware resources in NBU for VMWare manual.  

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

srdama's picture

Hi Marianne,

Please find the Resource limit settings. This is the same way it is configured on both VC we have. Other VC which has around 150 VM's to backup every night is working fine. The affected VC has only 90 VM's to backup, but still issues seen like snapshot and socket problems (error 156 and 13)

Vcenter  6
Snapshot No Limit
Cluster  No Limit
ESXserver 4
VMXDatastore No Limit
Datastore 4

 

Thanks

Sreekanth

Yasuhisa Ishikawa's picture

I suppose some sort of connecting limit at vCenter side. I don't know much about that, there are some limitaion at vCenter side like number of simultaneous connections, and number of connections in given time frame.

Ask VMware admin to check logs and settings.

Authorized Symantec Consultant(ASC) Data Protection in Tokyo, Japan

Yasuhisa Ishikawa's picture

Most likely network issue.

Even if some connection have no problem, I can not conclude that there are no issue in network.

for example incorrect trunking configuration may lead such unstable condition.

adding to reviewing VMware side, check network between VMware Access Host and vCenter. Creating and using another VMwareAccess Host may help to isolate problem, if possible.

Authorized Symantec Consultant(ASC) Data Protection in Tokyo, Japan