After upgrading to 7.6.0.1/2.6.0.1, VMware backups via SAN may fail with Status 130 due to bpbkar core dump

Article:TECH214512  |  Created: 2014-01-29  |  Updated: 2014-08-12  |  Article URL http://www.symantec.com/docs/TECH214512
Article Type
Technical Solution

Product(s)


Issue



After upgrading to 7.6.0.1/2.6.0.1, VMware backups via SAN on Linux VMware Backup host may fail with Status 130  (system error occurred) due to bpbkar catching a SIGPIPE signal.


Error



Detail Status:
17-1-2014 21:42:47 - Info bpbkar(pid=61092) INF - Transport Type = san
17-1-2014 21:51:51 - Error bpbrm(pid=61074) from client VMCLIENT: ** (process:61092): WARNING **: MXUserDumpExclLock: Exclusive lock @ 0x96a590
17-1-2014 21:51:51 - Error bpbrm(pid=61074) from client VMCLIENT: ** (process:61092): WARNING **: signature 0x47C7DC65
17-1-2014 21:51:51 - Error bpbrm(pid=61074) from client VMCLIENT: ** (process:61092): WARNING **: name vixDiskLibVimLock
17-1-2014 21:51:51 - Error bpbrm(pid=61074) from client VMCLIENT: ** (process:61092): WARNING **: rank 0x0
17-1-2014 21:51:51 - Error bpbrm(pid=61074) from client VMCLIENT: ** (process:61092): WARNING **: serial number 1
17-1-2014 21:51:51 - Error bpbrm(pid=61074) from client VMCLIENT: ** (process:61092): WARNING **: count 1
17-1-2014 21:51:51 - Error bpbrm(pid=61074) from client VMCLIENT: ** (process:61092): WARNING **: address of owner data 0x96a5f8
17-1-2014 21:51:51 - Error bpbrm(pid=61074) from client VMCLIENT: ** ERROR **: MXUser_DestroyExclLock: Destroy of an acquired exclusive lock
17-1-2014 21:51:51 - Error bpbrm(pid=61074) from client VMCLIENT: aborting...
17-1-2014 21:51:52 - Critical bpbrm(pid=61074) from client VMCLIENT: FTL - terminated by signal 6
17-1-2014 21:51:52 - Critical bpbrm(pid=61074) from client VMCLIENT: FTL - cleanup() failed, status 130
17-1-2014 21:51:54 - Error bptm(pid=61098) media manager terminated by parent process
 
17-1-2014 21:52:14 - Info bpbkar(pid=0) done. status: 130: system error occurred
17-1-2014 21:52:14 - end writing; write time: 0:11:20 system error occurred(130)

 
bpbkar:
10:01:20.920 [34196] <32> bpfsmap: FTL - catchsignal(): caught signal=13
10:01:20.920 [34196] <2> vxms_shutdown: vxms_started = 1 (../vxms_bridge.cpp:918)
10:01:23.062 [34196] <32> handle_core_signals: FTL - terminated by signal 6
10:01:23.062 [34196] <16> bpbkar Exit: reached 11627
10:01:23.062 [34196] <16> bpbkar Exit: reached 11633
10:01:23.062 [34196] <16> bpbkar Exit: reached 11698
10:01:23.062 [34196] <16> bpbkar Exit: reached 11811
10:01:23.062 [34196] <2> ol_cleanup: INF - removing /tmp/vmware+34196+1.std_filelist
10:01:23.062 [34196] <8> bpbkar Exit: FTL - cleanup() failed, status 130
10:01:23.093 [34196] <16> bpbkar Exit: reached 11839
10:01:23.093 [34196] <16> bpbkar Exit: ERR - bpbkar FATAL exit status = 130: system error occurred
10:01:23.093 [34196] <16> bpbkar Exit: reached 11885
10:01:23.093 [34196] <4> bpbkar Exit: INF - EXIT STATUS 130: system error occurred

 
The issue occurs after VDDK receives a SAN error during the backups:
vdRead:VixInterface.cpp:559 <TRACE> : Read(9064624, 68476160, 256)
g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2014-01-23T10:01:19.917Z [7EFF73DBE700 info 'Libs'] FileIOErrno2Result: Unexpected errno=5, Input/output error
g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2014-01-23T10:01:19.917Z [7EFF73DBE700 info 'Libs'] read failed, errno=5, Input/output error
g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2014-01-23T10:01:19.917Z [7EFF73DBE700 error 'Default'] Incomplete read from, Wanted 131072 Got 0, Error 2 (5)

 
The following can be seen in the messages log on the VMware Backup Host at the time:
kernel: [511349.964136] sd 3:0:1:45: reservation conflict
kernel: [511349.964147] sd 3:0:1:45: [sds] Unhandled error code
kernel: [511349.964161] sd 3:0:1:45: [sds] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
kernel: [511349.964163] sd 3:0:1:45: [sds] CDB: Read(10): 28 00 d7 9b a5 80 00 01 00 00
kernel: [511349.964169] end_request: I/O error, dev sds, sector 3617301888
kernel: [511349.964583] sd 3:0:1:45: reservation conflict
kernel: [511349.964592] sd 3:0:1:45: [sds] Unhandled error code
kernel: [511349.964595] sd 3:0:1:45: [sds] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
kernel: [511349.964599] sd 3:0:1:45: [sds] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
kernel: [511349.964617] end_request: I/O error, dev sds, sector 0


Environment



This issue has been seen with only on Linux VMware Backup Host running backups via SAN.


Solution



The formal resolution for this issue (Etrack 3415512) is included in the following releases:

  • NetBackup 7.6 Maintenance Release 2 (7.6.0.2)
  • NetBackup 52x0 Appliances 2.6.0.2

More information on these releases can be found in the Related Articles linked below.

Workaround:
If this issue is experienced on a VMware backup host at NetBackup 7.6.0.1 or on a NetBackup Appliance at version 2.6.0.1, please apply the relevant hotfix attached to this document.

Note: These hotfixes are no longer publicly available, as all of thjeir fixes are included in the NetBackup 7.6.0.2 and NetBackup Appliances 2.6.0.2 maintenance releases.  If this issue is experienced, the supported resolution is to apply the latest maintenance release.

NetBackup 7.6.0.1:

Symantec Bug ID: ET 3415800

Installation Location:  VMware backup host

Installation Instructions:  
Please follow the EEB Installer instructions available in the Related Article linked below.

Package Contents:
Please choose the appropriate platform after download:
eebinstaller.3415800.4.linuxR_x86_2.6.18     RedHat x64 Installation
eebinstaller.3415800.4.linuxS_x86_2.6.16     Suse x64 Installation

Checksums:
1671399294 751371 linuxS_x86_2.6.16/libbpfsmap.so
3911379910 610255 linuxS_x86_2.6.16/bpbkar
603527459 740410 linuxR_x86_2.6.18/libbpfsmap.so
354997656 604027 linuxR_x86_2.6.18/bpbkar

Recommended service state: Ensure that no backups or restores are running on the backup host when installing this EEB.

NetBackup 52x0 Appliances 2.6.0.1:
Please download the RPM attached below and access the Related Article linked below for instructions on applying the hotfix on an Appliance.


Supplemental Materials

SourceETrack
Value3415512
Description

After upgrading to 7.6.0.1 from 7.5.0.6, bpbkar core dumps intermittently with SAN base backups on Linux VMware Backup Host


SourceError Code
Value130
Description

system error occurred




Article URL http://www.symantec.com/docs/TECH214512


Terms of use for this information are found in Legal Notices