nbsl experiences a memory leak which results in a core dump.

Article:TECH201979  |  Created: 2013-01-23  |  Updated: 2013-12-16  |  Article URL http://www.symantec.com/docs/TECH201979
Article Type
Technical Solution


Issue



nbsl daemon crashes intermittently and creates a core dump.

It was observed that memory consumed by nbsl was constantly growing, and when it reached somewhere around 4GB, nbsl would crash.

The core's stack trace that can be retrieved by running pstack command against the core file shows that the crash happens in ACE copy methods, which is just an indication that nbsl ran out of memory:

# pstack <core>

#0 0xc00000000d820c80:0 in v6::ACE_Message_Block::copy ()
at Message_Block.cpp:93
93 Message_Block.cpp: No such file or directory.
in Message_Block.cpp
(gdb) bt
#0 0xc00000000d820c80:0 in v6::ACE_Message_Block::copy ()
at Message_Block.cpp:93
#1 0xc00000000bb78b60:0 in v6::TAO_Transport::handle_input ()
at Transport.cpp:1388
#2 0xc00000000bb5c8a0:0 in v6::TAO_Connection_Handler::handle_input_eh ()
at Connection_Handler.cpp:257
#3 0xc00000000ac01810:0 in PBXIOP_Connection_Handler::handle_input ()
at PBXIOP_Connection_Handler.cpp:448
#4 0xc00000000d6742c0:0 in v6::ACE_TP_Reactor::dispatch_socket_event ()
at TP_Reactor.cpp:586
#5 0xc00000000d6dd060:0 in v6::ACE_TP_Reactor::dispatch_i ()
at TP_Reactor.cpp:451
#6 0xc00000000d6dc4c0:0 in v6::ACE_TP_Reactor::handle_events ()
at TP_Reactor.cpp:172
#7 0xc00000000d5b78f0:0 in v6::ACE_Reactor::handle_events ()
at Reactor.cpp:373
#8 0xc00000000bb33c40:0 in v6::TAO_ORB_Core::run () at ORB_Core.cpp:1858
#9 0xc00000000be0a990:0 in v6::CORBA_ORB::run () at ORB.cpp:277
#10 0xc000000008cf39c0:0 in Orb::runThisThread () at ../Orb.cpp:1882
#11 0xc000000008d20950:0 in OrbService::start () at ../OrbService.cpp:626
#12 0x40000000048ac520:0 in Veritas::NetBackup::OrbServiceSL::StartFromMain ()
at ../OrbServiceSL.cpp:297
#13 0x4000000002ea34c0:0 in main () at ../NBSL.cpp:87


Cause



There were several memory leaks identified in nbsl that can cause the problem:

  1. Memory leaks related to Cloud.
    In the nbsl log look for Cloud calls - for example, search for CloudQFImpl::getCloudConfigBlob
  2. Memory leak was high in case of job and image collection from OpsCenter.
  3. Memory leak was found in device.

Solution



The formal resolution for this issue (Etrack 3123064) is included in the following release:

  • NetBackup 7.5 Maintenance Release 6 (7.5.0.6)

NetBackup 7.5.0.6 is now available - please access the Related Article below for more information.

Hotfix:
Note: This hotfix is no longer publicly available, as all of its fixes are included in the NetBackup 7.5.0.6 Maintenance Release.  If any of these issues are experienced in a NetBackup 7.5.0.x environment, the supported resolution is to apply the latest maintenance release. 

Note: The instructions included in this TechNote supersede any instructions provided within the hotfix itself.

The Attachment contains a replacement for nbsl for NetBackup 7.5.0.5.

Symantec Bug ID: ET 3123001

Installation Location:  master server

Installation Instructions: 
Please follow the EEB Installer instructions available on http://www.symantec.com/docs/TECH64620

Package Contents:
Please choose the appropriate platform after download:
eebinstaller.3123001.1.hpia64                HP-UX Itanium Installation
eebinstaller.3123001.1.linuxR_x86            RedHat x64 Installation
eebinstaller.3123001.1.linuxS_x86            Suse x64 Installation
eebinstaller.3123001.1.rs6000                AIX Installation
eebinstaller.3123001.1.solaris               Solaris SPARC Installation
eebinstaller.3123001.1.solaris_x86           Solarix x64 Installation
eebinstaller.3123001.1.AMD64.exe             Windows x64 Installation
eebinstaller.3123001.1.x86.exe               Windows x86 Installation

Checksums:
3821575800 151308920 hp_ux/nbsl
3184508857 51427935 linuxR_x86/nbsl
1876520371 51729461 linuxS_x86/nbsl
2928355275 145989575 rs6000/nbsl
3144517946 79573320 solaris/nbsl
1455699172 95916432 solaris_x86/nbsl
3446688255 20056576 AMD64/nbsl.exe
3615498117 16182784 x86/nbsl.exe
 

Recommended service state: Although ONLY the nbsl daemon (NetBackup Service Layer Service on Windows) needs to be stopped at the time this hotfix is applied, it is recommended that ALL services on the master server be shut down before installation.


Supplemental Materials

SourceETrack
Value3123064
Description

Memory leak in nbsl causing nbsl to crash.




Article URL http://www.symantec.com/docs/TECH201979


Terms of use for this information are found in Legal Notices