During an installation of the NetBackup 7.5.0.7 maintenance release, clustered master server NetBackup processes core dump or application fault.

Article:TECH213788  |  Created: 2014-01-08  |  Updated: 2014-05-27  |  Article URL http://www.symantec.com/docs/TECH213788
Article Type
Technical Solution


Issue



During the installation of the NetBackup 7.5.0.7 maintenance release on a clustered NetBackup master server, numerous NetBackup processes may core dump or generate application faults.

ALL of the following conditions must be met for this symptom to occur:

  • The NetBackup master server must be a NetBackup 7.5.0.7 upgrade candidate
  • with either CLUSTER_NAME or REQUIRED_INTERFACE configured
  • and ANY_CLUSTER_INTERFACE = YES (or 1 or ON)
  • and PREFERRED_NETWORK is not configured

To check the values of these configuration parameters, execute the following commands:

# bpgetconfig CLUSTER_NAME
# bpgetconfig REQUIRED_INTERFACE
# bpgetconfig ANY_CLUSTER_INTERFACE
# bpgetconfig PREFERRED_NETWORK

This issue is not present in NetBackup 7.6 and later.

Note: There is a possibility that this issue may be encountered on servers which are not master servers if all of the configuration items noted above are present.  (This is a very unlikely configuration.)


Error



Snippet from master:
# cat -s /usr/openv/pack/pack.history

...

Installing required pack, NB_7.5.0.7, now.
=====================================================
NB_update.install: Fri Dec 27 14:08:30 MET 2013
=====================================================

Install pack NB_7.5.0.7 Fri Dec 27 14:08:30 MET 2013 Rev. 1.46.8.16

Running preinstall script.
See /usr/openv/pack/pack.history for more details.
/var/tmp/UNIX_7.5.0.7/VrtsNB_7.5.0.7.preinstall: Running. Hardware/OS Type=Solaris/Solaris10
Segmentation Fault - core dumped
Segmentation Fault - core dumped

Machine nbmaster1 is a master server, and it is not the EMMSERVER.

Saving files for rollback.

Extracting files out of /var/tmp/UNIX_7.5.0.7/VrtsNB_7.5.0.7.solaris.tar.gz.
Installing SYMCnetbp
More details regarding SYMCnetbp can be found in file
/tmp/install_netbppkg_trace.20420 on nbmaster1.

Running postinstall script.
See /usr/openv/pack/pack.history for more details.
/var/tmp/UNIX_7.5.0.7/VrtsNB_7.5.0.7.postinstall: Running. Hardware/OS Type=Solaris/Solaris10
Segmentation Fault - core dumped
Segmentation Fault - core dumped

Machine nbmaster1 is a master server, and it is not the EMMSERVER.

Copied files to /kernel/drv/sparcv9.

Doing add_drv of the sg driver

Leaving existing sg configuration.
If you wish to update the configuration you need to
rm -f /kernel/drv/sg.conf
and rerun /usr/openv/volmgr/bin/driver/sg.install.


Running /usr/openv/netbackup/client/Solaris/Solaris10/pdinstall to update PDDE.
Checking for SYMCpddea package.
SYMCpddea package is installed.
NetBackup Deduplication software is installed, overwriting binaries.
Existing version:
7.0005.0013.0531
+ Extracting PDDE agent package (/usr/openv/netbackup/client/Solaris/Solaris10/pddeagent.tar.gz)...
Using /tmp/pdde_pkg to store temporary package data.
Unpacking SYMCpddea package.
Checking for pre-existing SYMCpddea package.
Removing pre-existing SYMCpddea package.
Installing SYMCpddea package.
+ Extracting PDDE server package (/usr/openv/pddeserver.tar.gz)...
Using /tmp/pdde_pkg to store temporary package data.
Unpacking SYMCpddes package.
Checking for pre-existing SYMCpddes package.
Removing pre-existing SYMCpddes package.
Installing SYMCpddes package.
Keeping existing /usr/openv/lib/ost-plugins/pd.conf
Saving a copy of the uninstall script to /opt/pdde/pddeuninstall.sh
Starting setup for PDDE script
pdregistry.cfg exists.
removing /etc/default/pdde
Done setup for PDDE script
Checking to see if the PDDE configuration needs upgrading
Agent config file path found: {/etc/puredisk/Agent.cfg}
Making changes to /usr/openv/lib/ost-plugins/pd.conf
RESTORE_DECRYPT_LOCAL already in pd.conf
PREFETCH_SIZE already in pd.conf
META_SEGKSIZE already in pd.conf
CLIENT_POLICY_DATE already in pd.conf
Making changes to /usr/openv/lib/ost-plugins/pd.conf
FIBRECHANNEL already in pd.conf

PDDE install finished successfully.
Version now installed:
7.0006.0013.1122
Full PDDE installation log saved to: /var/log/puredisk/2013-12-27_14:34-pdde-install.log

Segmentation Fault - core dumped
Segmentation Fault - core dumped
Segmentation Fault - core dumped
Segmentation Fault - core dumped

The following command failed, BMR may not function properly.
bmrsetupmaster -patch


ERROR: Unable to stop Sybase.
Segmentation Fault - core dumped
Segmentation Fault - core dumped

ERROR: The nbazd daemon is not running; therefore,
NetBackup Access Control could not be upgraded. Run
"/usr/openv/netbackup/bin/admincmd/bpnbaz -Upgrade"
after this installation completes.

ERROR: Unable to stop Sybase.
Installation of pack NB_7.5.0.7 completed Fri Dec 27 14:08:30 MET 2013 Rev. 1.46.8.16.

Checking LiveUpdate registration for the following products: NB CLT
This may take a few minutes.

Product NB is installed and will be registered.
Product CLT_Solaris-X64 is installed and will be registered.
Product CLT_Linux-X64 is installed and will be registered.
Product CLT_Linux-PSERIES is installed and will be registered.
Product CLT_Linux-ZSERIES is installed and will be registered.

Updating LiveUpdate registration now...this may take some time.

...

This example was generated from a test environment on a Solaris 10 clustered master (Solaris Cluster 3.2).

When running pstack of one of the core files, it was discovered that the crash occurred when using /opt/openv/lib/libnbbaseST.so.  All processes core dump at nbconf_free_preferred_network_list + 18 .

There was no core dump when the previous (7.5.0.6) libnbbaseST.so file was used after restarting vnetd.  But NetBackup coredumps with the newer (7.5.0.7) one.

Note: On a Windows server, the affected library is libnbbase.dll; however, the visible symptoms will be Application Faults in daemons/processes utilizing this library as opposed to general messages about the library itself.


Solution



This symptom is not present in NetBackup 7.6.  The preferred resolution is to upgrade the clustered NetBackup master server to a current maintenance level of the NetBackup 7.6 release.

The following workaround will allow the NetBackup 7.5.0.7 maintenance to be installed and run on a clustered NetBackup master server:

1.  Ensure the following fictitious hostname resolves without delay on the host, add a hosts file entry if necessary.  The IP to which it resolves should not be assigned to any host.  NetBackup will not contact the hostname, but it needs to be resolvable to prevent excessive verbose debug logging.

On UNIX/Linux:

# bpclntcmd –clear_host_cache
...wait 10 seconds or longer...
# date
# bpclntcmd –hn see.TECH213788.at.symantec.com
# date

On Windows:

> bpclntcmd –clear_host_cache
...wait 10 seconds or longer...
> time <NUL
> bpclntcmd –hn see.TECH213788.at.symantec.com
> time <NUL

2.    Add this setting to all nodes of the clustered NetBackup Master Server configuration:

echo PREFERRED_NETWORK=see.TECH213788.at.symantec.com MATCH see.TECH213788.at.symantec.com | bpsetconfig

3.  The NetBackup 7.5.0.7 maintenance release may now be installed without issue.

4.  This PREFERRED_NETWORK parameter must remain set until the clustered master server is upgraded to NetBackup 7.6.  This issue is not present in 7.6 and later.  After upgrading to NetBackup 7.6, this entry should be removed from all nodes.


Supplemental Materials

SourceETrack
Value3402176
Description

During NetBackup 7.5.0.7 maintenance installation, clustered master server NetBackup processes core dump or application fault in function nbconf_free_preferred_network_list.



Article URL http://www.symantec.com/docs/TECH213788


Terms of use for this information are found in Legal Notices