On some UNIX master servers, Sybase is leaking semaphores and the EMM_DATA.db grows very large.

Article:TECH196040  |  Created: 2012-09-03  |  Updated: 2014-01-16  |  Article URL http://www.symantec.com/docs/TECH196040
Article Type
Technical Solution

Product(s)

Issue



In NetBackup version 7.5.0.3 and 7.5.0.4, UNIX master servers may see the NetBackup database files in /usr/openv/db/data grow very large.  Review of the the semaphore information on the system will show that the bpdbm pids associated with various database calls no longer exist, but the semaphore is still allocated. 


Error



When this occurs, running a script similar to the one below will show that many semaphores were last accessed by the bpdbm process.

    for semid in `ipcs -s | awk '/^0x/ {print $2}'`
    do
        ipcs -s -i $semid
    done

Review of the database files in /usr/openv/db/data will show that the files are growing rapidly.

Example:

On the 31st of July, the EMM_DATA.db file was huge, a rebuild of the database was accomplished.  The size of the file before the rebuild was approximately 10.61 GB:

# /usr/openv/db/data | grep EMM
total 14347528
-rw------- 1 root root 11392135168 Jul 19 14:15 EMM_DATA.db
-rw------- 1 root root    27725824 Jul 19 14:14 EMM_INDEX.db

After the rebuild, the EMM_DATA.db file was about 31Mg in size.

On the 16th of August, EMM_DATA.db was again very large, about 4.3 GB:

# ls -l /usr/openv/db/data | grep EMM total 8410712
-rw------- 1 root root 4627980288 Aug 14 20:17 EMM_DATA.db
-rw------- 1 root root   27201536 Aug 14 20:17 EMM_INDEX.db

 


Environment



NetBackup version 7.5.0.3 and 7.5.0.4

All UNIX platforms.


Cause



This issue is caused by a defect in Sybase. that is triggered if a SQLAnywhere client application exists abnormally.  If that happens the application will not clean up any System V semaphores that were allocated by the SA libraries. This affects all UNIX platforms. The issue is resolved in the following Sybase releases:

  •       12.0.1 3713
  •       11.0.1 2786

Solution



The formal resolution for this issue (Etrack 2873191) is included in the following release:

  • NetBackup 7.5 Maintenance Release 5 (7.5.0.5)

NetBackup 7.5.0.5 is now available - download links can be found in the Related Article linked below.

Workaround:
There is no work around for this issue. An Emergency Engineering Binary (EEB) bundle can be requested by contacting Symantec Technical Support and referencing Etrack 2873191.  

In addition to installing the EEB bundle, a database rebuild will need to be performed in order to decrease the EMM_DATA.db file to a reasonable size.  To rebuild the database complete the following steps:

1. Symantec strongly recommends to perform a full catalog backup prior to making any changes to your environment.

2. If on a clustered server, freeze the active node.  Example using the "Veritas Cluster Manager" command:

hagrp -freeze <NetBackup-group-name> -persistent

3. Stop NetBackup services:

/usr/openv/netbackup/bin/bp.kill_all

4. Make sure that all process are down except for pbx:

/usr/openv/netbackup/bin/bpps -x

5. Start NBDB database and nbemm

/usr/openv/db/bin/nbdbms_start_server

/usr/openv/netbackup/bin/nbemm &

6. Rebuild the database:

/usr/openv/db/bin/nbdb_unload -rebuild -verbose

 


Supplemental Materials

SourceETrack
Value2873191
Description

bpdbm is leaving many open semaphores on system resulting in max semaphores being reached.




Article URL http://www.symantec.com/docs/TECH196040


Terms of use for this information are found in Legal Notices