NetBackup 7.0GA and 7.0.1 Enterprise Media Manager database (nbemm) becomes unresponsive under load

Article:TECH153520  |  Created: 2011-02-16  |  Updated: 2011-10-21  |  Article URL http://www.symantec.com/docs/TECH153520
Article Type
Technical Solution


Issue



On NetBackup 7.0GA and 7.0.1 master servers, the Enterprise Media Manager service (nbemm) may become unresponsive under a heavy load, causing most NetBackup activity to stall.


Error



The most prominent indicator of this issue is that the nbdb_ping command does not return results and appears to hang. This occurs intermittently, the issue can take one hour or more after a startup of NetBackup processes to become visible. After the issue has occurred, a restart of Netbackup services will clear it temporarily.

Normally, the command nbdb_ping would either return "alive and well" or "not available" status depending on whether the NetBackup Relational Database service is running or not. However, on the master server affected by this issue, this command would hang with no output while the service is running.

The dbsrv11 process may also consume an unusually large amount of RAM while not responding to nbdb_ping command.


Environment



This issue can affect NetBackup 7.0GA and 7.0.1 master servers if all of the following conditions are met:

  1. The Operating System is either Windows or Linux;
  2. The number of Logical CPU cores is larger than 24. Note that hyper-threading CPU cores count as two. For example, a system with 4 Intel Xeon quad-core CPUs with hyper-threading enabled will show 4 x 4 x 2 = 32 CPUs and therefore could be affected.

To verify whether the system can be affected, check the file <Install Directory>\NetBackupDB\log\server.log for the lines similar to:

SQL Anywhere Network Server Version 11.0.1.2250

32 logical processor(s) on 8 physical processor(s) detected.

If the version of SQL Anywhere Server is 11.0.1.2250 and the number of logical processors detected by the SQL Anywhere Server is more than 24, the system could be affected.


Cause



The issue is caused by a known bug in this particular version of a Sybase SQL engine included with NetBackup 7.0GA and 7.0.1 that causes it to stop responding under load.

Sybase bug report:

http://search.sybase.com/kbx/changerequests?bug_id=646169


Solution



Formally resovled in NetBackup 7.1GA

To work around the issue, reduce the number of logical processors the server uses by specifying -gtc <num>, where num is less than or equal to 24.

Windows:

  1. Edit, the [Install Path]\NetBackupDB\CONF\server.conf file.(Windows) [
  2. On the line that includes "-gd DBA -gk DBA -gl DBA" add -gtc with the number of CPUs/cores desired just before the "-gd DBA".
    • For example, -n NB_MASTER -x tcpip(LocalOnly=YES;ServerPort=13785) -ct+ -gtc 24 -gd DBA -gk DBA -gl DBA -gp 4096 -ti 0 -c 25M -ch 500M -cl 25M -zl -os 1M -o "C:\Program Files\Veritas\NetBackupDB\log\server.log"
       
  3. Ensure there are not any jobs running, and restart the NetBackup Services.

Unix:

  1. Edit the /usr/openv/var/global/server.conf file.
  2. On the line that includes "-gd DBA -gk DBA -gl DBA" add -gtc with the number of CPUs/cores desired just before the "-gd DBA".
    • For example, -n NB_nbupub1
         -x tcpip(LocalOnly=YES;ServerPort=13785)  -gp 4096 -gtc 24 -gd DBA -gk DBA -gl DBA -ti 0 -c 25M -ch 500M -cl 25M -zl -os 1M  -o /usr/openv/db//log/server.log
      -ud
  3. Ensure there are not any jobs running, and restart the NetBackup Services.

 


Supplemental Materials

SourceETrack
Value2252693
Description

EMM becomes unresponsive under load


SourceETrack
Value2096107
Description

Upgrade Sybase EBF



Article URL http://www.symantec.com/docs/TECH153520


Terms of use for this information are found in Legal Notices