Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

EMM database is going down repeatedly - Help!

Created: 13 Mar 2013 • Updated: 19 Mar 2013 | 10 comments
This issue has been solved. See solution.

Netbackup 7.5.0.4 Windows server 2008r2

 

Hello all,

This morning is the second morning is a row in which my EMM database has gone unresponsive. In both cases, the CPU, memory, and drive space on the master server look to be fine, but in the application even viewer I am seeing that SQLANY64 shows "configuration update thread out of memory" and I cannot get the database to start back up. I can access the NetBackup DataBase Administration console, but most of the options are greyed out.

 

In the server.log file, I see this -

I. 03/13 09:59:13. Cache size adjusted to 529920K
I. 03/13 09:59:46. Starting checkpoint of "NBDB" (NBDB.db) at Wed Mar 13 2013 09:59
I. 03/13 09:59:48. Finished checkpoint of "NBDB" (NBDB.db) at Wed Mar 13 2013 09:59
I. 03/13 10:00:13. Cache size adjusted to 514676K
I. 03/13 10:01:13. Cache size adjusted to 480588K
E. 03/13 10:01:59. *** ERROR *** Out of memory[NBDB] (11.0.1.2789)
Out of memory

Have any of you seen this issue, and dealt with it successfully?

What other information can I provide?

Thank you!

 

Todd

Operating Systems:

Comments 10 CommentsJump to latest comment

RamNagalla's picture

E. 03/13 10:01:59. *** ERROR *** Out of memory[NBDB] (11.0.1.2789)

 

the Cache memory that is allocated to the netbackup is not enough...

 

see if you could add up some space with below command..

<install_path>\NetBackup\bin>NbDbAdmin.exe

 

SOLUTION
Toddman214's picture

Thanks Nagalla. I did run that. Thats the Netbackup DataBase Admin console that I mentioned in my original post. Most of the options are greyed out, and it shows ? marks in stead of data sizes.

 

Is it possible that the SQLANY64 version 11 used by NetBackup 7.5.0.4 is sharing memory with other things, whereas whatever version of SQLANY used by Netbackup 7.1.0.4 (my porevious version) perhaps was not shared? I dont understand why this just suddenly started happening.  

Toddman214's picture

Ok, for some reason, the EMM database came back online, and I can see everything now. While it was up, I was finally able to view/change settings in the database admin console. The recommended setting for a SMALL organization is 200mb paging size. We were set at 100mb! LOL I guess that's a default setting or something. I back up between 30 and 45tb of data per night across 1300+ clients. So, I upped the number to equate to at least a Medium sized organization - 768mb. Hopefully this will prevent this from happening again, but I dont want to mark this as a solution just yet.

RamNagalla's picture

go ahead and set it for Large size organization..  :-)

Will Restore's picture

30 and 45tb of data per night across 1300+ clients is Medium size organization?  indecision

Will Restore -- where there is a Will there is a way

Dyneshia's picture

Was there a recent upgrade ?  What are the resutls of  : nbdb_admin -validate NBDB -full

Toddman214's picture

This is my first job as a backup administrator, so I dont really have a basis for comparison. But, I would think compared to some environments, we might be considered medium sized. Would we be considered on the larger side?

Dyneshia's picture

Are you using McAfee ? If so, This tech note describes a best practice to configure McAfee McShield for optimal backup performance
http://www.symantec.com/docs/TECH124795
http://www.symantec.com/docs/TECH56658

 

You may need more cache memory for the EMM database.  Please upload your server.conf

 

Confrim TCP Chimney Offload is disabled. - http://www.symantec.com/docs/TECH60844 http://www.symantec.com/docs/TECH145791

 

Anshu Pathak's picture

The strange thing in your environment is ....In NetBackup 7.5 maximum databaes memory cache value has been increased to 1GB (default) however in your case it is showing 512MB.

Check and modify the "-ch" value in server.conf file

 

On UNIX systems:
/usr/openv/var/global/server.conf
 
On Windows systems:
install_path\Program Files\Veritas\NetBackupDB\CONF\Server.conf
 
What value to set:
-ch 1G (for systems with up to 1M images)
-ch 4G (for systems with up to 3M images)
-ch 6G (for systems with up to 5M images)
-ch 12G (for systems with up to 10M images)
 
As you are running backups for 1300+ clients I am assuming that you will have more than 1million images and hence "-ch" value in your environment should be more than 1GB
 
Determine the number of images on your system:
On UNIX systems:
/usr/openv/netbackup/bin/admincmd/bpimagelist -idonly -d
"01/01/1970 00:00:00" | wc -l
 
On Windows systems:
install_path\NetBackup\bin\admincmd\bpimagelist -idonly -d
"01/01/1970 00:00:00" | %SystemDrive%\Windows\System32\find.exe
/C " ID: "

 

Anshu Pathak

Toddman214's picture

Upping the paging for the EMM to the maximum (as we have many millions of images) seems to have stopped the issue. The problem was that because the db was totally unresponsive, I could not access the administration console until after I restarted services, to adjust the settings I needed to set. After restarting services, I was able to make my changes.