Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

NBDB upgrade from 7.1.0.3 to 7.5 fails

Created: 24 Oct 2013 • Updated: 25 Oct 2013 | 14 comments
This issue has been solved. See solution.

Hi.. I'm trying to upgrade NB 7.1.0.3 to 7.5 (then 7.5.0.6) on VCS 5.1 cluster installed on AIX 6.1.

During upgrade installer shows this error message:

Database [NBDB] can't be rebuilt.

Unable to create/upgrade the NB database. Refer to the
log file in /usr/openv/netbackup/logs/nbdb for more
information. Rerun /usr/openv/netbackup/bin/install_bp
when the problem has been resolved.

nbdb.log:

<16> rebuild_db: There are files in the way that cannot be removed. {i.e. NBDB.dbR, BMR.dbR,...}

 

NBCC and nbdb validation executed before installation showed no errors.

DNS and IP resolution works fine.

Maybe you had similiar issue ??

Operating Systems:

Comments 14 CommentsJump to latest comment

RamNagalla's picture

NBDB.dbR, BMR.dbR does not looks like the regular DB name, it has R added in the end..

ls -l /usr/openv/db/data   --> show us what all files that you are seeing in this location..

 

Marek Kedzierski's picture

-rwxr-xr-x    1 root     system          101 Jul 03 2012  .odbc.ini.az
-r--------    1 root     system     26218496 Jul 03 2012  DARS_DATA.db
-r--------    1 root     system     26218496 Jul 03 2012  DARS_INDEX.db
-r--------    1 root     system     26218496 Oct 24 13:49 DBM_DATA.db
-r--------    1 root     system     26218496 Oct 24 11:15 DBM_INDEX.db
-r--------    1 root     system     44126208 Oct 24 13:49 EMM_DATA.db
-r--------    1 root     system     52727808 Oct 24 13:49 EMM_INDEX.db
-r--------    1 root     system      2433024 Oct 24 15:04 NBAZDB.db
-r--------    1 root     system         4096 Oct 24 15:04 NBAZDB.log
-r--------    1 root     system      3698688 Oct 24 15:04 NBDB.db
-r--------    1 root     system         4096 Oct 24 15:04 NBDB.log
-rw-------    1 root     system          212 Oct 24 14:27 vxdbms.conf

mph999's picture

How did I know this issue was going to be an AIX ...

I have to go out, at first glance i have no idea what is going on, my first thought is that there is something wrong with one of the config files perhaps, vxdbms.conf, server.conf, bp.conf etc ... But then again, if the system was running previously I would have thought such a problem would stop the DB from starting.

Probably best to log a call on this - the system is currently down and so you need some fairly fast resolution.

Depending on how broken this is you might need to roll back, so try and have install media for the previous version available, along with the catalog dr files just in case.

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
watsons's picture

The size of NBDB.log does not look correct to me.. I mean 4096 is more like a default (see the NBAZDB.log) but if you have been running NBU for some time, NBDB.log should have a larger size.

I would think NBDB.log is corrupted, but unsure if it's before or during the upgrade.

To confirm, verify the issue in your /usr/openv/db/log/server.log  to check if NBDB.log can be processed during NBDB startup.

If confirm NBDB.log is the culprit, use this technote http://www.symantec.com/docs/TECH173414 to rebuild the transaction log. 

IMPORTANT NOTE: Running the above can be dangerous, log a support call if you don't feel comfortable doing that and they can guide you through.

It is also best to get ready with your last catalog backup, in case you need to recover these NBDB from the past.

Marek Kedzierski's picture

It was a size just after a catalog backup so log was truncated.

Marianne's picture

Hi Marek 

I apologize for the questions that I'm going to ask as I have great respect for your level of knowledge....

Please share preparation steps before you started with the upgrade.

In what state was NBU service group?
Hopefully just NetBackup resource offlined and not entire service group?

PS:

Hopefully high priority call logged with Support by now?

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Marek Kedzierski's picture

service group was freezed, NBDB works fine, I can start the database server, nbdb_ping reports alive, server.log shows that database starts without problems.

Marek Kedzierski's picture

Before upgrade I checked consistency using NBCC and NBDB structure using nbdb full validation.

mph999's picture

ANything in the server.log ?

The .R files are normal, part of the upgrade process.

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
Marianne's picture

Not sure if this is applicable to 7.5 as well - please have a look:

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Marek Kedzierski's picture

Ok, we've found a solution.

It was a problem with vxdbms.conf file. I don't know why but some entries were missing:

VXDBMS_NB_DATABASE = NBDB
VXDBMS_NB_PASSWORD = 2de0630e88a124027901a7c2f51d973f578efbf4df3d1b4b
AZ_DB_PASSWORD = Jj8mkP3sKTo=
VXDBMS_NB_FULL_KEYWORD = NBDB:2873833:1382602519:F
VXDBMS_NB_INCREMENTAL = NBDB.log.1
 

We added missing entries and upgrade ended successfully

VXDBMS_NB_SERVER = NB_netbackup
VXDBMS_NB_REMOTE_SERVER = NB_netbackup
VXDBMS_NB_PORT = 13785
VXDBMS_NB_DATABASE = NBDB
VXDBMS_AZ_DATABASE = NBAZDB
VXDBMS_NB_DATA = /netbackup/db/data
VXDBMS_NB_INDEX = /netbackup/db/data
VXDBMS_NB_TLOG = /netbackup/db/data
VXDBMS_NB_STAGING = /netbackup/db/staging

VXDBMS_NB_PASSWORD = 596b2bc5a31344abe6c1749d1684fad3b48892a55cab796c
AZ_DB_PASSWORD = Jj8mkP3sKTo=
VXDBMS_NB_FULL_KEYWORD = NBDB:2874248:1382686920:F
VXDBMS_NB_INCREMENTAL = NBDB.log.1
VXDBMS_AZ_INCREMENTAL = NBAZDB.log.1

SOLUTION
Marianne's picture

Great stuff! Thanks for the feedback!

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

mph999's picture

So. as I thought then ....

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805