Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Catalog backup and restore test

Created: 11 Sep 2013 • Updated: 13 Sep 2013 | 24 comments
iaw's picture
This issue has been solved. See solution.

Hi,

I test run catalog backup and it completed succesfully.
When I tried, recover the catalogs it says error, failed to load DR image /backup/drfile/ (20)

The disaster recovery files are there in /backup/drfile
-rw------- 1 root root 1923 Sep  8 08:49 CatalogBackup_1378604926_FULL
-rw------- 1 root root 2054 Sep 12 21:11 CatalogBackup_1378995069_FULL
-rw------- 1 root root 2973 Sep 12 21:23 CatalogBackup_1378995803_FULL

Are there any missed step I need to follow?

Thanks,
IAW

Operating Systems:
Discussion Filed Under:

Comments 24 CommentsJump to latest comment

Marianne's picture

So, what exactly did you enter when you were asked for the DR file?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

iaw's picture

Hi, I tried twice, enter /backup/drfile and /backup/drfile/ , both gave the same error above.

mph999's picture

You have to enter the exact filename as well.
eg
/backup/drfile/CatalogBackup_11378995803_FULL

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
Marianne's picture

You need to add the full path to the DR file that you want to use for the restore.

You need to choose which of the 3 files you want to use - usually the latest one.

So, specify :

 /backup/drfile/CatalogBackup_1378995803_FULL

Have a look at this TN: http://www.symantec.com/docs/HOWTO72776

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

SOLUTION
iaw's picture

Hi, How long the catalog restore finish?
I think, I have small catalog size but it takes some time to recover catalog.

It stuck in starting up database .... already 30 mins

How to check catalog restore log?

Thanks,

Marianne's picture

How long did the backup take?

Did you follow requirements such as exact same hostname, same path installation, same patch level?

Restore log is in /usr/openv/netbackup/logs/user_ops/root/logs/

Look for file like this: Recover##### 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

iaw's picture

Backup just took few mins, 10-15 mins, I guess.

Yes, all exact the same.
It still stuck, below is the last message log.

15:51:14 INF - Source /usr/openv/db/staging/SEARCH_INDEX.db
15:51:14 INF - Destination /usr/openv/db/data/SEARCH_INDEX.db
15:51:14 INF - Creating /usr/openv/db/data/vxdbms.conf
15:51:14 INF - Creating /usr/openv/var/global/server.conf
15:51:14 INF - Creating /usr/openv/var/global/databases.conf
15:51:14 INF - Starting up databases

What to do?
Anyway to check the exact process, what's currenly running?

Thanks

Marianne's picture

Check current processes with:

bpps -x 

Please also check server.log in /usr/openv/db/log.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

iaw's picture

W. 09/12 15:51:13. Direct IO disabled for file '/usr/openv/db/staging/NBAZDB.log.1' because transparent hugepage support is enabled and madvise is not supported
I. 09/12 15:51:13.     Checkpointing...
I. 09/12 15:51:13. Starting checkpoint of "NBAZDB" (NBAZDB.db) at Thu Sep 12 2013 15:51
I. 09/12 15:51:13. Finished checkpoint of "NBAZDB" (NBAZDB.db) at Thu Sep 12 2013 15:51
I. 09/12 15:51:13. Recovery complete
I. 09/12 15:51:13. Database server shutdown automatically after log applied
I. 09/12 15:51:13. Disallowing new connections
I. 09/12 15:51:13. Shutting down databases
I. 09/12 15:51:13. Completing server shutdown
I. 09/12 15:51:13. Database server stopped at Thu Sep 12 2013 15:51

Look likes DB can't auto start after restore, is that correct?

Should I start the db manually? how?

Mark_Solutions's picture

Do you use NBAC?

If so you need to disable it before doing a catalog recovery of you do.

You then have to do an additional step to recover the NBAC details - see the Troubleshooting guide for full details

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

iaw's picture

What is NBAC?

I use netbackup GUI -> recover the catalog
I dont stop anything before start :(

What to do now? :(

Marianne's picture

NBAC is NetBackup Access Control.

Was that the entire server.log?
I was expecting to see a bit more?

You have not showed us 'bpps -x' output?
(Command is in /usr/openv/netbackup/bin)

You can try to stop everything:

/usr/openv/netbackup/bin/goodies/netbackup stop
 /opt/VRTSpbx/bin/vxpbx_exchanged stop

Then start:

 /opt/VRTSpbx/bin/vxpbx_exchanged start
/usr/openv/netbackup/bin/goodies/netbackup

Let us know what happens.
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

iaw's picture

Please find bpps output attached.
With this condition, can I stop all, by bp.kill then start all again?

AttachmentSize
bpps.txt 3.53 KB
mph999's picture

You ran a catalog backup and the a catalog recovery, was the recovery run to the same server that you made the backup from ?

If it was, did you create a blank EMM database, as described in the trobleshooting guide before you started.

I suspect you will need to follow the guide, and create a blank EMM database before trying the restore again.

Martin

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
Marianne's picture

Did you run 'bpps' or 'bpps -x'?

We need to see MM and PBX processes as well.

Do not stop anything yet - according to bpps output bprecover is still running.

There is this log (according to the output) that you can check:

/usr/openv/netbackup/logs/user_ops/root/logs/jbpRestore-20130912155027.log

Other than the one defunct process, all NBU processes seem to be up and running - still need to see MM and PBX processes.

Not sure what else to check at this point....

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Mark_Solutions's picture

I am with Martin - we need more info:

Please tell us more :

Is this the original server?

Does it have the same name?

Did you re-install?

Same NetBackup Version?

Same installation path?

Tell us exactly what you are doing and where you are testing this recovery

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

iaw's picture

I was run bpps -a, attached is the bpps -x

@Martin,
Please tell us more :

Is this the original server? Yes

Does it have the same name? Yes

Did you re-install? NO

Same NetBackup Version? Yes

Same installation path? Yes

Tell us exactly what you are doing and where you are testing this recovery
I just test run recovery catalog from Netbackup GUI, in master server.

I dont create a blank EMM database either before start.

AttachmentSize
bpps_x.txt 3.88 KB
Marianne's picture

As per Martin's excellent post - if you are restoring back to the same server, you should create a blank database.

We never knew that you were restoring back to the same server - most people test this at their DR site or on a test system - not their live, production server.

The process to create a blank database is described in Chapter 5 of the Troubleshooting Guide.

You did not mention your NBU version - you can find links to all NBU manuals in 'Handy NBU Links' in my signature.
(But Symantec KB site is down at the moment)

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

iaw's picture

My NBU version is 7.5.0.6
Should I wait?

What is the worst case I must do?

Marianne's picture

You will probably have to stop everything, recreate the EMM db as per the manual and then start the catalog recovery again.

Just create all logs before you start all over - 
bprd
bpdbm
bptm
bpdm (if restoring from Disk stu)
tar

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Mark_Solutions's picture

As this is titled "test" could you explain why this was done on yoru live system - did it actually have an issue?

Also, any reason for not using the Catalog Recovery Wizard from the Admin Console?

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

iaw's picture

I leave it till the catalog recovering completed.
It looks like my nbu is running normally now.

Thanks all.
 

Marianne's picture

Curious to know if there were any more entries in the Recover## or jbpRestore-## log files?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

iaw's picture

15:51:14 INF - Creating /usr/openv/var/global/databases.conf
15:51:14 INF - Starting up databases
22:41:33 INF - Database recovery successfully completed

22:41:33 INF - Recovery successfully completed
22:41:34 INF - Catalog recovery has completed.
22:41:34
WRN - NetBackup will not run scheduled backup jobs until NetBackup is restarted.
 

It took around 8 hours to recover catalog with size ~400MB ?
Is it normal time?
 

Below are my catalog backup files.

-rw------- 1 root root 379027456 Sep 12 21:26 app1_1378995803_C1_F1.1378995803.img
-rw------- 1 root root      4096 Sep 12 21:26 app1_1378995803_C1_F1.1378995803.info
-rw------- 1 root root      4096 Sep 12 21:26 app1_1378995803_C1_HDR.1378995803.img
-rw------- 1 root root      4096 Sep 12 21:26 app1_1378995803_C1_HDR.1378995803.info
-rw------- 1 root root     98304 Sep 12 21:26 app1_1378995803_C1_TIR.1378995803.img
-rw------- 1 root root      4096 Sep 12 21:26 app1_1378995803_C1_TIR.1378995803.info