Video Screencast Help

Restore Catalog from Tape

Created: 16 Jul 2012 • Updated: 23 Jul 2012 | 14 comments
This issue has been solved. See solution.

Have a complex issue on our NBU 7.5 Master Server. Our Master Server crashed. Was running on a Dell PE 2950, with an MD1000 attached. Something happened with the OS of the Master Server and it would no longer boot into Windows. So I grabbed a Dell PE 1950 and configured it for Windows, same name, same IP's, same level of NBU 7.5.

Previously, long before any issues, had moved the images catalog directory to another drive, on the MD1000. Used this procedure to do this: http://www.symantec.com/business/support/index?page=content&id=TECH68160

After the setup on the new server, I again changed the catalog directory to point to this MD1000. After this, and a reboot, I could see the policies from before the crash, but there are still issues with media servers missing, no catalog information if I try to do a restore, Volume Pools missing. So it seems after the reinstall and pointing it to the catalog directory, that not everything is visible again.

I do have the old server that crashed, and using a recovery boot disk I can access the old OS partition, the files still exist.

My question: are there any files I can copy from the crashed server to the new server that, in addition to the catalog information on the MD1000, will get me back in a working condition with the catalog information available?

Also, because it was what I tried first, it seems our catalog backup has been having issues. We do it to disk and tape (Multiple Copies), but for some reason the BTD only contains the differential backups for the last few weeks, no full backup it seems. And I'm working on recalling backup tapes to see if any of those have a full backup, but it's possible we don't have a full catalog backup on tape. But since all the files are available, either on the old crashed server or the MD1000, I'm hoping there is somthing we can do.

I have a ticket with Symnatec, but as this is quite urgent I wanted to ask here as well.

Thanks for your help!

Comments 14 CommentsJump to latest comment

mpw613's picture

Forgot to mention, OS is Windows 2008 R2 SP-1

mph999's picture

So if understood correctly ...

You have rebuilt a master server with NBU installed at exactly the same version.  On this new server you have configured the 'catalog' path to match the old sever.

Thinking you mught need to pull the DR file over (as you will have done) and run the nbcatsync

Kinda safe, you can do a dry run, if it says no changes, it ain't that ...

If it wants to make a changes, that could be the issue, or at least part of it ...

nbcatsync -sync_dr_file dr_file_path -dryrun

 

Other ideas ...

You could copy across ...netbackup\db\images

This is ok, as if using nbcatsync, you are only meant to use a partial catalog recovery anyway.

Problem you will/ might have, is the media DB maight be out of sync with the images DB.  Recommend to run nbcc to rebuild this.

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mpw613's picture

Trying nbcatsysc to the backup file location is giving errors, "nbcatsync: EXIT Status 95" followed by "nbcatsync: requested media id is not assigned to this host in the EMM database (95)"

I'm thinking we're missing on our disk storage (which is on a another physical server) some of the catalog files. I have the last 3 weeks tapes fro the catalog backup, but I'm not sure the proper procuded to recover the catalog if the backups spans a few tapes.

As for the netbackup\db\images folder. This is already available int eh new configuration. Which this is what I am most confused about, I have the policy information for all the policies, whcih is stored in this lcoation, but I thought it would also contain the catalog information and after the reconfig and changing the DB to this location, the policy information, and catalog information would be recovered. Apparently not.

Currently I'm importing two of the three catalog tape backups, one already imported.

Thanks

mph999's picture

OK - completely scrap my reply - the clue was in the thread "Recovery from TAPE"

My apologies, for spome reason I was thinking this was DISK (hey, it's Monday).

Further, you would need to have images DB copied acrosss anyway to use nbcatsync ...

So, let us start again .

1)

If you have the catalog tapes, no need to impotrt them, just run the catalog recovery wizard.

Have you tried this ?

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mpw613's picture

I think this is where I'm having issues. I've searched but have not found a good article/answer on doing the catalog recovery from tape. In the catalog recovery wizard the first step is to point to the Disaster Recovery File. I'm assuming this is either a) on the tapes or b) since I still have access to the crashed server C drive, a file on this old server. Guess it could be in the netbackup\db\images folder under the Master Server name, but not sure which file to use. At least so far the ones I've pointed to do not work. Same errors.

Wouldn't I have to import the tapes since the catalog is missing? Or is there something "special" about a catalog backup to tape since if it is ever needed obviously it would not be in the current catalog?

 

Thanks again!

mph999's picture

OK.

The DR file is emailed out or copied to a directory as specified in the Catalog Policy.

We will come back to this is shortly.

The other requiredment is you have the catalog tapes.

Catalog recovery goes like this :

1.  Build new server, NBU install the same, install path the same.

2. Configure tape drives/ library

3. Put catalog tapes in library

4.  Run inventory

5.  Run cartalog recovery wizard.

6. Supply DR file location.

7. Select Full recovery 

Let me look at the DR File issue ...

Martin

 

 

 

 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mph999's picture

OK, the DR file is contained in the catalog, but we shouldn't be coping it from there ... no idea if it would work.  In theory, it probably would, as the DR file is effectivly the header file for the backup, but if it is EXACTLY the same, I would have to test.

However, if you have the old server, then look in this dir ...

<install> \veritas\netbackup\db\class\<catalog policy name>\

You should see a file 'catdrinfo'

In that file, you will see the location of the PATH the DR file is copied to ...

Eg.  PATH /netbackup/catback

It will also show the email address it is copied to (if set)

Does this allow you to track down the DR file ?

Martin

 

 
 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mph999's picture

Hmm, Seems the DR file is the same as the catalog header file.

But, as the header file is located in the EMM database, you're going to need NBU running to extract it.

Think this is a no go ...

First thing first then, was the DR file emailed out, or is it obtainable from the server as described above.

 

martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mpw613's picture

Our DAS, the MD1000 housing the DB for NetBackup has this location, \veritas\netbackup\db\class\<catalog policy name>, which does show our online path to the catalog. However, this location only shows the names of the files as Policy-#######_INCR. There is no _Full.

I'm afraid this storage location may have gotten "cleaned up" due to space issues, and the last FULL copy of the catalog was removed before it was checked that it could be removed. Yet another issue, I know, that one has been resolved going forward!

And yet another issue from the past, a couple of weeks ago, the Master Server had a hardware failure and the entire catalog was restored from disk, and this went fine. But after that restore, the e-mail was not set up properly, so no e-mails where going out with catalog DR information for the past two weeks.

At this point the only thing we seem to have is the catalog backup on multiple tapes. I believe going back to the last two weeks (of our 4 week tape rotation schedule) I should have a full catalog backup on the tape from two weeks ago, and incrementals up until the Master Server crashed.

So I guess I'm stuck again recovering the catalog from the tape copy. In the Troubleshooting manual, I get to step 6 on page 210 which states "Make available to Netbackup the media that contains the catalog backup". Which I have on tape. I'm wondering if I need to do a file level restore of the data on the catalog tapes and these will include the full Catalog file for the DR?

But not sure really at this point what to do about the catalog on tape restore.

mph999's picture

How many tapes are there in total ?

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mph999's picture

If you can identify the incremental (the last incremental) the DR file will contain details for the other tapes needed.

If you can get to the header files (as you explained you can) - then taking the last catalog backup with the latest ctime and following this TN http://www.symantec.com/docs/TECH47797 should allow a catalog recovery.

 

If this doesn't work, all I can think to do is ...

Option 1

Phase 1 import the tapes ...

Follow this 

http://www.symantec.com/docs/TECH47797

To 'find' the DR file, which = the latest catalog backup header file.

Option 2

Phase 1 / 2 import all the tapes and recreate all catalog info.  Recover policy info from the backups when this done.

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mpw613's picture

Worked with Symnatec Support over the phone and this is a summary they sent that, well, sums up what it took to get it working:

-- To recover and resynchronize the NetBackup relational databases (NBDBs),tried bprecover -r -nbdb.
still same issue.

-- checked latest catalog backup and found latest catalog backup on 14th July.
-- Seems,due to the latest catalog backup on 14th July i.e.after crash,while recover NBDB,it is capturing information from 14th July's full catalog information.
-- Expired the catalog backup images which has been taken after server crash,so that bprecover can recover the NBDB before the server crash.
-- Run the bprecover -r -nbdb to recover NBDB.
-- Able to see the volume pools,storage units and groups.
-- Checked with Restore and Image cleanup.Working fine.

 

At first could not access the Media server, but after a reboot of the entire environment this was OK as well.

Thanks for all your help mph999 Martin! Very much appreciated.

SOLUTION
mph999's picture

How did it take a catalog backup after the crash, I thought it was 'down'.

No matter, the important thing is you have it back.

Please ensure you always copy/ mail of the DR files - it makes recovery a lot easier.  Also, if you can, I recommend a full catalog backup daily.

If possible, please make the most helpful post as the solution, if you liked any of them ...

Regards,

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mpw613's picture

After the reinstall, pointing it to the MD1000 all the policies were there, but not all the settings were recovered. But since the catalog backup went to a netowrk location it did run again as scheduled. So this needed to be removed to get the correct catalog recovered.