Video Screencast Help

Does incremental catalog recovery work?

Created: 18 Jun 2012 • Updated: 20 Jul 2012 | 18 comments
This issue has been solved. See solution.

Today I was planning to migrate my old Master Server to a nice new one.  I had pre-prepared the new server; same name, ip etc and pre-installed same version of NetBackup on it.

I then performed a cumulative hot catalog backup on the old Master Server and shut it down, moved the cables and restarted the new one.

Confirming all was ok and I could see the required tapes I then initiated a catalog recovery from tape using the latest recovery file (from the incremental).  All looked ok; it started, recovered 3 lots of information and then finished very quickly.  I soon realised it had not recovered the "full" files; just the incremental files.  I was missing a majority of my image catalog.

Symantec are looking at this but wondering if others have seen or heard of this?

NetBackup version 7.1.0.4, RHEL 6.

Comments 18 CommentsJump to latest comment

mph999's picture

Yes, they should work.

Apart from the staging process of the EMM DB, there is nothing special about a catalog backup.  The tapes can be browsed like any other tape if you wish (set policy type to nbu-catalog in BAR gui).

The DR file should contain the details of the tapes used, and in fact the DR file is just a copy of the image header file, which is all that is required to run a restore.

What is the case number ?

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
thesanman's picture

Case number is 418-439-612

What I find interesting is that there's no reference to the Full tape number in the DR file; yes it references the full .f file (PREV_BACKUP_IMAGE Catalog-Backup_1339967083_FULL.f) but no indication on what tape that is.

There's no errors during the restore procedure; it simply doesn't go and restore the "files" from the full image.

It performs 3 restore jobs; I woudl expect 4 if the full was referenced.

NBU v7.5.0.6 Master and Media servers on RHEL 5/6 & Win2008; SAN based LTO3, 4 and 6 tape libraries
Linux, Solaris, Windows and OpenVMS clients.
PureDisk, SLP, VMware, HyperV, Oracle, Netezza, SQL/Server,

mph999's picture

This is what I believe should happen ... with Incremental Catalogs Recovery.

If an online catalog backup policy includes both full backups and incremental backups, the disaster recovery email may indicate either a full backup or an incremental backup for recovery. An incremental backup recovery completely recovers the entire catalog because it references information from the last full backup. You don't need to first recover the last full catalog backup, then follow with subsequent incremental backups.

I'll try it on my server when I get a few mins.

 

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
SOLUTION
revaroo's picture

You are correct in this assumption, it does reference back to the full.

mph999's picture

Here is the business bit of the header file.

This was one Full backup followed by one diff inc.  We see in the DR_MEDIA lines that the media id's are mentioned. (R0TP01 and R0TP02 )

 

 
# FRAG: c# f# K rem mt den fn id/path host bs off md dwo f_flags desc exp mpx rl chkpt rsm_nbr seq_no media_subtype keep_date copy_date fragment_state data_format slp_index_num
FRAGMENT 1 -1 120 0 2 12 3 R0TP02 womble 65536 4530 1340102160 0 0 *NULL* 0 0 0 0 0 0 1 0 0 1 0 0
FRAGMENT 1 1 448 0 2 12 2 R0TP02 womble 65536 4521 1340102160 0 0 *NULL* 1341311890 0 65537 0 0 0 1 0 1340102312 1 1 0
#DR_MEDIA_REC: ver rtype mtype host c# p# classes classes kbytes nimg vimg hsize off subtype client_type sched_type run_time id/path frag_id media_desc bcode den
DR_MEDIA_REC 2 0 2 womble 1 1 0 287904 1 1 1024 4503 1 7 0 1340099915 R0TP01 *NULL* -------- 12
DR_MEDIA_REC 2 0 2 womble 1 1 0 287904 1 1 1024 4503 1 35 0 1340100042 R0TP01 *NULL* -------- 12
DR_MEDIA_REC 2 0 2 womble 1 1 0 288640 1 1 1024 4514 1 7 1 1340100317 R0TP00 *NULL* -------- 12
DR_MEDIA_REC 2 0 2 womble 1 1 0 577056 2 2 1024 9023 1 7 1 1340100633 R0TP00 *NULL* -------- 12
DR_MEDIA_REC 2 0 2 womble 1 1 0 289056 1 1 1024 4521 1 7 1 1340102160 R0TP02 *NULL* -------- 12
DR_MEDIA_REC 2 0 2 womble 1 1 0 289624 3 3 1024 4534 1 35 1 1340102290 R0TP02 *NULL* -------- 12
 
Do you see similar ?

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
thesanman's picture

Yes, I see DR_MEDIA_REC entries for the various catalog backups that I have.  In particular, for the process today, entries for the full and cumulative backups.

I'll have to work though a similar restore process in my QA environment tomorrow.

NBU v7.5.0.6 Master and Media servers on RHEL 5/6 & Win2008; SAN based LTO3, 4 and 6 tape libraries
Linux, Solaris, Windows and OpenVMS clients.
PureDisk, SLP, VMware, HyperV, Oracle, Netezza, SQL/Server,

thesanman's picture

Went through a similar process using my QA environment and yes, I can see during the 'file' component of the restore process it applies the cuml, then the full backup files from the catalog backups.  This I did not see for my new Master Server yesterday.

All very strange

NBU v7.5.0.6 Master and Media servers on RHEL 5/6 & Win2008; SAN based LTO3, 4 and 6 tape libraries
Linux, Solaris, Windows and OpenVMS clients.
PureDisk, SLP, VMware, HyperV, Oracle, Netezza, SQL/Server,

mph999's picture

Just tested this, restored fine using both separate tapes on my server.

I suspect that there was something wrong with the DR file though exactly what I can't image at the moment.

Can you try and recover the same catalog on your test server, using the same DR file.

We know this separate environment of yours can restore, as you have just done it, so this would narrow down the problem quite nicely.

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
thesanman's picture

Was thinking exactly the same; problem is my primary recovery tapes are at another site.  The 2nd copies are local which could be easily retrieved whereas the primaries cannot.  All are on SAN attached libraries.

Am trying to think of a way to retrieve the tapes from my DR site.

Another way to do the switch might be to simply synchronise a copy of local /usr/openv to the new Master Server as (say) /usr/openv.new  Then simply shutdown current Master Server; rename new Master server /usr/openv away and rename the /usr/openv.new to usr/openv and restart NetBackup.  Might have to rebuild the local tape config but other than that the rest should be the same.

This was the way we renamed NBU Master before with help from symantec to merge the EMM databases which I don't need to do this time as the name is the same.

Doesn't fix this restore issue I have BUT it get's me on the new Master Server.

Thoughts?

 

NBU v7.5.0.6 Master and Media servers on RHEL 5/6 & Win2008; SAN based LTO3, 4 and 6 tape libraries
Linux, Solaris, Windows and OpenVMS clients.
PureDisk, SLP, VMware, HyperV, Oracle, Netezza, SQL/Server,

mph999's picture

If you happy just to get things working then I guess you 'might' not be too concerned about this cause.

Persoanlly, I would want to check this new server can do a cat recov. off and inct backup, as you may have to do this for real in the future.

If, the issue really is with the DR file off the old system, then providing the catalog backups on the new server work, it doesn't matter, you're getting rid of the system after all - however, your thoughts may differ.

You can recover from the 2nd copy of the tapes ...

We can't merge the EMM DBs / or the image DB for that matter  ...

Persoanlly I get nervous when people talk about syncing /usr/openv/etc ...

Why you ask, after all, you have done this in the past etc ...  other people do this etc ...

As you've worked out, I work for Sym - I do this all day/ every day and I have lost count of situations where people have called up one week after a 'migration'  bacuase something is broken ...

So then I ask, when you ran the Catalog recovery ...

Oh no, we copied across ....

Really I say ... as tears begin to fill my eyes, and I consider a new career ...

You get the idea  - sure, coping across /usr/openv is probably ok ... IF you install the correct (equal) version of NBU first, to set up things like /etc/services etc ..

Persoanlly, and you did ask, if possible my preference would be to run a proper cat recov. , as documented in the manuals.  That way, if it goes wrong, you can turn round to Symantec and say ..." I did as described in the manuals, using a supported method".

I apologise in advance for my views, I see from your very reasonable comments that you are not looking to take short cuts, and are slightly backed into a corner.  But speaking, 'on average' from the systems I see that are often very onwell, it is no surprise when we find a short cut has been taken at some point.

Martin

 

 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
thesanman's picture

I hear and completely understand your comments.  The whole reason for the change is to make my catalog backup run faster; thus reducing my daily vault run time when it does it's full offsite tape backup.

I will see what I can do about using the 2nd copy tapes in my QA system.  What I don't understand is how to make the recovery process use them.  From the GUI it simply seems to looks for the primary copy.  The daily recovery e-mail I get mentions briefly how to promote the 2nd copy BUT this seems to be only if you do the recovery by hand which looks all a bit complicated to me.

NBU v7.5.0.6 Master and Media servers on RHEL 5/6 & Win2008; SAN based LTO3, 4 and 6 tape libraries
Linux, Solaris, Windows and OpenVMS clients.
PureDisk, SLP, VMware, HyperV, Oracle, Netezza, SQL/Server,

mph999's picture

I thought you'd ask that ...

bprecovery -wizard -copy 2

The command line version of the wizard. 

Martin

 

 

 

 

 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
thesanman's picture

Did it; same result!  Using Production "copy 2" tape initiated a Full catalog restore in my QA environment.  Same problem; only pulled in the incremental file.

Points to an issue with my Production DR recovery file I think.  I will see what my case support person has to say.

NBU v7.5.0.6 Master and Media servers on RHEL 5/6 & Win2008; SAN based LTO3, 4 and 6 tape libraries
Linux, Solaris, Windows and OpenVMS clients.
PureDisk, SLP, VMware, HyperV, Oracle, Netezza, SQL/Server,

thesanman's picture

Too easy!

Will give it a go in the morning.  That way I'm not compromising my primary recovery copy at the DR site.

NBU v7.5.0.6 Master and Media servers on RHEL 5/6 & Win2008; SAN based LTO3, 4 and 6 tape libraries
Linux, Solaris, Windows and OpenVMS clients.
PureDisk, SLP, VMware, HyperV, Oracle, Netezza, SQL/Server,

mph999's picture

EDIT...

Can't edit my above post for some reason ...

This bit 

"We can't merge the EMM DBs / or the image DB for that matter  ..."  should read, without engaging a symantec partnet for consulting services.  It can be done, just not by Tech support.

 

M

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
thesanman's picture

Yes; I realised that.  We went down that route once before using Symantec support when we migrated the Master Server from Solaris to RHEL, changing the name as we went.

NBU v7.5.0.6 Master and Media servers on RHEL 5/6 & Win2008; SAN based LTO3, 4 and 6 tape libraries
Linux, Solaris, Windows and OpenVMS clients.
PureDisk, SLP, VMware, HyperV, Oracle, Netezza, SQL/Server,

thesanman's picture

I have been working on this over the past month or so; finally worked out a sequence that works for me.

With a fresh install and the required catalog tapes (full and daily incremental) located and the last recovery file in a known location.

Run bprecover -wizard

Specifiy the latest "incremental" recovery file.

Then specify a "Partial" recovery which in my case only recovers the catalog files from the incremental only.

Then without stopping and restarting Netbackup; do exactly the same again BUT this time specifiy a "Full" recovery which works as it should, including the database.

This time around not only is the complete catalog file set recovered and then the incremental files but also the EMM database as well.

I'm still talking to Symantec about this but at least I now appear to have a working solution which will allow me to update to my nice new Master Server.

NBU v7.5.0.6 Master and Media servers on RHEL 5/6 & Win2008; SAN based LTO3, 4 and 6 tape libraries
Linux, Solaris, Windows and OpenVMS clients.
PureDisk, SLP, VMware, HyperV, Oracle, Netezza, SQL/Server,

thesanman's picture

I have successfully used this mechanism to migrate my master server.

Symantec support cannot explain what's going on and I keep hearing that they can't replicate the problem yet I can at will.

So my only advice to people is to somehow test your catalog restores.

NBU v7.5.0.6 Master and Media servers on RHEL 5/6 & Win2008; SAN based LTO3, 4 and 6 tape libraries
Linux, Solaris, Windows and OpenVMS clients.
PureDisk, SLP, VMware, HyperV, Oracle, Netezza, SQL/Server,