Video Screencast Help

Import process taking too much time

Created: 10 Oct 2011 | 22 comments
Zahid.Haseeb's picture

Environment

Veritas Netbackup 7.1

OS = win2003

Primary Site Netbackup Server name = abc

Secondary Site Netbackup Server name = xyz (high configuration than Primary Site Netbackup Server)

Tape Drive is same on both Side

 

Problem

I took backup on Primary Site Netbackup Server name abc and send this Tape Cartridge to Secondary Site Netbackup Server. On Secondary Site Netbackup Server I started doing import activity (as both Netbackup Server's name are different). Now when I started doing the import activity which(import activity) has two phases. The two phases are initiate import and import. While importing the Secondary Site Netbackup Server is taking enough time. My question is does this is the usual process(Although I have high hardware configuration of Secondary Site Netbackup Server). See the progress report below:

 

Netbackup Server Primary Site name abc

I have two flat file backups in this tape and both took around 1+4 hours in backed up

backup 1 size around = 9GB  (took around 1hour)

backup 2 size around = 16GB (took around 4hour)

 

Netbackup Server Primary Site name xyz

Now on the Seconadry Site Netbackup Server I started doing import activity. See the result below.

Phase I

initiate import of backup 1    

initiate import of backup 2   both initiate import took one day on Secondary Site Netbackup Server

Phase II

Import backup 1 started on 6 OCT 2011 10:52 AM and ended on 7 OCT 2011 9:37 PM

import backup 2 started on 9 OCT 2011 9:11 AM and till running

 

Note: This same happens with me on earlier Tape Cartridge too and I also experienced that my earlier Tape Cartridge got Frozen while the Import stage. I am also sure that my Secondary Site Tape Drive is fine.

 

Comments 22 CommentsJump to latest comment

Marianne's picture

It is perfectly normal for imports to take very long, especially if images on tape are multiplexed. The amount of files in the backup image up also plays a part -  lots of small files take longer to catalog.

Check that buffer sizes are the same at the 2 sites.

If you have network connectivity between the sites, you can use the 'Recovery without Import' method. In short, you simply copy images to 2nd master, write protect the tapes, put them in a pool that will never be used for backups at site 2 (e.g. RESTORE), add 'Media Host Override' in master server's Host Properties under General Server, and start the restore.

Herewith Alex Davies' 2 white papers - the 1st 3-page doc is still valid. The 2nd one has a lot of additional useful info:

http://eval.symantec.com/mktginfo/products/White_P...

http://eval.symantec.com/mktginfo/enterprise/white...

 

Reason for Frozen tapes can be found in media server's bptm log. You can also check Tape Logs report.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

J.H Is gone's picture

I also had the same setup and it use to take me 6 to 8 hours to do the import and restore.

Then like Marianne I copied the images dir from one site to the server on the other site (I put in different dir - and make a link when I need to do a restore - so my database consistency check does not have issues)

Now I take a tape to the other site and I am doing a restore in less then 10 minutes.

I don't have to know how to spell....I work on Unix.
NetBackup 7.0.1 - AIX & Windows

Zahid.Haseeb's picture

1.) Cant we move the multiplexed backup tapes to the recovery site/DR site ?

2.) I have a particular client ABC in veritas\netbackup\db\images\client\ABC directory. I only want to restore last one day backup of that particular client ABC from primary site tape to the dr site/recovery site(also the Tape only contain last one day backup).

So is that possible that I can only copy the particular date client data files from veritas\netbackup\db\images\client\ABC directory and merge it to dr site/recovery site. So that I dont need to copy whole ABC folder.

====================================================

Second thing is that the ABC folder have few folders whose size is in GB. if let suppose we have to copy the whole ABC folder Primary Site and have to merge to the DR Site/Recovery Site , do we have to copy the highlighted folder too which are in the ABC folder.

3.) @ J.Hin ""and make a link when I need to do a restore""

How can we make a link ?

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

'link' is only applicable to Unix master.

For Windows master, simply copy client folders from db\images on Master1 to same folder on Master2.

You can copy the entire ABC folder to copy all images for client ABC, or else create ABC folder on remote master under images, and copy the 131..... folder with the latest images to the remote master. Actual image headers and .f files can be found in these 10-digit sub-folders.

To convert the 10-digit number to normal time, run 'bpdbm -ctime <10-digit-number>'

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

 

copy the 131..... folder with the latest images to the remote master. Actual image headers and .f files can be found in these 10-digit sub-folders.

To convert the 10-digit number to normal time, run 'bpdbm -ctime <10-digit-number>'

 

Would you please elaborate the above more.

 

===========================================

 

What is the difference between the below two highlighted boxes please. (Actually I am not able to see the 10 and 11 OCT 2011 images in any 131... folder but can see the images of 10 and 11 OCT files under the \ABC folder.)

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

1317877590 = Thu Oct 06 2011 07:06:30

1317878313 = Thu Oct 06 2011 07:18:33

I am surprised that these files are directly under ABC folder and not in 1317000000 folder. I cannot explain why these files are there. Can you browse images for client ABC for 6 Oct between 07:00 and 08:00?

Backups for 10 and 11 Oct should be in 1318000000 folder.

You might want to run 'bpdbm -consistency' when there are no active backups.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

Can you browse images for client ABC for 6 Oct between 07:00 and 08:00?

 

 

You might want to run 'bpdbm -consistency' when there are no active backups.

I am showing result only for client name ABC

 

 

check directory of cli
______________________
checking image file <A
checking files file <A
checking image file <A
checking files file <A
checking image file <A
checking files file <A
checking image file <C
checking files file <C
checking image file <C
checking files file <C
checking image file <C
checking files file <C
checking image file <H
checking files file <H
checking image file <H
checking files file <H
checking image file <H
checking files file <H
checking image file <I
checking files file <I
checking image file <I
checking files file <I
checking image file <I
checking files file <I
checking image file <I
checking files file <I
checking image file <I
checking files file <I
checking image file <I
checking files file <I
checking image file <I
checking files file <I
checking image file <I
checking files file <I
checking image file <I
checking files file <I
checking image file <I
checking files file <I
checking image file <I
checking files file <I
checking image file <I
checking files file <I
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <L
checking files file <L
checking image file <N
checking files file <N
checking image file <N
checking files file <N
checking image file <P
checking files file <P
checking image file <P
checking files file <P
checking image file <P
checking files file <P
checking image file <R
checking files file <R
checking image file <R
checking files file <R
checking image file <R
checking files file <R
checking image file <S
checking files file <S
checking image file <S
checking files file <S
checking image file <S
checking files file <S
checking image file <U
checking files file <U
checking image file <U
checking files file <U
checking image file <U
checking files file <U
 
 
check directory of cli
======================================
Still not able to see the 1318..... folder 

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

Then you do not have any backups for Client ABC for 10 and 11 October.

All backups between these 2 timestamps will all be in 1318... folder:

 bpdbm -ctime 1318000000
1318000000 = Fri Oct  7 17:06:40 2011

 bpdbm -ctime 1319000000
1319000000 = Wed Oct 19 06:53:20 2011

Backups since Wed Oct 19 06:53:20 will be in folder 1319... folder.

 

Did bpdbm -consistency complete without any errors?  You may also want to run the command with higher level:

-consistency [0-2]
Runs the consistency checks on the catalog. The following are the three
consistency levels:
0 - A quick check of the NetBackup image database (the default).
1 - Performs more checks than the default check.
2 - The most in-depth consistency check. In addition to the level 0 and 1
checks, this level checks that the media that is mentioned in the image exists.
(That is, it cross-references the media servers databases.) On a large NetBackup installation, the process takes much longer to complete than the other checks.

 

 

Please check if these files are also showing up in the 1317... folder - the images are shown correctly in BAR:

XXX_1317877590_FULL
XXX_1317877590_FULL.f
XXX_1317878313_FULL
XXX_1317878313_FULL.f

 

I have checked various client image folders in our own environment - I do not see image files anywhere but in <timestamp> sub-folders.  Do you have any idea how this came about?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Marianne's picture

We are deviating from your initial request...

Catalog consistentency is a separate issue. To summarize:

  1. Lengthy imports are to be expected.
  2. "Recovery without Import" can be used to prevent lengthy imports.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Marianne's picture

Zahid

Feedback please?

This post is still appearing under 'Can you solve these?'

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

thanks for your kind follow-uo.Actually I will do this but its pending from my side for somw other reason/delay :(

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Zahid.Haseeb's picture

Hello all.

What I did I actually copy all files like the below from Primary NBU Server and pasted to the Secondary NBU Server. But these files were not in the folder under 131.......  or 132........... They were at the path of Veritas\Netbackup\db\Images\Machine_Name .. I copied all these specific date file from Primary Site and Pasted to Secondary Site with same path but not able to search the backup images from the Catalog. The NBU Catalog search of Secondary Site NBU says that no images were found.

XXX_1317877590_FULL
XXX_1317877590_FULL.f
XXX_1317878313_FULL
XXX_1317878313_FULL.f

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Zahid.Haseeb's picture

I did one another thing. I ran the consistensy check ( bpdbm -consistency ) at the Seconadry Site and finish without any error.

Now what I saw I actaully saw a new folder created name 1322000000 at the path under Veritas\Netbackup\db\Images\Machine_Name\1322000000. And this folder is not created earlier. Now I move all the images to that folder which files I pasted outside from this folder 1322000000 as this folder was not created.

Now I Ran the Catalog search and saw all the backup details.

Finally I think to start a restore from the Backup, Archive and Restore. I open Backup, Archive and Restore and select a backup to restore(I also able to saw the data when I further go under the backup image in the Backup, Archive and Restore as this is the Flat file data).

 

Now when I started the restore I faced the below error in the Activity Monitor.

12/29/2011 5:33:50 PM - begin Restore 12/29/2011 5:33:51 PM - 1 images required 12/29/2011 5:33:51 PM - media A00033 required 12/29/2011 5:33:54 PM - restored image techServer_1322653660 - (server name not found in the NetBackup configuration(254)) 12/29/2011 5:33:54 PM - end Restore; elapsed time: 00:00:04 invalid error number(2826)

Status View details

17:41:15 (242.xxx) INF - Status = invalid error number.

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

Zahid - as explained in October - client images MUST be in ...\images\client-name\10-digit-datestamp\XXX_131787....

Image files directly under \images\client-name\  WILL NOT WORK. You cannot restore from them unless they are in 10-digit-datestamp folder.

Don't forget about the FORCE_RESTORE_MEDIA_SERVER entry as described in Recovery without Import doc.  Windows equivalent:
Host Properties -> Master -> General Server. Specify original and new media server under Media Host Override.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

Image files directly under \images\client-name\  WILL NOT WORK. You cannot restore from them unless they are in 10-digit-datestamp folder.

We can restore if the files are not under 10-digit-datestamp folder.

1.) Create a 10 digit time stamp folder. Verify that the folder created with correct time-stamp. Copy the images from the client-name folder to the 10 digit time stamp folder which created.

2.) Same thing do with the catstore folder.

I did it and it worked :)

==========================================================

If the images are not under \images\client-name\ mean the below:

 

The default value of the file name IDIRSTRUCT is 2 which is under Program Files\Veritas\NetBackup\db folder. With this value in place, all Netbackup image information (header files, data files etc) will be written to the following paths:

On Windows:
<install_path>\VERITAS\NetBackup\db\images\<client_name>\<ctime>

SO,

If the IDIRSTRUCT file is deleted, NetBackup will recreate this file with a value of 1; which is a change from the default value of 2.  This will result in the following behavior:
 
All new NetBackup image information (header files, data files etc) will be written directly under the the following paths:
On Windows:

<install_path>\VERITAS\NetBackup\db\images\<client_name>\

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Zahid.Haseeb's picture

@ Marianne

Zahid - as explained in October - client images MUST be in ...\images\client-name\10-digit-datestamp\XXX_131787....

Image files directly under \images\client-name\  WILL NOT WORK. You cannot restore from them unless they are in 10-digit-datestamp folder.

 

Any way to repair(I tried to run the bpdbm -consistency[2] but did not corrected the creation of image files to correct location) so that the images will be created to the right location ?

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

You need to find out how image files landed up in the first place under the client name. NetBackup did not do that.

Just move the image files into their correct sub-folders:

XXX_1317877590_FULL
XXX_1317877590_FULL.f
XXX_1317878313_FULL
XXX_1317878313_FULL.f

need to be in 1317000000 folder.

If those files already exist in the folder, it means that 'someone' has copied them to the parent folder.  Just delete them from ...\db\images\client-name.

 

Hopefully we can close this thread now.... It's been going close to 3 months now....

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

Same thing I did As I tried to Move the files but facing the below error when I ran the consistency check.

<C:\Program Files\Veritas\NetBackup\db\images\tech-srv1\xxxx000000\ABCVSS_xx
xx627314_FULL.f> does not exist or is corrupted
 

Second I am also not able to restore when I try to select the backup fro Backup, Archive and Restore. See the error below for reference:

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

Sounds like you've got more problems than just the 10-digit folder.

Suggestion:

Download and run nbcc http://www.symantec.com/docs/TECH67840

Open a Support call, submit the nbcc output for a trained engineer to analyze the output and suggest fixes.

If you carried on with imports when you started in October, all would be over and done with by now.... Imports and restores....

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

My problem is resolved. I had two problems.

Problem # 1 

My 10 digit folders were not able to be created

Resolution # 1

The default value of the file name IDIRSTRUCT is 2 which is under Program Files\Veritas\NetBackup\db folder. With this value in place, all Netbackup image information (header files, data files etc) will be written to the following paths:

On Windows:
<install_path>\VERITAS\NetBackup\db\images\<client_name>\<ctime>

SO,

If the IDIRSTRUCT file is deleted, NetBackup will recreate this file with a value of 1; which is a change from the default value of 2.  This will result in the following behavior:
 
All new NetBackup image information (header files, data files etc) will be written directly under the the following paths:
On Windows:

<install_path>\VERITAS\NetBackup\db\images\<client_name>\

For more details follow the below TN

http://www.symantec.com/business/support/index?pag...

 

Problem # 2

I was facing the below Error

 

Resolution # 2

Actually I did not copy completefiles from the catstore. Then I copied all files of the specific 10digit folder then I am not facing the above error

For a proof I saw the file is missing(which I did not copied) under bpdbm logs

 ERROR:MemoryMappedFile:: Open(C:\Program Files\Veritas\NetBackup\db\images\Client_Name\xxx2000000\catstore\PolicyName_xxx2645479_FULL.f_imgHeader0 - current file size:0)
13:53:24.013 [4040.5040] <2> process_request: request complete: exit status 227 no entity was found; query type: 79

 

=================================================================

When I started restore I also faced a small problem which is just the Activity Monitor said you need to restart the Server Service. So I just restarted the Server Service from Administrative Tools. See the below Activity Monitor > Detail Status for your reference:

Detail Status

1/3/2012 3:20:30 PM - Info bptm(pid=3336) A00033            
1/3/2012 3:20:30 PM - Info bptm(pid=3336) INF - Waiting for positioning of media id A00033 on server tpsme for reading.
1/3/2012 3:20:30 PM - positioning A00033 to file 5
1/3/2012 3:20:34 PM - Info tar32(pid=5276) Restore started.           
1/3/2012 3:20:34 PM - connected; connect time: 00:00:07
1/3/2012 3:21:27 PM - positioned A00033; position time: 00:00:57
1/3/2012 3:21:27 PM - begin reading
1/3/2012 3:21:27 PM - Info bpbrm(pid=2232) sending media manager msg: CONTINUE RESTORE       
1/3/2012 3:21:28 PM - Warning bpbrm(pid=4168) from client tpsme: WRN - 'Server' service needs to be restarted for share to take effect: D:\Backups 30-11-2011\Z\abcFolder\ABC

 

In the end I really thankful to all people who gave me there valuable comments and thanks directly to Symantec for there valuable TN.

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

"I saw the file is missing(which I did not copied)"

Instructions in Recovery Without Import tells us to copy entire client folder:

Copy the desired client directories at the originating site. This would be the contents of /usr/openv/netbackup/db/images/<client name> on UNIX master servers or
\Program Files\VERITAS\netbackup\db\images\<client name> on Windows master servers.

If you really MUST copy less, you need EVERYTHING under the 10-digit folder ( 'C:\Program Files\Veritas\NetBackup\db\images\Client_Name\xxx2000000\').

I was wondering why Recovery Without Import seems to work for all of us except for you...

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

If you really MUST copy less, you need EVERYTHING under the 10-digit folder ( 'C:\Program Files\Veritas\NetBackup\db\images\Client_Name\xxx2000000\').

Actually Marianne In my case I did not have the 10-digit folder thats why I had to copy a specific series of 10 digit time stamp folder. For that I searched the Catalog and saw what is the time-stamp of my backup images which I want to restore. I then created a folder with specific time-stamp and than copy/pate the specific time-stamp image files.

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com