Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Restore is too slow

Created: 06 May 2013 | 19 comments
Zahid.Haseeb's picture

Environment

Primary Site:

NBU Master/Media Server 7.5.0.3 (32bit)

OS = Windows 2003 SP2 (32bit)

Tape Drive = Standalone Tape Drive

Secondary Site:

NBU Media Server 7.5.0.3 (32bit)

OS = Windows 2003 SP2 (32bit)

Tape Drive = Standalone Tape Drive

(Primary Site and Secondary Site are connected via VPN)

Query

I created a tape Cartridge from Primary Site and sent it to Secondary Site. Inserted the tape Cartridge in to Secondary Site Media Server. Started restore but the restore was taking long time. In the event viewer I saw below error messages which directly says that there must be some problem in tape Cartridge. I replaced the tape Cartridge and started restore again. Now I am not able to see the error messages like below , but the restore is taking too much time.

Errors

TapeAlert Code: 0x03, Type: Warning, Flag: HARD ERROR, from drive SEAGATE.DATDAT72-000.001 (index 0), Media Id A00013

===

TapeAlert Code: 0x05, Type: Critical, Flag: READ FAILURE, from drive SEAGATE.DATDAT72-000.001 (index 0), Media Id A00013

===

TapeAlert Code: 0x14, Type: Critical, Flag: CLEAN NOW, from drive SEAGATE.DATDAT72-000.001 (index 0), Media Id A00013

===

Faulting application tar32.exe, version 7.500.12.207, faulting module libtfi.dll, version 7.500.12.207, fault address 0x0000698a.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

====

The application, C:\Program Files\Veritas\NetBackup\bin\tar32.exe, generated an application error The error occurred on 05/02/2013 @ 15:43:47.233 The exception generated was c0000005 at address 2B80698A (libtfi!Symantec::NetBackup::Ncf::Tfi::SharedMemoryInfoRestore::getRestoreDetail)

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

===

Operating Systems:

Comments 19 CommentsJump to latest comment

Mouse's picture

Did you try to run cleaning on tape drive? 

Magnetic heads reported dirty by the TA flag: CLEAN NOW

Zahid.Haseeb's picture

No, Not yet. Will do it

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

mph999's picture

The tape alerts are sent from the hardware, so this issue is outside NBU, NBU is just a casualty of the drive issues, it is not causing them.

If cleaning does not resolve, then you either have a faulty or worn out tape drive.  It is impossible for NetBackup to cause Tapealerts, it just cannot happen.

Martin

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
Marianne's picture

You have device errors as well as tar32 errors.

Tell us more about the desination client.
Is remote media server restoring to itself or to a network client?

Have you enabled logs?
On media server: bptm and bpbrm
On client: tar.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

Thanks Martin , I will clean the Tape in a while

@ Marianne

Tell us more about the desination client.

Restoring on the same machine (Media Server of Secondary Site), not Network client

Logs attached..Password emailed for the attached file

AttachmentSize
logs.rar 37.84 KB

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

You seem to be restoring from a multiplexed backup. 

What was MPX level for the backup?
What was fragment size on STU setting that did the backup?

High MPX value and default fragment size of 1 TB, are normally the reason for slow restores.
The higher the MPX value is, the longer it will take to find one client's file among interleaved files from other clients.
Large fragment size will result in a single, large 'tar' file on tape and the entire session/fragment needs to be scanned from beginning to end to find the files that need to be restored.

Yesterday a single file restore was started at 11:28 and completed at 11:31:

11:31:11.004 [2116.532] <4> mpx_read_backup: successfully restored 1 of 1 requests, read total of 864 Kbytes at 37.667 Kbytes/sec

Most of the time was taken up by position and locating of data on tape.

There was also no performance tuning done: 
using 30 data buffers, buffer size is 32768

There were no errors in bptm logs for yesterday and today.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

SO we need to increase data buffer ?

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

Please read the rest of my post?

It all depends on how the backup was done at the source site.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

See the below for details.

1_0.png

2_1.png

3.png

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

You have showed us properties for a DSU.

The backup was done to tape, not disk.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

Ahh sorry for this. See the currect below please.

4.png

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

OK, no multiplexing, but default fragment size of 1 TB. 

This is the biggest cause of slow restores - the entire 'tar-file' must be searched for the files.

We normally make it 5000 MB.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

Let me try this and will share the result

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Zahid.Haseeb's picture

After changing fragment size to 5000 MB, still restore is too slow. I have a restore of almost 22GB and Tape is still getting read after three and half hours. See the below snap for reference:

5.png

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

Did you perform a new backup with changed fragment size?

Are you now restoring from the new backup?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

No. Its old backup images on Tape.

Actually I need to restore the old tape for my use

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

jim dalton's picture

As MVDB has hinted at, you cant re-arrange the tape after having written it hence why she asked about the new backup. From way back when DAT drives werent the most performant but that could have changed in the intervening years.

Jim 

Zahid.Haseeb's picture

First thanks for your words Jim..The question is not to backup again and then restore the new backups

. The question is, I need to restore the old backups. Almost 20hours passed but the restore was not successful. I did restore many times but never faced that much time. This seems something else.

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

Your backup was done as one large 'tar' file, so nothing we can do to quickly locate files.

Have you compared buffer settings between original media server and restore media server?

We see in bptm log that no buffer sizes were changed, but we also see in the same log that there were no 'waits' for empty buffers:
mpx_read_data: waited for empty buffer 0 times, delayed 0 times

This simply says to me that read speed from the DAT tape drive is very slow.

You need to check the media server on the remote site - bus speed, memory, cpu, SCSI controller, specs of the drive itself, etc....

Best test will possibly to do a test backup and restore on your remote media server and see if there is a difference.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links