Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

ORA-19612 Error restoring Oracle 10G database on AIX - NetBackup status 0!!!

Created: 15 Nov 2010 | 3 comments

All,

I have a curious, sensitive (obviously as this is our production SAP env), and mystifying restore issue with Oracle RMAN. 

First the environment:

  • Master Server: Windows 2008 R2 VM, with NBU 7.0.1
  • Media Servers: Mixture, but this one is AIX 6.1 with NBU 7.0.1
  • Clients: All AIX 6.1 with NBU 7.0.1, running Oracle 10.2.0.4
  • Library: HP ESL 712e, LTO3 / LTO4 tapes call configured as HCART, all firmware reasonably up2date
  • Policy uses 12 MPX (chosen to maximise backup speed, no significant issue for restore speeds for us)

The Task:

RMAN backup to tape from production client > AIX media server, then restore across dark fibre link to test client in remote DC (approx 14km or so). This is an activity that we have completed many many times without issue. 

The Problem: 

Latest attempt to do the restore from backup produced, and so far has been continuing to produce the following errors:

 Database CT1, error is

channel tac1: reading from backup piece d8lsr8lt_1_1

ORA-19870: error reading backup piece d8lsr8lt_1_1

ORA-19612: datafile 16 not restored due to missing or corrupt data


Database RT1, error is

channel tac1: reading from backup piece lelsr8mk_1_1

ORA-19870: error reading backup piece lelsr8mk_1_1

ORA-19612: datafile 4 not restored due to missing or corrupt data


Database ST1, error is

channel tac1: reading from backup piece evlsr8ma_1_1

ORA-19870: error reading backup piece evlsr8ma_1_1

ORA-19612: datafile 19 not restored due to missing or corrupt data 

We have done this a number of times now using different tapes, and am still getting the issues. 

The Mystery:

  • The backups Validate against the original data (mostly - one didn't)
  • NetBackup reports Status 0 with both the backups and the restores

The Workaround Attempt:

We are now attempting a restore using a windows media server located at the remote DC. This will remove the AIX media server and possibly the TCP/IP connection (will now be SAN mounted tape from Prod DC) from the transport path. We have also enabled robust logging in an effort to capture any failure or issue. 

Has anyone seen anything like this? 

Comments 3 CommentsJump to latest comment

Yasuhisa Ishikawa's picture

Using NetBackup backup templates, backup piece name must be in form of "bk_*", "arch_*" or "ctrl_*".
Check these backup pieces have been stored through sbt channels or disk channels. I assume these backups have been taken without netbackup, cannot be restored from sbt channels.

Authorized Symantec Consultant(ASC) Data Protection in Tokyo, Japan

Will Restore's picture

per this site: http://dbadoc.blogspot.com/2008/03/error-code-rman-06026-error-reading.html

Solution:

The RMAN backup set is corrupt and you be unable to use the backup set.

To resolve the issue , take another RMAN backup and validate all backup sets.

To get a list of all backup sets

RMAN > list backupsets;

RMAN > validate backupset ;

Will Restore -- where there is a Will there is a way

Jonathan Bourke's picture

The last attempt to restore using a SAN mounted tape drive rather than using TCP/IP to our remote DC worked. 

So it appears that some sporadic interference corrupted our restore over the network link, as outlandish as that sounds, yet with no obvious errors.

We are looking at doing some inline copies of backups to assist with validation.

Jonathan

Website | LinkedIn