ORA-19612 Error restoring Oracle 10G database on AIX - NetBackup status 0!!!
I have a curious, sensitive (obviously as this is our production SAP env), and mystifying restore issue with Oracle RMAN.
First the environment:
- Master Server: Windows 2008 R2 VM, with NBU 7.0.1
- Media Servers: Mixture, but this one is AIX 6.1 with NBU 7.0.1
- Clients: All AIX 6.1 with NBU 7.0.1, running Oracle 10.2.0.4
- Library: HP ESL 712e, LTO3 / LTO4 tapes call configured as HCART, all firmware reasonably up2date
- Policy uses 12 MPX (chosen to maximise backup speed, no significant issue for restore speeds for us)
RMAN backup to tape from production client > AIX media server, then restore across dark fibre link to test client in remote DC (approx 14km or so). This is an activity that we have completed many many times without issue.
Latest attempt to do the restore from backup produced, and so far has been continuing to produce the following errors:
Database CT1, error is channel tac1: reading from backup piece d8lsr8lt_1_1 ORA-19870: error reading backup piece d8lsr8lt_1_1 ORA-19612: datafile 16 not restored due to missing or corrupt data Database RT1, error is channel tac1: reading from backup piece lelsr8mk_1_1 ORA-19870: error reading backup piece lelsr8mk_1_1 ORA-19612: datafile 4 not restored due to missing or corrupt data Database ST1, error is channel tac1: reading from backup piece evlsr8ma_1_1 ORA-19870: error reading backup piece evlsr8ma_1_1 ORA-19612: datafile 19 not restored due to missing or corrupt data
We have done this a number of times now using different tapes, and am still getting the issues.
- The backups Validate against the original data (mostly - one didn't)
- NetBackup reports Status 0 with both the backups and the restores
The Workaround Attempt:
We are now attempting a restore using a windows media server located at the remote DC. This will remove the AIX media server and possibly the TCP/IP connection (will now be SAN mounted tape from Prod DC) from the transport path. We have also enabled robust logging in an effort to capture any failure or issue.
Has anyone seen anything like this?