BUG REPORT: Oracle Recovery Manager (RMAN) restore fails with ORA-27192 when using Veritas NetBackup (tm) Enterprise Server.

Article:TECH45761  |  Created: 2005-01-28  |  Updated: 2006-01-18  |  Article URL http://www.symantec.com/docs/TECH45761
Article Type
Technical Solution

Product(s)

Environment

Issue



BUG REPORT: Oracle Recovery Manager (RMAN) restore fails with ORA-27192 when using Veritas NetBackup (tm) Enterprise Server.

Error



ORA-27192: skgfcls: sbtclose2 returned error - failed to close file;
ORA-19511: Error received from media manager layer, error text:
Failed to process backup file

Solution



Bug:  Unable to complete Oracle (Tablespace) restore (3 datafiles) - Unable to read comm file

Bug ID(s):  ETrack 613795, ETrack 519760, ETrack 519758,  ETrack 519755, ETrack 565953, ETrack 565956, ETrack 565959 and ETrack 613795

Symptom(s):
An Oracle RMAN restore or recovery fails with ORA-27192.  The data is successfully restored to the client and the NetBackup Activity Monitor shows a successful restore, Status Code 0.  But there is an interruption in the restore after Oracle has received all of the data it requires even though there may be more data in the backup piece.  The media manager does not close properly, causing the client process to time out after 15 minutes waiting for the server status to be written to the client's communications log.  This causes the restore to fail on the client with an RMAN error ORA-27192.

Log Files:
On the Oracle Client
Oracle RMAN output
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: Error received from media manager layer, error text:
 Failed to process backup file <bk_uonh67b5e_s791_p1_t576957614>

dbclient log:
14:50:59.718 [1932.1552] <2> int_ReadData: INF - exiting sbtread2 : Read <4539>
buffers, <297467904> bytes
14:50:59.718 [1932.1552] <2> sbtclose2: INF - entering
14:50:59.718 [1932.1552] <2> int_CloseImage: INF - entering
14:50:59.718 [1932.1552] <2> int_CloseImage: INF - Restore - closing <bk_uonh67b5e_s791_p1_t576957614>
14:50:59.718 [1932.1552] <4> serverExitStatus: entering serverExitStatus.
14:50:59.718 [1932.1552] <4> readCommMessages: Entering readCommMessages
15:06:04.724 [1932.1552] <16> readCommFile: ERR - timed out after 900 seconds while reading from C:\Program Files\VERITAS\NetBackup\Logs\user_ops\dbext\logs\1552.0.1135025111

On the media server
bpbrm log:
14:50:59.747 [19990] <2> bpbrm tell_mm: sending media manager msg: STOP RESTORE algol_1134518480 EXIT=0
15:06:05.288 [19990] <2> bpbrm read_media_msg: read from media manager: MEDIA NOT READY

bptm log:
14:50:59.752 [19995] <2> process_brm_msg: GCACTIVE but CHILD_PID not alive
14:50:59.752 [19995] <2> process_brm_msg: could not find backupid algol_1134518480 to terminate
14:50:59.752 [19995] <2> process_brm_msg: no pending message from bpbrm
15:06:04.748 [19998] <2> write_bytes: ignoring [32] for DB Restore
15:06:04.748 [19998] <2> terminate_restore: [19995] waited for full buffer 142 times, delayed 246 times

Workaround:
This problem was introduced with the following NetBackup versions:
- 5.0 Maintenance Pack 6 (MP6)
- 5.1 Maintenance Pack 4 (MP4)
- 6.0 General Availability (GA).  

For both UNIX and Windows, it is possible to simply rerun the script or RMAN commands to complete the restore or recovery.  It is, however, necessary to repeat this process after each occurrence of ORA-27192.  RMAN senses the previous data was successfully recovered from the restore and proceeds to the next step in the restore and/or recovery.

Alternatively, for NetBackup versions 5.0 Maintenance Pack 6 (MP6) and 5.1 Maintenance Pack 4 (MP4) it is possible to temporarily replace a component of NetBackup with the same component from the previous patch release.  If you choose to implement this work-around, be sure to restore the original files after the restore is complete.

For UNIX:
- Stop all instances of Oracle linked to NetBackup on the host prior to replacing libobk.
- Replace the current libobk binaries with the corresponding libobk binaries from the previous patch release (see below).
- libobk is located in /usr/openv/netbackup/bin and has an extension of .so(64), .sl(64) or .a(64) depending upon the operating system.
- For 5.0 MP6, replace libobk with the libobk from 5.0 MP5, or revert the NetBackup client and all extensions to 5.0 MP5 or earlier.
- For 5.1 MP4, replace libobk with the libobk from 5.1 MP3, or revert the NetBackup client and all extensions to 5.1 MP3 or earlier.

For Windows:
- Stop the Oracle service for all instances of Oracle on the host prior to replacing xbsa.dll.
- replace xbsa.dll located in <install_path>\veritas\netbackup\bin with the xbsa.dll from the previous patch (see below).
- For 5.0 MP6, replace xbsa.dll with the xbsa.dll from 5.0 MP5, or uninstall NetBackup 5.0 MP6 with the Windows Add/Remove programs.
- For 5.1 MP4, replace xbsa.dll with the xbsa.dll from 5.1 MP3, or uninstall NetBackup 5.1 MP4 with the Windows Add/Remove programs.

For NetBackup version 6.0GA and 6.0MP1, it is not possible to replace xbsa.dll or libobk with a previous, working version.  If repeating the restore sequence is not acceptable, then the only other workaround is to upgrade the NetBackup client and Oracle extension to 6.0 MP3 (or later).

NOTE: Some customers have reported that the work-around does not resolve the problem.  In this case, the only remedy is to repeat the restore or recover command until the restore or recover is complete, or to install the specified maintenance packs listed below.

ETA of Fix:
A complete resolution to this issue was included in the following NetBackup patches:

    - NetBackup 5.0 Maintenance Pack 7 (MP7), released in December, 2006
- NetBackup 5.1 Maintenance Pack 6 (MP6), released in December, 2006.
- NetBackup 6.0 Maintenance Pack 3 (MP3), released in June, 2006.


Contact Symantec Enterprise Technical Support for NetBackup if the work-around or formal resolution, listed above, is not possible for you.



Supplemental Materials

SourceETrack
Value519760
DescriptionUnable to complete Oracle (Tablespace) restore (3 datafiles) - Unable to read comm file

SourceETrack
Value519758
DescriptionUnable to complete Oracle (Tablespace) restore (3 datafiles) - Unable to read comm file

SourceETrack
Value519755
DescriptionUnable to complete Oracle (Tablespace) restore (3 datafiles) - Unable to read comm file

SourceError Code
Value19511
DescriptionOracle RMAN error: ORA-19511

SourceError Code
Value27192
DescriptionOracle RMAN error: ORA-27192

SourceETrack
Value565953
DescriptionOracle restore fails or hangs

SourceETrack
Value565956
DescriptionOracle restore fails or hangs

SourceETrack
Value565959
DescriptionOracle restore fails or hangs

SourceETrack
Value613795
DescriptionOracle restore fails or hangs


Legacy ID



281191


Article URL http://www.symantec.com/docs/TECH45761


Terms of use for this information are found in Legal Notices