Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Oracle Backups failing with status 6

Created: 07 Jan 2014 | 23 comments
Andre_Torres's picture

Hi everyone,

I've migrated from an old Red Hat Master Server to a new NetBackup Appliance 5230 Master Server. I've done the following procedure to do the backups on new server:

1)Stop NBU services on client
2)Change SERVER entry on bp.conf on client pointing to new Master Server
3)Change hosts file adding IP of new Master Server
4)Start NetBackup services on client
5)Run bpclntcmd -clear_host_cache
 
Tested doing a file system backup and it worked. But when I try to do an Oracle Backup it gives me error 6
 
error6.png
 
As if it is a Script error, I've contacted the DBA Oracle and he said it gives him this error:
 
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch00 channel at 01/07/2014 02:03:24
ORA-19506: failed to create sequential file, name="bkp_dbf_587_1_836185699", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
VxBSACreateObject: Failed with error:
Server Status: Communication with the server has not been initiated or the server status has not been retrieved from the serve
 
 
 
Just for you all to know it worked on my old server, the only changes were the ones done which I said before.
Operating Systems:
Discussion Filed Under:

Comments 23 CommentsJump to latest comment

sri vani's picture

Please verify this link: http://www.symantec.com/business/support/index?pag...

If nt resolved your issue please attach dbclient log in .txt format

Solution

This issue's cause was incorrect permissions on the bp.conf file.

Modification of the permission to 644 resolved the problem.

Andre_Torres's picture

sri vani, it was already on permission 644.

dbclient log:

01:28:44.143 [9100] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
01:28:44.143 [9100] <2> int_LogSystemInfo: INF - 
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: titus.petrobras.com.br
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
User name:   oracle
Client Host: TITUS
 
01:28:44.143 [9100] <2> int_GetMMInfo: INF - Initialized Signal
01:28:44.143 [9100] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
01:28:50.154 [9100] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<petrobras_titus_db_ora_jbossrep_hot_full_dia>
01:28:50.155 [9100] <2> int_ProcessCommand: INF - Policy <petrobras_titus_db_ora_jbossrep_hot_full_dia> will be used for this API session.
01:28:51.156 [9100] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SCHED> cmd_val=<Default-Application-jbossrep-db-full-dia>
01:28:51.156 [9100] <2> int_ProcessCommand: INF - Schedule <Default-Application-jbossrep-db-full-dia> will be used for non-proxy API session.
01:28:54.160 [9100] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+veritas_pbx,1,20,2,1,0+tbcpetromarj01.txt
01:28:54.162 [9100] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0ab/b575b0ab+0,1,402,0,1,0+10.27.253.249.txt
01:28:54.162 [9100] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+vnetd,1,20,2,1,0+tbcpetromarj01.txt
01:28:54.163 [9100] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+bprd,1,20,2,1,0+tbcpetromarj01.txt
01:28:54.167 [9100] <2> vnet_pbxConnect: pbxConnectEx Succeeded
01:28:54.168 [9100] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.15929 TO 10.27.253.249.1556 fd = 17
01:28:54.193 [9100] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
01:28:54.197 [9100] <2> vnet_pbxConnect: pbxConnectEx Succeeded
01:28:54.198 [9100] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.62601 TO 10.27.253.249.1556 fd = 18
01:29:28.626 [9247] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
01:29:28.626 [9247] <2> int_LogSystemInfo: INF - 
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: titus.petrobras.com.br
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
User name:   oracle
Client Host: TITUS
 
01:29:28.626 [9247] <2> int_GetMMInfo: INF - Initialized Signal
01:29:28.626 [9247] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
01:29:34.635 [9247] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<petrobras_titus_db_ora_jbossrep_arc>
01:29:34.635 [9247] <2> int_ProcessCommand: INF - Policy <petrobras_titus_db_ora_jbossrep_arc> will be used for this API session.
01:29:35.635 [9247] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SCHED> cmd_val=<Default-Application-jbossrep-arc>
01:29:35.636 [9247] <2> int_ProcessCommand: INF - Schedule <Default-Application-jbossrep-arc> will be used for non-proxy API session.
01:29:39.641 [9247] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+veritas_pbx,1,20,2,1,0+tbcpetromarj01.txt
01:29:39.643 [9247] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0ab/b575b0ab+0,1,402,0,1,0+10.27.253.249.txt
01:29:39.643 [9247] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+vnetd,1,20,2,1,0+tbcpetromarj01.txt
01:29:39.644 [9247] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+bprd,1,20,2,1,0+tbcpetromarj01.txt
01:29:39.646 [9247] <2> vnet_pbxConnect: pbxConnectEx Succeeded
01:29:39.647 [9247] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.29615 TO 10.27.253.249.1556 fd = 17
01:29:39.672 [9247] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
01:29:39.677 [9247] <2> vnet_pbxConnect: pbxConnectEx Succeeded
01:29:39.678 [9247] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.33800 TO 10.27.253.249.1556 fd = 18
01:43:56.740 [9100] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/9100.0.1389065334
01:43:56.740 [9100] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/9100.0.1389065334>
01:43:56.740 [9100] <16> CreateNewImage: ERR - serverResponse() failed
01:43:56.740 [9100] <16> VxBSACreateObject: ERR - Could not create new image with file /bkp_dbf_585_1_836184533.
01:43:56.740 [9100] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
01:43:59.744 [9100] <4> sbtend: INF - --- END of SESSION ---
01:43:59.745 [9100] <8> close_image: Session being terminated abnormally, cleaning up
01:43:59.745 [9100] <4> close_image: INF - backup FAILED
01:43:59.745 [9100] <4> close_image: INF ---- end of Backup ---
 
01:43:59.745 [9100] <16> VxBSAEndTxn: ERR - Transaction ended with active Backup/Restore.
01:43:59.745 [9100] <16> xbsa_EndTransaction: ERR - VxBSAEndTxn: Failed with error:
   The transaction was aborted.
01:44:41.901 [9247] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/9247.0.1389065379
01:44:41.901 [9247] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/9247.0.1389065379>
01:44:41.901 [9247] <16> CreateNewImage: ERR - serverResponse() failed
01:44:41.901 [9247] <16> VxBSACreateObject: ERR - Could not create new image with file /bkp_arch_586_1_836184579.
01:44:41.901 [9247] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
01:44:45.913 [9247] <4> sbtend: INF - --- END of SESSION ---
01:44:45.914 [9247] <8> close_image: Session being terminated abnormally, cleaning up
01:44:45.914 [9247] <4> close_image: INF - backup FAILED
01:44:45.914 [9247] <4> close_image: INF ---- end of Backup ---
 
01:44:45.914 [9247] <16> VxBSAEndTxn: ERR - Transaction ended with active Backup/Restore.
01:44:45.914 [9247] <16> xbsa_EndTransaction: ERR - VxBSAEndTxn: Failed with error:
   The transaction was aborted.
01:48:09.890 [11482] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
01:48:09.891 [11482] <2> int_LogSystemInfo: INF - 
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: titus.petrobras.com.br
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
User name:   oracle
Client Host: TITUS
 
01:48:09.891 [11482] <2> int_GetMMInfo: INF - Initialized Signal
01:48:09.891 [11482] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
01:48:15.901 [11482] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<petrobras_titus_db_ora_jbossrep_hot_full_dia>
01:48:15.901 [11482] <2> int_ProcessCommand: INF - Policy <petrobras_titus_db_ora_jbossrep_hot_full_dia> will be used for this API session.
01:48:16.903 [11482] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SCHED> cmd_val=<Default-Application-jbossrep-db-full-dia>
01:48:16.903 [11482] <2> int_ProcessCommand: INF - Schedule <Default-Application-jbossrep-db-full-dia> will be used for non-proxy API session.
01:48:19.908 [11482] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+veritas_pbx,1,20,2,1,0+tbcpetromarj01.txt
01:48:19.911 [11482] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0ab/b575b0ab+0,1,402,0,1,0+10.27.253.249.txt
01:48:19.911 [11482] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+vnetd,1,20,2,1,0+tbcpetromarj01.txt
01:48:19.912 [11482] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+bprd,1,20,2,1,0+tbcpetromarj01.txt
01:48:19.922 [11482] <2> vnet_pbxConnect: pbxConnectEx Succeeded
01:48:19.923 [11482] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.39060 TO 10.27.253.249.1556 fd = 17
01:48:19.947 [11482] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
01:48:19.951 [11482] <2> vnet_pbxConnect: pbxConnectEx Succeeded
01:48:19.951 [11482] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.27983 TO 10.27.253.249.1556 fd = 18
02:03:22.281 [11482] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/11482.0.1389066499
02:03:22.281 [11482] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/11482.0.1389066499>
02:03:22.281 [11482] <16> CreateNewImage: ERR - serverResponse() failed
02:03:22.281 [11482] <16> VxBSACreateObject: ERR - Could not create new image with file /bkp_dbf_587_1_836185699.
02:03:22.282 [11482] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
02:03:25.287 [11482] <4> sbtend: INF - --- END of SESSION ---
02:03:25.287 [11482] <8> close_image: Session being terminated abnormally, cleaning up
02:03:25.287 [11482] <4> close_image: INF - backup FAILED
02:03:25.287 [11482] <4> close_image: INF ---- end of Backup ---
 
02:03:25.287 [11482] <16> VxBSAEndTxn: ERR - Transaction ended with active Backup/Restore.
02:03:25.287 [11482] <16> xbsa_EndTransaction: ERR - VxBSAEndTxn: Failed with error:
   The transaction was aborted.
11:06:10.992 [31600] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
11:06:10.992 [31600] <2> int_LogSystemInfo: INF - 
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: titus.petrobras.com.br
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
User name:   oracle
Client Host: TITUS
 
11:06:10.992 [31600] <2> int_GetMMInfo: INF - Initialized Signal
11:06:10.992 [31600] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
11:06:17.001 [31600] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<petrobras_titus_db_ora_jbossrep_hot_full_dia>
11:06:17.001 [31600] <2> int_ProcessCommand: INF - Policy <petrobras_titus_db_ora_jbossrep_hot_full_dia> will be used for this API session.
11:06:18.003 [31600] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SCHED> cmd_val=<Default-Application-jbossrep-db-full-dia>
11:06:18.003 [31600] <2> int_ProcessCommand: INF - Schedule <Default-Application-jbossrep-db-full-dia> will be used for non-proxy API session.
11:06:21.010 [31600] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+veritas_pbx,1,20,2,1,0+tbcpetromarj01.txt
11:06:21.012 [31600] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0ab/b575b0ab+0,1,402,0,1,0+10.27.253.249.txt
11:06:21.012 [31600] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+vnetd,1,20,2,1,0+tbcpetromarj01.txt
11:06:21.013 [31600] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+bprd,1,20,2,1,0+tbcpetromarj01.txt
11:06:21.017 [31600] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:06:21.017 [31600] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.59670 TO 10.27.253.249.1556 fd = 17
11:06:21.042 [31600] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
11:06:21.046 [31600] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:06:21.047 [31600] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.18961 TO 10.27.253.249.1556 fd = 18
11:21:22.870 [31600] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/31600.0.1389099981
11:21:22.870 [31600] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/31600.0.1389099981>
11:21:22.870 [31600] <16> CreateNewImage: ERR - serverResponse() failed
11:21:22.870 [31600] <16> VxBSACreateObject: ERR - Could not create new image with file /bkp_dbf_588_1_836219180.
11:21:22.870 [31600] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
11:21:25.875 [31600] <4> sbtend: INF - --- END of SESSION ---
11:21:25.875 [31600] <8> close_image: Session being terminated abnormally, cleaning up
11:21:25.875 [31600] <4> close_image: INF - backup FAILED
11:21:25.875 [31600] <4> close_image: INF ---- end of Backup ---
 
11:21:25.875 [31600] <16> VxBSAEndTxn: ERR - Transaction ended with active Backup/Restore.
11:21:25.875 [31600] <16> xbsa_EndTransaction: ERR - VxBSAEndTxn: Failed with error:
   The transaction was aborted.
11:43:07.678 [2429] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
11:43:07.679 [2429] <2> int_LogSystemInfo: INF - 
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: titus.petrobras.com.br
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
User name:   oracle
Client Host: TITUS
 
11:43:07.679 [2429] <2> int_GetMMInfo: INF - Initialized Signal
11:43:07.679 [2429] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
11:43:13.690 [2429] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<petrobras_titus_db_ora_jbossrep_hot_full_dia>
11:43:13.690 [2429] <2> int_ProcessCommand: INF - Policy <petrobras_titus_db_ora_jbossrep_hot_full_dia> will be used for this API session.
11:43:14.692 [2429] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SCHED> cmd_val=<Default-Application-jbossrep-db-full-dia>
11:43:14.692 [2429] <2> int_ProcessCommand: INF - Schedule <Default-Application-jbossrep-db-full-dia> will be used for non-proxy API session.
11:43:17.699 [2429] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/06b/c413626b+0,1,402,0,1,0+localhost.txt
11:43:17.700 [2429] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+veritas_pbx,1,20,2,1,0+tbcpetromarj01.txt
11:43:17.701 [2429] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0ab/b575b0ab+0,1,402,0,1,0+10.27.253.249.txt
11:43:17.702 [2429] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+vnetd,1,20,2,1,0+tbcpetromarj01.txt
11:43:17.703 [2429] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+bprd,1,20,2,1,0+tbcpetromarj01.txt
11:43:17.707 [2429] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:43:17.708 [2429] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.48403 TO 10.27.253.249.1556 fd = 17
11:43:17.728 [2429] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
11:43:17.732 [2429] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:43:17.733 [2429] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.33124 TO 10.27.253.249.1556 fd = 18
11:44:18.420 [2582] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
11:44:18.421 [2582] <2> int_LogSystemInfo: INF - 
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: titus.petrobras.com.br
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
User name:   oracle
Client Host: TITUS
 
11:44:18.421 [2582] <2> int_GetMMInfo: INF - Initialized Signal
11:44:18.421 [2582] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
11:44:24.430 [2582] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<petrobras_titus_db_ora_jbossrep_hot_full_dia>
11:44:24.430 [2582] <2> int_ProcessCommand: INF - Policy <petrobras_titus_db_ora_jbossrep_hot_full_dia> will be used for this API session.
11:44:25.432 [2582] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SCHED> cmd_val=<Default-Application-jbossrep-db-full-dia>
11:44:25.432 [2582] <2> int_ProcessCommand: INF - Schedule <Default-Application-jbossrep-db-full-dia> will be used for non-proxy API session.
11:44:28.436 [2582] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/06b/c413626b+0,1,402,0,1,0+localhost.txt
11:44:28.437 [2582] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+veritas_pbx,1,20,2,1,0+tbcpetromarj01.txt
11:44:28.439 [2582] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0ab/b575b0ab+0,1,402,0,1,0+10.27.253.249.txt
11:44:28.439 [2582] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+vnetd,1,20,2,1,0+tbcpetromarj01.txt
11:44:28.440 [2582] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+bprd,1,20,2,1,0+tbcpetromarj01.txt
11:44:28.442 [2582] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:44:28.443 [2582] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.57942 TO 10.27.253.249.1556 fd = 17
11:44:28.466 [2582] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
11:44:28.470 [2582] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:44:28.471 [2582] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.32691 TO 10.27.253.249.1556 fd = 18
11:58:19.922 [2429] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/2429.0.1389102197
11:58:19.922 [2429] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/2429.0.1389102197>
11:58:19.922 [2429] <16> CreateNewImage: ERR - serverResponse() failed
11:58:19.922 [2429] <16> VxBSACreateObject: ERR - Could not create new image with file /bkp_dbf_589_1_836221396.
11:58:19.922 [2429] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
11:58:22.929 [2429] <4> sbtend: INF - --- END of SESSION ---
11:58:22.929 [2429] <8> close_image: Session being terminated abnormally, cleaning up
11:58:22.929 [2429] <4> close_image: INF - backup FAILED
11:58:22.929 [2429] <4> close_image: INF ---- end of Backup ---
 
11:58:22.929 [2429] <16> VxBSAEndTxn: ERR - Transaction ended with active Backup/Restore.
11:58:22.929 [2429] <16> xbsa_EndTransaction: ERR - VxBSAEndTxn: Failed with error:
   The transaction was aborted.
11:59:30.536 [2582] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/2582.0.1389102268
11:59:30.536 [2582] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/2582.0.1389102268>
11:59:30.536 [2582] <16> CreateNewImage: ERR - serverResponse() failed
11:59:30.536 [2582] <16> VxBSACreateObject: ERR - Could not create new image with file /bkp_dbf_590_1_836221467.
11:59:30.536 [2582] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
11:59:33.542 [2582] <4> sbtend: INF - --- END of SESSION ---
11:59:33.542 [2582] <8> close_image: Session being terminated abnormally, cleaning up
11:59:33.542 [2582] <4> close_image: INF - backup FAILED
11:59:33.542 [2582] <4> close_image: INF ---- end of Backup ---
 
11:59:33.542 [2582] <16> VxBSAEndTxn: ERR - Transaction ended with active Backup/Restore.
11:59:33.543 [2582] <16> xbsa_EndTransaction: ERR - VxBSAEndTxn: Failed with error:
   The transaction was aborted.
15:21:05.361 [26318] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
15:21:05.361 [26318] <2> int_LogSystemInfo: INF - 
Veritas NetBackup for Oracle - Release 7.5 (2012020801)
System name: Linux
Node name: titus.petrobras.com.br
Release: 2.6.18-128.el5
Version: #1 SMP Wed Dec 17 11:41:38 EST 2008
Machine: x86_64
User name:   oracle
Client Host: TITUS
 
15:21:05.361 [26318] <2> int_GetMMInfo: INF - Initialized Signal
15:21:05.361 [26318] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
15:21:11.372 [26318] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<petrobras_titus_db_ora_jbossrep_hot_full_dia>
15:21:11.373 [26318] <2> int_ProcessCommand: INF - Policy <petrobras_titus_db_ora_jbossrep_hot_full_dia> will be used for this API session.
15:21:12.373 [26318] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SCHED> cmd_val=<Default-Application-jbossrep-db-full-dia>
15:21:12.373 [26318] <2> int_ProcessCommand: INF - Schedule <Default-Application-jbossrep-db-full-dia> will be used for non-proxy API session.
15:21:15.378 [26318] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+veritas_pbx,1,20,2,1,0+tbcpetromarj01.txt
15:21:15.380 [26318] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0ab/b575b0ab+0,1,402,0,1,0+10.27.253.249.txt
15:21:15.380 [26318] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+vnetd,1,20,2,1,0+tbcpetromarj01.txt
15:21:15.381 [26318] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/13a/50fc933a+bprd,1,20,2,1,0+tbcpetromarj01.txt
15:21:15.385 [26318] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:21:15.387 [26318] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.44735 TO 10.27.253.249.1556 fd = 17
15:21:15.408 [26318] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
15:21:15.412 [26318] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:21:15.413 [26318] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.65387 TO 10.27.253.249.1556 fd = 18
 
Will Restore's picture

client RMAN script  NB_ORA_SERV=   entry needs to point to the new Master

Will Restore -- where there is a Will there is a way

Andre_Torres's picture

It points a variable filled by bphdb, looking for it, it is pointing to my new server.

sri vani's picture

1)Add the appliance master server name as a first server to the bp.conf file on the client.

2)Check that /etc/hosts entry for the master is correct.

To verify if there any name resolution issues:

what is the output of 

bpcltncmd -pn

bpcltncmd -self  command from client

and also check the logs of dbclient(on client) and bprd on master server

Andre_Torres's picture

Add the appliance master server name as a first server to the bp.conf file on the client.

-Done

Check that /etc/hosts entry for the master is correct.

-Done

what is the output of bpcltncmd -pn

expecting response from server ptrbkpapl01
ptrbkpapl01 ptrbkpapl01 10.27.240.63 46976
 
 

bpcltncmd -self  command from client

yp_get_default_domain failed: (12) Local domain name not set
NIS does not seem to be running: (1) Request arguments bad
gethostname() returned: titus
host titus: titus.petrobras.com.br at 10.27.246.198
aliases:     titus.petrobras.com.br     titus     10.27.246.198
getfqdn(titus) returned: titus.petrobras.com.br
 

and also check the logs of dbclient(on client)

-See attachment

and bprd on master server

-See attachment

AttachmentSize
logbprd.010714.zip 1.71 MB
logdbclient.010714.zip 2.85 KB
sri vani's picture

I see the below error on dbclient log:

<16> readCommFile: ERR - timed out after 900 seconds while reading from

can you plz try Increase the CLIENT_READ_TIMEOUT to 5400 to give more time and try a backup

Andre_Torres's picture

same error. I don't think it's a timeout error, because I can do a file system backup.

Marianne's picture

In addition to what everyone else has said up to now - we can see that there is a comms problem between client and master server:

Client is trying to connect to bprd on master server via pbx (port 1556):

 logconnections: BPRD CONNECT FROM 10.27.252.41.39060 TO 10.27.253.249.1556 fd = 17

It gets no response from the master server:

Failed with error:    Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the server
Two things to check:
1. Port connectivity on port 1556 in both directions. If there is a firewall, check that this port is open.
2. Forward and reverse lookup in both directions.
Have you added hosts entry on the appliance for the client?
Can the master server resolve client's IP address to hostname that appears in the Oracle policy?
 
Please post bprd log file on the master server after backup attempt has failed.
Copy the log to bprd.txt and upload as File Attachment.
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Andre_Torres's picture

Have you added hosts entry on the appliance for the client?

-Yes

Can the master server resolve client's IP address to hostname that appears in the Oracle policy?

-Yes

Please post bprd log file on the master server after backup attempt has failed.

Copy the log to bprd.txt and upload as File Attachment.

-Posted before, can't post it as .txt because in txt mode it's 25MB.

sri vani's picture

Can you please try this :

Change the file modes of the following directories to 777 - "chmod 777 <path>" where <path> is:
/usr/openv/netbackup/logs/user_ops
/usr/openv/netbackup/logs/user_ops/dbext
/usr/openv/netbackup/logs/user_ops/dbext/logs and try backup

Andre_Torres's picture

These already were in permission 777

sri vani's picture

I suspect this wld be the reason:

http://www.symantec.com/business/support/index?pag...

Review the NB_ORA_CLIENT, NB_ORA_POLICY, and NB_ORA_SCHED variables configured for use by RMAN and NetBackup.  Ensure they do not contain any spaces which will result in an argument parsing problem on the master server.

Andre_Torres's picture

No spaces. As I said before, it worked before on my old server, and NB_ORA_CLIENT, NB_ORA_POLICY, and NB_ORA_SCHED were not changed.

Marianne's picture

Please help us to understand the different IP addresses that we see in the logs.

In dbclient log, we see that the client IP address is 10.27.252.41, trying to connect to bprd at IP address 10.27.253.249. This IP address should the master's IP.

01:28:54.168 [9100] <2> logconnections: BPRD CONNECT FROM 10.27.252.41.15929 TO 10.27.253.249.1556 fd = 17

When I look at bprd log, the master IP seems to be 10.27.240.63.

01:28:01.068 [27604] <2> logconnections: BPRD ACCEPT FROM 10.27.243.247.60434 TO 10.27.240.63.1556 fd = 9

You can see that the master is logging connections to itself on IP address 10.27.240.63.

What is weird about the backup attempt from the client at 01:28, (requesting backup of policy name petrobras_titus_db_ora_jbossrep_hot_full_dia)  is that we see a backup request for the SAME policy name from HOST=PTRWSVGW01 IPSTR=10.27.243.247 at more or less the same time.

01:29:19.094 [27822] <2> vnet_cached_getnameinfo: [vnet_addrinfo.c:1904] found via getnameinfo OUR_HOST=PTRWSVGW01 IPSTR=10.27.243.247
 
01:29:19.095 [27822] <2> process_request: immediate backup request = 329199 55 root other NONE PTRWSVGW01 PTRWSVGW01 NONE petrobras_titus_db_ora_jbossrep_hot_full_dia Diario_Full 0 0 13 0 C C C C C 0 0 -1 0 0

Can you help us to understand this descrepancy in IP addresses and hostnames?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Will Restore's picture

Yes this is really confusing as none of those match this response

bpcltncmd -self  command from client

host titus: titus.petrobras.com.br at 10.27.246.198

Will Restore -- where there is a Will there is a way

Andre_Torres's picture

Both Master and Client have multiple vLans.

10.27.252.41 is the IP of Backup vLan in the client

10.27.253.249 is the IP of Backup vLan in my old Master Server

10.27.240.63 is the IP of Management vLan of my new Master Server.

Client should use 10.27.252.41 and Master should use 10.27.252.163

Will Restore's picture

>>Client should use 10.27.252.41 and Master should use 10.27.252.163

Well in that case you need to set up name resolution so both Master and Client see themselves and each other that way as the current output shows otherwise.

Will Restore -- where there is a Will there is a way

Andre_Torres's picture

I did put this on both hosts file

Marianne's picture

10.27.253.249 is the IP of Backup vLan in my old Master Server

So, your client seems to try to connect to the old master.

Carefully check ALL of the following on the client:

NB_ORA_SERV in the backup script

1st SERVER entry in bp.conf

SERVER entry in $ORACLE_HOME/bp.conf

CLIENT_NAME in bp.conf on the client

NB_ORA_CLIENT name in the backup script

If we look at bpcltncmd -pn output, is ptrbkpapl01 the hostname of the backup LAN on the new master?

expecting response from server ptrbkpapl01
ptrbkpapl01 ptrbkpapl01 10.27.240.63 46976
 
You can see that the outgoing IP address of the client is 10.27.240.63. 
Master server is resolving this IP address as  ptrbkpapl01.
What is the CLIENT_NAME in bp.conf on the client? 
 
 
You need to get hostnames and IP's fixed in all directions in order to get Oracle backup to work.
 
See:
How to verify name resolution for NetBackup (tm) systems, using the "bpclntcmd" command
http://www.symantec.com/docs/TECH27430

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Andre_Torres's picture

NB_ORA_SERV in the backup script

-New Server

1st SERVER entry in bp.conf

-New Server

 

SERVER entry in $ORACLE_HOME/bp.conf

-There wasn't bp.conf file in $ORACLE_HOME so I've copied the one from /usr/openv/netbackup/

CLIENT_NAME in bp.conf on the client

-Correct Name

NB_ORA_CLIENT name in the backup script

-Correct Name

Still getting same error

Marianne's picture

So, if you now use 'bpclntcmd -pn' on the client, is the output correct?  

 expecting response from server new-master 
   client-backup-LAN-name client-IP <port-#>

Can you see connection from client in master's bprd log directly after client's 'bpclntcmd -pn' command?

When Oracle backup is started, can you see in dbclient log which IP address is selected by the client for bprd connection? (Source and destination IP).

Please post extracts of bprd and dbclient logs that show connection attempts.

*******

One more thought:

If bpclntcmd -pn is doing the 'right thing' but Oracle backup is not, there is only one thing left to do - restart Oracle.

I have once seen a similar issue on a Windows Oracle server that was moved to a new master.
ALL of NBU config and rman scripts were changed, but Oracle backup still kept on connecting to old master. Only when Oracle was restarted did the backup request go to the new master.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links