Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Problem backup with Oracle 11 on Oracle Linux

Created: 21 May 2013 • Updated: 01 Jul 2013 | 21 comments
MMoret's picture
This issue has been solved. See solution.

Hi,

I have a new Oracle Linux 6.2 client running Oracle 11.
I cannot get the backup to work. A normal filesystem backup works just fine.

In the log near the end there is a problem with a file I do not understand, all the permissions are in order.

# ls -l /usr/openv/netbackup/logs/user_ops/dbext/logs/8231.0.1369146890
-rw-r--r-- 1 oracle oinstall 42 May 21 16:34 /usr/openv/netbackup/logs/user_ops/dbext/logs/8231.0.1369146890

Here is the log:

16:49:18.775 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:18.776 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:18.777 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:18.777 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:18.778 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:18.778 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:18.779 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:18.779 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:18.780 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:18.780 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:18.781 [11803] <8> xbsa_GetEnv: WRN - NBBSA_SCHEDULE not found in environment block
 
16:49:18.781 [11803] <2> int_LogSystemInfo: INF -
Veritas NetBackup for Oracle - Release 7.5 (2012060523)
        System name:    Linux
        Node name:      sv-ora15-p.domain.local
        Release:        2.6.39-200.32.1.el6uek.x86_64
        Version:        #1 SMP Wed Sep 26 23:11:38 PDT 2012
        Machine:        x86_64
        User name:      oracle
        Client Host:    sv-ora15-p.domain.local
 
16:49:18.781 [11803] <2> int_GetMMInfo: INF - Initialized Signal
16:49:18.781 [11803] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
16:49:19.549 [11803] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_CLIENT> cmd_val=<sv-ora15-p.domain.local>
16:49:19.549 [11803] <2> int_ProcessCommand: INF - Client <sv-ora15-p.domain.local> will be used for this API session.
16:49:19.549 [11803] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SID> cmd_val=<tmeld>
16:49:19.549 [11803] <2> int_ProcessCommand: INF - SID <tmeld> will be used for metadata collection.
16:49:19.549 [11803] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_SERV> cmd_val=<sv-bu01-p.domain.local>
16:49:19.549 [11803] <2> int_ProcessCommand: INF - Server <sv-bu01-p.domain.local> will be used for this API session.
16:49:19.549 [11803] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_POLICY> cmd_val=<P_Oracle_DBs>
16:49:19.549 [11803] <2> int_ProcessCommand: INF - Policy <P_Oracle_DBs> will be used for this API session.
16:49:19.550 [11803] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_PARENT_JOBID> cmd_val=<39461>
16:49:19.550 [11803] <2> int_ProcessCommand: INF - Parent Job ID <39461> will be used for this API session.
16:49:19.638 [11803] <4> int_ProcessCommandString: INF - cmd_key=<NB_ORA_PLOG> cmd_val=</usr/openv/netbackup/logs/user_ops/dbext/oracle/progress.1369147
749.11697.log>
16:49:19.638 [11803] <2> int_ProcessCommand: INF - Progress Log </usr/openv/netbackup/logs/user_ops/dbext/oracle/progress.1369147749.11697.log> will be
used for this API session.
16:49:20.552 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:20.552 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:20.558 [11803] <2> vnet_pbxConnect: pbxConnectEx Succeeded
16:49:20.561 [11803] <2> logconnections: BPRD CONNECT FROM 10.100.100.77.23558 TO 10.100.100.120.1556 fd = 11
16:49:20.653 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:20.653 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:20.654 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:20.654 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:20.660 [11803] <2> vnet_pbxConnect: pbxConnectEx Succeeded
16:49:20.663 [11803] <2> logconnections: BPRD CONNECT FROM 10.100.100.77.37045 TO 10.100.100.120.1556 fd = 11
16:49:21.369 [11803] <16> writeToServer: ERR - send() to server on socket failed: Bad file descriptor (9)
16:49:21.369 [11803] <16> dbc_RemoteWriteFile: ERR - could not write progress status message to the NAME socket
16:49:21.369 [11803] <16> VxBSABeginProgressLogging: ERR - Unable to write to progress file.
16:49:21.369 [11803] <8> xbsa_ProgressLogSetup: WRN - VxBSABeginProgressLogging: Failed with error:
   Server Status:  system error occurred
16:49:21.369 [11803] <8> xbsa_ProgressLogSetup: WRN - Job will proceed without progress logging
16:49:21.370 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:21.370 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:21.371 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:21.371 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:21.372 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:21.372 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:21.373 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:21.373 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:21.375 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:21.376 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:21.376 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:21.376 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:21.380 [11803] <2> vnet_pbxConnect: pbxConnectEx Succeeded
16:49:21.382 [11803] <2> logconnections: BPRD CONNECT FROM 10.100.100.77.50826 TO 10.100.100.120.1556 fd = 12
16:49:21.384 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:21.384 [11803] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:51.391 [8231] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/8231.0.1369146
890
16:49:51.391 [8231] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/8231.0.1369146890>
16:49:51.391 [8231] <16> CreateNewImage: ERR - serverResponse() failed
16:49:51.391 [8231] <16> VxBSACreateObject: ERR - Could not create new image with file /04oa6vpa_1_1.
16:49:51.391 [8231] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
16:49:51.395 [8231] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.436: errno: 2 2 0x00000002
16:49:51.395 [8231] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.437: path: /home/oracle/bp.conf
16:49:51.396 [8231] <4> sbtend: INF - --- END of SESSION ---
16:49:51.396 [8231] <8> close_image: Session being terminated abnormally, cleaning up
16:49:51.396 [8231] <4> close_image: INF - backup FAILED
16:49:51.396 [8231] <4> close_image: INF ---- end of Backup ---
 
16:49:51.396 [8231] <16> VxBSAEndTxn: ERR - Transaction ended with active Backup/Restore.
16:49:51.396 [8231] <16> xbsa_EndTransaction: ERR - VxBSAEndTxn: Failed with error:
   The transaction was aborted.
 
Any help??
 
Thanks in advance!
Martijn
Operating Systems:

Comments 21 CommentsJump to latest comment

huanglao2002's picture

1 Are you set up the oracle backup policy correct?

2 From the error log

/home/oracle/bp.conf

Netbackup try to  get some info from oracle home bp.conf file.

3  From logs

16:49:21.369 [11803] <16> writeToServer: ERR - send() to server on socket failed: Bad file descriptor (9)
16:49:21.369 [11803] <16> dbc_RemoteWriteFile: ERR - could not write progress status message to the NAME socket
16:49:21.369 [11803] <16> VxBSABeginProgressLogging: ERR - Unable to write to progress file.
16:49:21.369 [11803] <8> xbsa_ProgressLogSetup: WRN - VxBSABeginProgressLogging: Failed with error:
Server Status: system error occurred

You can theck /usr/openv/netbackup/logs directory permission.

4. in my test env,you can reference

[root@ logs]# ls -ld user_ops
drwxrwxrwx 5 root bin 4096 05-18 22:31 user_ops

[root@ user_ops]# ls -l
总计 12
drwxrwxrwx 4 root root 4096 04-22 10:22 dbext
drwxrwxrwx 2 root bin  4096 05-22 00:04 nbjlogs
drwxr-x--x 4 root root 4096 01-28 09:08 root
[root@redhatovocs user_ops]# ls -lR dbext/
dbext/:
总计 8
drwxrwxrwx 2 root root 4096 05-22 00:02 jobs
drwxrwxrwx 2 root root 4096 05-22 00:02 logs

dbext/jobs:
总计 12
-rw-r--r-- 1 root root 177 05-20 00:00 vxbsa.1368979208.13111.prog.pcb_std.j
-rw-r--r-- 1 root root 177 05-21 00:00 vxbsa.1369065610.7885.prog.pcb_std.j
-rw-r--r-- 1 root root 177 05-22 00:04 vxbsa.1369152167.2886.prog.pcb_std.j

dbext/logs:
总计 36
-rw-r--r-- 1 root root 2726 05-20 00:00 vxbsa.1368979208.13111.files.1
-rw-r--r-- 1 root root 4994 05-20 00:00 vxbsa.1368979208.13111.prog.pcb_std
-rw-r--r-- 1 root root 2726 05-21 00:00 vxbsa.1369065610.7885.files.1
-rw-r--r-- 1 root root 4994 05-21 00:00 vxbsa.1369065610.7885.prog.pcb_std
-rw-r--r-- 1 root root 2726 05-22 00:02 vxbsa.1369152167.2886.files.1
-rw-r--r-- 1 root root 4994 05-22 00:04 vxbsa.1369152167.2886.prog.pcb_std
[root@redhatovocs user_ops]#

Marianne's picture
16:49:20.663 [11803] <2> logconnections: BPRD CONNECT FROM 10.100.100.77.37045 TO 10.100.100.120.1556 fd = 11
16:49:21.369 [11803] <16> writeToServer: ERR - send() to server on socket failed: Bad file descriptor (9)
Oracle client seems to be unable to connect to bprd on the master server via port 1556.
Is that IP address (10.100.100.120) correct for master server?
 
Firstly, check that port 1556 is open in both directions between master and client.
 
Next, verify correct forward and reverse name lookup between master and client.
Use 'bpclntcmd -hn <name>' and 'bpclntcmd -ip <ip-address>' in both directions to check.
If you need to change DNS or hosts entries to fix lookup issues, remember to clear host cache:
bpclntcmd -clear_host_cache
 
Hope this helps.
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

MMoret's picture

Thanks for your replies!

The communication is fine, normal file level backups run ok.

the 10.100.100.120 is the Master server.

[root@sv-ora15-p bin]# ./bpclntcmd -hn sv-bu01-p
host sv-bu01-p: sv-bu01-p.domain.local at 10.100.100.120
aliases:     sv-bu01-p.domain.local     sv-bu01-p     10.100.100.120
[root@sv-ora15-p bin]# ./bpclntcmd -ip 10.100.100.120
host 10.100.100.120: sv-bu01-p.domain.local at 10.100.100.120
aliases:     sv-bu01-p.domain.local     10.100.100.120
 
C:\Users\adm>bpclntcmd -hn sv-ora15-p
host sv-ora15-p: sv-ora15-p.domain.local at 10.100.100.77
aliases:     sv-ora15-p.domain.local     sv-ora15-p     10.100.100.77
 
C:\Users\adm>bpclntcmd -ip 10.100.100.77
host 10.100.100.77: sv-ora15-p.domain.local at 10.100.100.77
aliases:     sv-ora15-p.domain.local     10.100.100.77
Marianne's picture

Comms for file system backups work different - there is no connection that is initiated from the client directly to the master.

Your master is resolving client's IP address as FQDN: sv-ora15-p.domain.local.
How is hostname defined in the Oracle policy? Shortname or FQDN?

To know what is happening on the master server, we need to see bprd log.
If the log folder does not exist, please create it and restart NBU.

After next failure, copy bprd log to bprd.txt and post as file attachment.
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

MMoret's picture

The client is FQDN in the policy.

Will generate the log file.

Thanks!
Martijn

MMoret's picture

Hi,

Here is the bprd log file from the master server.

Regards,
Martijn

AttachmentSize
bprd.zip 7.06 KB
Marianne's picture

OK - no comms errors.

This is what I see in bprd log:

bkarfiles: User backup failed (client = sv-ora15-p.domain.local user = oracle group = oinstall): system error occurred

add_msg_to_progress_file: Can't become user oracle and group oinstall on sv-ora15-p.domain.local 

Can Oracle dba login as user oracle (group oinstall) and run rman backup commands?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

MMoret's picture

Yes, we only have two accounts (root and oracle).

[root@sv-ora15-p ~]# rman target backup/*****@db nocatalog
 
Recovery Manager: Release 11.2.0.3.0 - Production on Wed May 22 10:59:35 2013
 
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
 
connected to target database: DB (DBID=2014933852)
using target database control file instead of recovery catalog
 
RMAN>
 
[oracle@sv-ora15-p ~]$ rman target backup/*****@db nocatalog
 
Recovery Manager: Release 11.2.0.3.0 - Production on Wed May 22 11:00:25 2013
 
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
 
connected to target database: DB (DBID=2014933852)
using target database control file instead of recovery catalog
 
RMAN>
Marianne's picture

Do you have bphdb and dbclient logs on the client?

If not, create folders and chmod 777 on both of them.

This will hopefully tell us why NBU believes that it cannot su to oracle user.

Please also post the rman script and/or template.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

MMoret's picture

Hi,

I really appreciate your effort!

Here are the logs and template.

Thanks again!
Regards, Martijn

AttachmentSize
Logs_template.zip 7.34 KB
MMoret's picture

I also tried a scripted backup, the same result, RMAN starts and times out with no data transfer.

Any more ideas?

Regards,
Martijn

Marianne's picture

Apologies - I have not had a chance to look at the logs yet.

What are the Client Connect and Client Read timeouts on the media server?

The default of 300 (5 min) is normally not enough for large databases.
We have good experience with timeout of 1800 for both settings.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

MMoret's picture

Hi,

I found something weird, this customer has two servers running fine Oracle backups.
When checking differences, I found that I cannot login to the Java interface with the oracle account.

This is not a problem on the servers running fine.

Error message: 

ServerInterface:setDebugLevel:262144

BpjavaLoginModule:Setting ServerRequest debug:262144

BpjavaLoginModule:connectServer:[oracle][<users_pw>][sv-ora15-p.domain.local][0]

Connecting to vnetd service over PBX port = 1556

Acknowledgement from PBX1

        Protocol Code: 101

        Status: 31

        Time Taken: 194ms

        Error Msg: could not set user id for process

        Server Locale: en_US.UTF-8

        TO[0]: oracle

        TO[1]: sv-ora15-p.domain.local

        TO[2]: en_US.UTF-8

        TO[3]: XXXX

        TO[4]: auth.conf

        TO[5]: 750000 IPC

        FROM[0]: could not set user id for process

        Aux data: null

error_java_interface.png
huanglao2002's picture

status code 31 explain

NetBackup status code: 31

Message: could not set user id for process

Explanation: Could not set the user ID of a process to the user ID of the requesting user. NetBackup runs client processes as the requesting user.

Recommended Action: Check the NetBackup All Log Entries report for clues on where and why the failure occurred. For detailed troubleshooting information, create a debug log directory for the process that you think may have returned this status code. Then, retry the operation and check the resulting debug log.

Can you chek the oracle id on the both host,check the user/user group difference.

#id oracle

Marianne's picture

The NBU status code 31 'Recommended Action' does not help much.... 

I have in all honesty never seen this error and have no idea where to go from here.

Please compare user profiles - maybe restricted shell? 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

MMoret's picture

Hi,

Thanks for your comments.
The problem solved by the DBA, the oracle account did not have enough resources assigned.

Regards,
Martijn

Marianne's picture

Oracle user account did not have enough resources???

Please explain??

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

MMoret's picture

Hi,

This is what I got back from the DBA:

As there are many databases and instances,
The following files are modified:

/etc/security/limits.conf
/etc/security/limits.d/90-nproc.conf

65k to 125k (approx.)

I have asked for detailed adjustments to the server.

Regards,

Martijn

SOLUTION
Will Restore's picture

bad bug!

Description of problem:

/etc/security/limits.d/90-nproc.conf was introduced in pam package in RH6. It contains this line:

# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     1024

This line overrides the conventionally set /etc/security/limits.conf value of the same name. Years of expected behavior are thrown out the window and honest system administrators are exposed to outages on Redhat 6!

https://bugzilla.redhat.com/show_bug.cgi?id=919793

Will Restore -- where there is a Will there is a way

nasahasha's picture

Hi,

I had same problem, file system backups work fine and oracle backups fail.Netbackup version 7.5.0.5 and oracle 11.1.0.7 on RHEL.

I tried

1)Increasing time outs didnt worked.

2)recycling services reinstalling on client didnt work.

3)ports for 1556 were open

4)bpclntcmd -hn/ip worked fine

5)There are other clients with DB on them, all are working fine, only this client has problem

but bpclntcmd -pn didnt worked, then I concluded that client is not able to connect to bprd on master but dont know why?

For some reason I had to recycle netbackup services on master server and then oracle backups completed fine.

appreciate your help in finding what went wrong and why was it not able to connect to bprd, and Do I have to recycle services on master when I see this again?

Below is dbclient log 

root@medcdedb4:/usr/openv/netbackup/logs/dbclient
08:55:14 # ls
log.062813
root@medcdedb4:/usr/openv/netbackup/logs/dbclient
08:55:38 # cat log.062813
07:22:14.697 [63217] <2> int_LogSystemInfo: INF -
Veritas NetBackup for Oracle - Release 7.5 (2012091610)
        System name:    Linux
        Node name:      medcdedb4.cernerasp.com
        Release:        2.6.18-194.32.1.el5
        Version:        #1 SMP Mon Dec 20 10:52:42 EST 2010
        Machine:        x86_64
        User name:      oracle
        Client Host:    medcdedb4_bk.cernerasp.com
 
07:22:14.697 [63217] <2> int_GetMMInfo: INF - Initialized Signal
07:22:14.697 [63217] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
07:22:20.513 [63389] <2> int_LogSystemInfo: INF -
Veritas NetBackup for Oracle - Release 7.5 (2012091610)
        System name:    Linux
        Node name:      medcdedb4.cernerasp.com
        Release:        2.6.18-194.32.1.el5
        Version:        #1 SMP Mon Dec 20 10:52:42 EST 2010
        Machine:        x86_64
        User name:      oracle
        Client Host:    medcdedb4_bk.cernerasp.com
 
07:22:20.514 [63389] <2> int_GetMMInfo: INF - Initialized Signal
07:22:20.514 [63389] <2> int_GetMMInfo: INF - support for Proxy Copy enabled
07:22:36.529 [63389] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.428: errno: 2 2 0x00000002
07:22:36.529 [63389] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.429: path: /home/oracle/bp.conf
07:22:36.530 [63389] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/04d/359bc4d+veritas_pbx,1,20,2,1,0+taspmonby01_bk.cernerasp.com.txt
07:22:36.533 [63389] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0d5/374a20d5+0,1,402,2,1,0+medcdedb4_bk.cernerasp.com.txt
07:22:36.533 [63389] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0d5/374a20d5+0,1,20,2,1,0+medcdedb4_bk.cernerasp.com.txt
07:22:36.533 [63389] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/04d/359bc4d+vnetd,1,20,2,1,0+taspmonby01_bk.cernerasp.com.txt
07:22:36.533 [63389] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/04d/359bc4d+bprd,1,20,2,1,0+taspmonby01_bk.cernerasp.com.txt
07:22:36.535 [63389] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:22:36.535 [63389] <2> logconnections: BPRD CONNECT FROM 10.159.82.133.10100 TO 170.71.0.16.1556 fd = 25
07:22:36.549 [63389] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
07:22:36.553 [63389] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:22:36.553 [63389] <2> logconnections: BPRD CONNECT FROM 10.159.82.133.21569 TO 170.71.0.16.1556 fd = 26
07:22:36.720 [63217] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.428: errno: 2 2 0x00000002
07:22:36.720 [63217] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.429: path: /home/oracle/bp.conf
07:22:36.721 [63217] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/04d/359bc4d+veritas_pbx,1,20,2,1,0+taspmonby01_bk.cernerasp.com.txt
07:22:36.722 [63217] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0d5/374a20d5+0,1,402,2,1,0+medcdedb4_bk.cernerasp.com.txt
07:22:36.723 [63217] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/0d5/374a20d5+0,1,20,2,1,0+medcdedb4_bk.cernerasp.com.txt
07:22:36.723 [63217] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/04d/359bc4d+vnetd,1,20,2,1,0+taspmonby01_bk.cernerasp.com.txt
07:22:36.723 [63217] <8> file_to_cache_item: [vnet_addrinfo.c:6555] fopen() failed ERRNO=2 FILE=/usr/openv/var/host_cache/04d/359bc4d+bprd,1,20,2,1,0+taspmonby01_bk.cernerasp.com.txt
07:22:36.724 [63217] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:22:36.724 [63217] <2> logconnections: BPRD CONNECT FROM 10.159.82.133.24861 TO 170.71.0.16.1556 fd = 25
07:22:36.736 [63217] <2> xbsa_ProgressLogSetup: INF - No progress log: Job will proceed without progress logging
07:22:36.738 [63217] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:22:36.738 [63217] <2> logconnections: BPRD CONNECT FROM 10.159.82.133.31219 TO 170.71.0.16.1556 fd = 26
07:22:43.561 [63389] <8> verify_hashes: [vnet_vnetd.c:1641] hash_str 1f405a04db15f269485de7f3d0e99f65
07:22:43.561 [63389] <8> verify_hashes: [vnet_vnetd.c:1641] hash_str 58602a0e2dc1dba8e42c1f98ae195031
07:22:43.562 [63389] <8> verify_hashes: [vnet_vnetd.c:1641] hash_str 6b5897864dd49a5c8f3663aff1b5aa66
07:22:44.746 [63217] <8> verify_hashes: [vnet_vnetd.c:1641] hash_str 01b90b4a3d1957d1523e3d9e1a03ec5f
07:22:44.747 [63217] <8> verify_hashes: [vnet_vnetd.c:1641] hash_str c7fabdc636daa2fe29d8d81f8ac7efe0
07:22:44.747 [63217] <8> verify_hashes: [vnet_vnetd.c:1641] hash_str b3f44568ef27d6f86f91cd74e37fb04e
07:37:45.511 [63389] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/63389.0.1372418556
07:37:46.693 [63217] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/63217.0.1372418556
07:37:50.514 [63389] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.428: errno: 2 2 0x00000002
07:37:50.514 [63389] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.429: path: /home/oracle/bp.conf
07:37:51.696 [63217] <16> CopyToShM: connection broken. Error #40
07:37:51.696 [63217] <16> writeToServer: ERR - CopyToShM() failed, no files backed up
07:37:51.696 [63217] <16> write_xbsa: ERR - can't write XBSA data to DATA socket
07:37:51.696 [63217] <16> handshake: ERR - writeTarHeader() failed
07:37:51.696 [63217] <16> CreateNewImage: ERR - handshake() failed
07:37:51.736 [63217] <16> VxBSACreateObject: ERR - Could not create new image with file /C01821_ARC_201306280722_46oda5ls_1_1819271356.
07:37:51.736 [63217] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
07:52:52.420 [63389] <16> readCommFile: ERR - timed out after 900 seconds while reading from /usr/openv/netbackup/logs/user_ops/dbext/logs/63389.0.1372418556
07:52:52.420 [63389] <32> serverResponse: ERR - could not read from comm file </usr/openv/netbackup/logs/user_ops/dbext/logs/63389.0.1372418556>
07:52:52.420 [63389] <16> CreateNewImage: ERR - serverResponse() failed
07:52:52.420 [63389] <16> VxBSACreateObject: ERR - Could not create new image with file /C01821_ARC_201306280722_47oda5ls_1_1819271356.
07:52:52.420 [63389] <16> xbsa_CreateObject: ERR - VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
07:52:57.646 [63217] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.428: errno: 2 2 0x00000002
07:52:57.646 [63217] <2> conf_update_time: ../../libvlibs/nbconf_glue.cpp.429: path: /home/oracle/bp.conf
07:52:57.646 [63217] <4> sbtend: INF - --- END of SESSION ---
07:52:57.646 [63217] <8> close_image: Session being terminated abnormally, cleaning up
07:52:57.646 [63217] <4> CleanUpShM: INF - dbclient waited 0 times for empty buffer, delayed 0 times
07:52:57.646 [63217] <16> CopyToShM: connection broken. Error #40
07:52:57.646 [63217] <16> writeToServer: ERR - CopyToShM() failed, no files backed up
07:52:57.646 [63217] <16> CleanUpShM: ERR - writeToServer() failed attempting to contact the server
07:52:57.646 [63217] <4> close_image: INF - backup FAILED
07:52:57.646 [63217] <4> close_image: INF ---- end of Backup ---

Thank you in Advance.

Naseer

inn_kam's picture

if u can update the latest patch, may be it helps