Video Screencast Help

Netbackup Error

Created: 06 Sep 2012 • Updated: 16 Sep 2012 | 24 comments
This issue has been solved. See solution.

Dear Friends

I am using Netbackup 7.0.

While doing Robot inventory, I am getting an error : EMM Database Error (Status 196) and The Enterprise Media Manager Server is not responding (Status 25).

The following Daemons has stopped running.

bmrd,nbsvcmon,nbars, spad,nbrb,spoold,nbemm,nbaudit,nbkms.

Kindly advice, how to rectify these error.

Thank You

 

Comments 24 CommentsJump to latest comment

girish_nair1984's picture

While running this command tpautoconf –report_disc

I am getting an error : EMM interface initialization failed, status 77

mph999's picture

Stop NBU (what's left of it)

Reboot Server

What NBU processes are running ?

 

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
girish_nair1984's picture

I created a log folder (/usr/openv/netbackup/logs/admin) the following log was generated

15:24:02.823 [26398] <4> db_getSTUNITlist: emmserver_name = his1

15:24:02.823 [26398] <4> db_getSTUNITlist: emmserver_port = 1556

15:24:12.266 [26410] <2> bprdreq: INITIATING: version NetBackup 7.0 created: 2010070800

15:24:12.266 [26410] <2> logparams: /usr/openv/netbackup/bin/admincmd/bprdreq -M localhost -terminate

15:24:12.267 [26410] <2> bprdreq: VERBOSE = 0

15:24:12.269 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5661: 0: fopen() failed: 2 0x00000002

15:24:12.269 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5662: 0: fopen() failed: /usr/openv/var/host_cache/06b/c413626b+veritas_pbx,1,20,2,1,0+localhost.txt

15:24:12.270 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5661: 0: fopen() failed: 2 0x00000002

15:24:12.270 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5662: 0: fopen() failed: /usr/openv/var/host_cache/06b/c413626b+vnetd,1,20,2,1,0+localhost.txt

15:24:12.271 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5661: 0: fopen() failed: 2 0x00000002

15:24:12.271 [26410] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.5662: 0: fopen() failed: /usr/openv/var/host_cache/06b/c413626b+bprd,1,20,2,1,0+localhost.txt

15:24:12.272 [26410] <2> vnet_pbxConnect: pbxConnectEx Succeeded

15:24:12.272 [26410] <2> logconnections: BPRD CONNECT FROM 127.0.0.1.19974 TO 127.0.0.1.1556 fd = 4

15:24:12.377 [26414] <2> nbpemreq: INITIATING: version NetBackup 7.0 created: 2010070800

15:24:12.381 [26414] <2> logparams: /usr/openv/netbackup/bin/admincmd/nbpemreq -suspend_scheduling

15:24:12.390 [26414] <2> Orb::init: initializing ORB Default_CLIENT_Orb with: Unknown -ORBSvcConfDirective "-ORBDottedDecimalAddresses 0" -ORBSvcConfDirective "static PBXIOP_Factory '-enable_keepalive'" -ORBSvcConfDirective "static EndpointSelectorFactory ''" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory PBXIOP_Factory'" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory IIOP_Factory'" -ORBSvcConfDirective "static PBXIOP_Evaluator_Factory '-orb Default_CLIENT_Orb'" -ORBSvcConfDirective "static Resource_Factory '-ORBConnectionCacheMax 1024 '" -ORBSvcConf /dev/null -ORBSvcConfDirective "static Server_Strategy_Factory '-ORBMaxRecvGIOPPayloadSize 268435456'"(Orb.cpp:759)

15:24:13.306 [26410] <2> doreq: ...ok, received 0 from bprd

15:24:16.307 [26410] <2> bprdreq: EXIT status = 0

15:25:01.896 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT

15:25:01.897 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'

OMG minor code (2), described as '*unknown description*', completed = NO

15:26:05.953 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT

15:26:05.953 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'

OMG minor code (2), described as '*unknown description*', completed = NO

15:27:10.007 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT

15:27:10.007 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'

OMG minor code (2), described as '*unknown description*', completed = NO

15:28:19.172 [27916] <2> nbpemreq: INITIATING: version NetBackup 7.0 created: 2010070800

15:28:19.172 [27916] <2> logparams: /usr/openv/netbackup/bin/admincmd/nbpemreq -resume_scheduling

15:28:19.180 [27916] <2> Orb::init: initializing ORB Default_CLIENT_Orb with: Unknown -ORBSvcConfDirective "-ORBDottedDecimalAddresses 0" -ORBSvcConfDirective "static PBXIOP_Factory '-enable_keepalive'" -ORBSvcConfDirective "static EndpointSelectorFactory ''" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory PBXIOP_Factory'" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory IIOP_Factory'" -ORBSvcConfDirective "static PBXIOP_Evaluator_Factory '-orb Default_CLIENT_Orb'" -ORBSvcConfDirective "static Resource_Factory '-ORBConnectionCacheMax 1024 '" -ORBSvcConf /dev/null -ORBSvcConfDirective "static Server_Strategy_Factory '-ORBMaxRecvGIOPPayloadSize 268435456'"(Orb.cpp:759)

15:28:31.001 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::COMM_FAILURE

15:28:31.001 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/COMM_FAILURE:1.0'

TAO exception, minor code = 13 (failed to recv request response; ECONNRESET), completed = MAYBE

15:28:35.183 [28198] <4> db_getSTUNITlist: emmserver_name = his1

15:28:35.183 [28198] <4> db_getSTUNITlist: emmserver_port = 1556

15:29:12.327 [28263] <2> logparams: /usr/openv/netbackup/bin/admincmd/bpversion

15:29:12.331 [28263] <2> vnet_pbxConnect: pbxConnectEx Succeeded

15:29:12.331 [28263] <2> logconnections: BPCD CONNECT FROM 192.168.1.13.26281 TO 192.168.1.13.1556 fd = 3

15:29:12.332 [28263] <2> vnet_pbxConnect: pbxConnectEx Succeeded

15:29:12.333 [28263] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1716: 0: via PBX: VNETD CONNECT FROM 192.168.1.13.50004 TO 192.168.1.13.1556 fd = 4

15:29:12.333 [28263] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.540: 0: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a

15:29:12.373 [28263] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.557: 0: ipc_string: /tmp/vnet-28265346930952373257000000000-x3xOeR

15:29:12.413 [28263] <2> local_bpcr_connect: expected reserved port, got 1556

15:29:12.523 [28263] <2> vnet_pbxConnect: pbxConnectEx Succeeded

15:29:12.523 [28263] <2> logconnections: BPCD CONNECT FROM 192.168.1.13.43878 TO 192.168.1.13.1556 fd = 3

15:29:12.523 [28263] <2> vnet_pbxConnect: pbxConnectEx Succeeded

15:29:12.524 [28263] <2> do_pbx_service: ../../libvlibs/vnet_connect.c.1716: 0: via PBX: VNETD CONNECT FROM 192.168.1.13.22652 TO 192.168.1.13.1556 fd = 4

15:29:12.524 [28263] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.540: 0: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a

15:29:12.564 [28263] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.557: 0: ipc_string: /tmp/vnet-28269346930952564247000000000-pYOPo5

15:29:12.604 [28263] <2> local_bpcr_connect: expected reserved port, got 1556

15:29:34.259 [28198] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT

15:29:34.259 [28198] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'

OMG minor code (2), described as '*unknown description*', completed = NO

15:29:35.064 [26398] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT

15:29:35.064 [26398] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'

OMG minor code (2), described as '*unknown description*', completed = NO

15:29:40.064 [26398] <16> db_getSTUNITlist: (-) Translating EMM_ERROR_CorbaTransient(3000001) to 25 in the NetBackup context

15:29:40.064 [26398] <16> db_getSTUNITlist: error (25) initializing EMM interface.

15:29:40.064 [26398] <16> bpstsinfo/update ERROR: unable to retrieve stu list from server (25)

15:30:38.315 [28198] <16> emmlib_initializeEx: (-) Exception! CORBA::TRANSIENT

15:30:38.315 [28198] <16> emmlib_initializeEx: (-) system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'

OMG minor code (2), described as '*unknown description*', completed = NO

girish_nair1984's picture

The following daemons are running

ltid

nbsl

bprd

vmd

bocompatd

nbstserv

nbsvsmon

NB_dbsrv

nbrmms

bpdbm

nbjm

nbpem

nbevtmgr

Mark_Solutions's picture

Corba Transient Error .... Sounds like you may be out of memory ... reboot at least!

Are you really on 7.0? I would advise you patch at least or preferably upgrade

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

mph999's picture

I meant reboot, then tell us what processes are runing ...

 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
girish_nair1984's picture

Yes I am using Netbackup 7.0.

 

But we have 2 Netback Servers with same partition size. 1 is working perfectly fine while another one is showing the above mentioned error.s

girish_nair1984's picture

After using this command /usr/openv/netbackup/bin/goodies/ netbackup stop

 and then bpps -a to verify the processes/daemons are running

 

NB Processes

------------

root      1066     1  0 12:13 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7

root      1085  1066  0 12:13 ?        00:00:00 [NB_dbsrv] <defunct>

root      1217     1  0 12:13 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7

root      1223  1217  0 12:13 ?        00:00:00 [NB_dbsrv] <defunct>

root      4714     1  0 12:28 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7

root      4719  4714  0 12:28 ?        00:00:00 [NB_dbsrv] <defunct>

root      4966     1  0 12:28 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7

root      4968  4966  0 12:28 ?        00:00:00 [NB_dbsrv] <defunct>

root     12441     1  0 Sep05 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7

root     12443 12441  0 Sep05 ?        00:00:00 [NB_dbsrv] <defunct>

root     13484     1  0 13:15 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem

root     28843 28802  0 15:34 pts/5    00:00:09 /usr/openv/java/jre/bin/java -Dvrts.NBJAVA_CONF=/usr/openv/java/nbj.conf -showversion -Dvrts.common.server.LCCTYPE="en_US" -Dvrts.common.server.LOG_CMDLINES=false -Djava.library.path=/usr/openv/lib -Dvrtsat.home=/usr/openv/netbackup/sec/at -Dvrts.common.utilities.DEBUG_PROPERTIES=/usr/openv/java/Debug.properties -Dvrts.nbe.HOSTNAME= -Dvrts.nbe.INST_REL=700000 -Dvrts.nbe.BUILDNO=20100707 -Djava.util.prefs.PreferencesFactory=vrts.common.launch.PropertiesPreferencesFactory -Xms36M -Xmx256M vrts.nbe.AdminConsole

root     28885     1  0 15:34 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 28882 noUserCredentialsFile

root     28887 28885  0 15:34 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 28882 noUserCredentialsFile

root     28893 28885  0 15:34 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 28882 noUserCredentialsFile

root     28967 28885  0 15:35 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 28882 noUserCredentialsFile

root     29471     1  0 15:40 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem

Mark_Solutions's picture

Have you rebooted yet as we advised? Will clear those orphaned processes that may be causing the memory / Corba errors - if they persist after the reboot you may also need to delete the orphaned corba file - but let us know first before we cover that

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

girish_nair1984's picture

@ Mark_Solutions.

Yes I have rebooted as per your advice, but still that error is there.

 

Marianne's picture

Have you checked disk space on the master server?  

EMM will shut down when filesystem runs low on disk space to prevent corruption of database.

Is PBX running?
Use 'bpps -x' to list all processes.

After stopping and starting NBU, check EMM logs for the last 10 minutes (or enough minutes to capture stop and start):

vxlogview -o 111 -t 00:10:00

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

badri's picture

by any chance was the server renamed or added to a different domain?

girish_nair1984's picture

@ Marianne van den Berg

Please find the logs after running bpps -x command

NB Processes
------------
root      1066     1  0 Sep06 ?    00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf 
@/usr/openv/var/global/databases.conf -hn 7
root      1085  1066  0 Sep06 ?        00:00:00 [NB_dbsrv] <defunct>
root      1217     1  0 Sep06 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv /usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root      1223  1217  0 Sep06 ?        00:00:00 [NB_dbsrv] <defunct>
root      4714     1  0 Sep06 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root      4719  4714  0 Sep06 ?        00:00:00 [NB_dbsrv] <defunct>
root     4966     1  0 Sep06 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv 
@/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root      4968  4966  0 Sep06 ?        00:00:00 [NB_dbsrv] <defunct>
root     12441     1  0 Sep05 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv 
@/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root     12443 12441  0 Sep05 ?        00:00:00 [NB_dbsrv] <defunct>
root     13484     1  0 Sep06 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root     15203     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -standalone
root     15206     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/bpcd -standalone
root     15269     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/nbevtmgr
root     15535     1  0 10:22 pts/3    00:00:00 /usr/openv/netbackup/bin/bprd
root     15540     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/bpcompatd
root     15551     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/nbjm
root     15572     1  0 10:22 pts/3    00:00:00 /usr/openv/netbackup/bin/bpdbm
root     15579 15572  0 10:22 pts/3    00:00:00 /usr/openv/netbackup/bin/bpjobd
root     15589     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/nbpem
root     15642     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/nbstserv
root     15690     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/nbrmms
root     15765     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/nbsl
root     16035     1  0 10:22 ?        00:00:00 /usr/openv/netbackup/bin/nbsvcmon
root     16263 16222  1 10:23 pts/3    00:00:05 /usr/openv/java/jre/bin/java - 
Dvrts.NBJAVA_CONF=/usr/openv/java/nbj.conf -showversion - 
Dvrts.common.server.LCCTYPE="en_US" - 
Dvrts.common.server.LOG_CMDLINES=false - 
Djava.library.path=/usr/openv/lib - 
Dvrtsat.home=/usr/openv/netbackup/sec/at - 
Dvrts.common.utilities.DEBUG_PROPERTIES=/usr/openv/java/Debug.properties -Dvrts.nbe.HOSTNAME= - Dvrts.nbe.INST_REL=700000 -Dvrts.nbe.BUILDNO=20100707 -Djava.util.prefs.PreferencesFactory=vrts.common.launch.PropertiesPreferencesFactory -Xms36M -Xmx256M vrts.nbe.AdminConsole
root     16314     1  0 10:23 ?  00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 
en_US /usr/openv/java/auth.conf 1 -1 16311 noUserCredentialsFile
root     16316 16314  0 10:23 ?        00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 
en_US /usr/openv/java/auth.conf 1 -1 16311 noUserCredentialsFile
root     16321 16314  0 10:23 ?    00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 16311 noUserCredentialsFile
root     16374 16314  0 10:23 ?     00:00:00 /usr/openv/netbackup/bin/bpjava-susvc root -1 -1 en_US /usr/openv/java/auth.conf 1 -1 16311 noUserCredentialsFile
root     16878     1  0 10:27 pts/3    00:00:00 /usr/openv/netbackup/bin/admincmd/bpstsinfo -UPDATE
root     17300 15589  0 10:30 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
root     29471     1  0 Sep06 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem

MM Processes
------------
root     15530     1  0 10:22 pts/3    00:00:00 /usr/openv/volmgr/bin/ltid
root     15694     1  0 10:22 pts/3    00:00:00 vmd

Shared Symantec Processes
-------------------------
root      3972     1  0 Mar20 ?         00:01:37 /opt/VRTSpbx/bin/pbx_exchange
 

 

Output of vxlogview -o 111 -t 00:10:00

[root@his1 ~]# /usr/openv/netbackup/bin/vxlogview -o 111 -t 00:10:00
09/11/2012 10:22:30.691 [EMMServer::EMMServer]  
09/11/2012 10:22:30.691 [DSMFSMORBConfig::DSMFSMORBConfig] Setting DSM ORB thread count to <10>
09/11/2012 10:22:30.691 [REMORBConfig::REMORBConfig] Setting REM ORB thread count to <10>
09/11/2012 10:22:30.691 [FATClientORBConfig::FATClientORBConfig] Setting FTClient ORB thread count to <10>
09/11/2012 10:22:30.774 [EMMMain] <ENTER>
09/11/2012 10:22:30.776 [EMMServer::isLocalHostEMMServer]  
09/11/2012 10:22:30.778 [EMMServer::isLocalHostEMMServer] LocalHost = <his1>, EmmServer = <his1>
09/11/2012 10:22:30.778 [NBEmmSvc::doShouldStart] Should start
09/11/2012 10:22:30.924 [NBEmmSvc::doInit] <ENTER>
09/11/2012 10:22:30.926 [NBEmmSvc::doInit] NumberOrbThreads = <10>, NumberDbConnections = <4>, NumberDbBrowseConnections = <3>
09/11/2012 10:22:30.927 [EMMServer::SetOrb] Orb is set
09/11/2012 10:22:30.927 [NBEmmSvc::doInit] Extra orb options:  -ORBCollocation no
09/11/2012 10:22:30.927 [EMMServer::init] Server initializing VxEMM
09/11/2012 10:22:30.927 [Info] V-111-1000 [EMMServer::init] EMMServer initializing = VxEMM
09/11/2012 10:22:30.927 [EMMServer::init] Option: -DbServer *** Unrecognized option -DbServer ***
09/11/2012 10:22:30.927 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: -DbServer *** Unrecognized option -DbServer ***
09/11/2012 10:22:30.927 [EMMServer::init] Option: NB_his1 *** Unrecognized option NB_his1 ***
09/11/2012 10:22:30.927 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: NB_his1 *** Unrecognized option NB_his1 ***
09/11/2012 10:22:30.927 [EMMServer::init] Option: -DbPort *** Unrecognized option -DbPort ***
09/11/2012 10:22:30.927 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: -DbPort *** Unrecognized option -DbPort ***
09/11/2012 10:22:30.927 [EMMServer::init] Option: 13785 *** Unrecognized option 13785 ***
09/11/2012 10:22:30.927 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: 13785 *** Unrecognized option 13785 ***
09/11/2012 10:22:30.928 [EMMServer::init] Option: -DbName *** Unrecognized option -DbName ***
09/11/2012 10:22:30.928 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: -DbName *** Unrecognized option -DbName ***
09/11/2012 10:22:30.928 [EMMServer::init] Option: NBDB *** Unrecognized option NBDB ***
09/11/2012 10:22:30.928 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: NBDB *** Unrecognized option NBDB ***
09/11/2012 10:22:30.928 [EMMServer::init] Option: -ConnectString *** Unrecognized option -ConnectString ***
09/11/2012 10:22:30.928 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: -ConnectString *** Unrecognized option -ConnectString ***
09/11/2012 10:22:30.928 [EMMServer::init] Option: PROWS=600;CS=utf8;UID=EMM_MAIN;PWD=******;LINKS=shmem,tcpip{PORT=13785};ENG=NB_his1;DBN=NBDB *** Unrecognized option PROWS=600;CS=utf8;UID=EMM_MAIN;PWD=nbusql;LINKS=shmem,tcpip{PORT=13785};ENG=NB_his1;DBN=NBDB ***
09/11/2012 10:22:30.928 [Info] V-111-1001 [EMMServer::init] EMMServer initialization option = Option: PROWS=600;CS=utf8;UID=EMM_MAIN;PWD=******;LINKS=shmem,tcpip{PORT=13785};ENG=NB_his1;DBN=NBDB *** Unrecognized option PROWS=600;CS=utf8;UID=EMM_MAIN;PWD=nbusql;LINKS=shmem,tcpip{PORT=13785};ENG=NB_his1;DBN=NBDB ***
09/11/2012 10:22:30.928 [EMMServer::run] Server running
09/11/2012 10:22:30.928 [Info] V-111-1002 [EMMServer::run] EMMServer running
09/11/2012 10:22:49.923 V-111-1051 [EMMServer::run] Can not establish initial DB connection
09/11/2012 10:22:49.923 [Error] V-111-1049 EMMServer generic error = Can not establish initial DB connection
09/11/2012 10:22:49.923 [Error] V-111-1049 EMMServer generic error = controlledRun() failed -> doShouldStart(), doInitialize() or doRun() failed
09/11/2012 10:22:49.923 [FATClientORBConfig::shutServant] Calling fini() on FAT Client
09/11/2012 10:22:49.923 [FATClientORBConfig::shutServant] Done fini() on FAT Client
09/11/2012 10:22:49.923 [REMORBConfig::shutServant] Calling fini() on REM
09/11/2012 10:22:49.923 [REMORBConfig::shutServant] Done fini() on REM
09/11/2012 10:22:49.923 [DSMFSMORBConfig::shutServants] Calling fini() on FSM
09/11/2012 10:22:49.923 [DSMFSMORBConfig::shutServants] Done fini() on FSM
09/11/2012 10:22:49.923 [DSMFSMORBConfig::shutServants] Calling fini() on DSM
09/11/2012 10:22:49.923 [DSMFSMORBConfig::shutServants] Done fini() on DSM
09/11/2012 10:22:49.923 [EMMServer::fini] EMM Server shutting down
09/11/2012 10:22:49.923 [Info] V-111-1003 [EMMServer::fini] EMMServer shutting down
09/11/2012 10:22:49.995 [Error] V-111-1049 EMMServer generic error = Exception caught for IORTable clean up <NotFound>
09/11/2012 10:22:49.995 [EMMServer::fini] EMM Server shut down complete
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Caught the following <BAD_INV_ORDER> CORBA exception
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.007 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.008 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
09/11/2012 10:22:50.009 [EMMMain] <EXIT>
09/11/2012 10:22:50.009 [Log::deinit]  
09/11/2012 10:22:50.009 [EMMServer::~EMMServer]  

 

 

 

Mark_Solutions's picture

It sounds like there is something wrong with SQL Anywhere daemon or your server.conf file - see this tech note which describes it :

http://www.symantec.com/docs/HOWTO33625

As the EMM database starts to come up it doesn't seem to understand anything in that file so it is either that or it may be SQL Anywhere itself.

It is the same sirt of error you can get after an upgrade - take a look through this thread:

https://www-secure.symantec.com/connect/forums/nbu-update-7101-failed-initialize-emm-connection-195

To summarise ... the above was a schema mismatch after a patch .. i dont see that in your EMM log but maybe it doesn't go back far enough?

So first run:

./nbdbms_start_server located in /usr/openv/db/bin.

You may be able to see in clear text why EMM doesn't start as intended.

Have you patched or anything prior to this error? Has anything changed? Have you plenty of disk space?

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Marianne's picture

Seems you did NOT reboot as requested...
pbx_exchange has been up and running since 20 March. Other processes since 5 and 6 Sept....

You have not told us if you have checked/verified disk space on the master?

The errors in emm log looks like possible corruption of one of more config files. Please check size and timestamp of all of the following files - none of them should be 0 bytes:

/usr/openv/var/global/server.conf
/usr/openv/var/global/databases.conf
/usr/openv/db/data/vxdbms.conf

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

mph999's picture

OK, this should not be taking this long to resolve.

Following on from Mariannes post, if the disk did fill up it is very likely that the transaction log has been toasted ...  There is no way to tell, so we just make a basic check, and if this fails, we rebuild it.

 

Do exactly these steps.

Stop NBU

Reboot server

Is NBU now working ... if yes, then the issue is fixed.

If No, do this :

stop NBU and then run

/usr/openv/db/bin/nbdb_ping

You should see a message saying :

Database [NBDB] is alive and well on server [NB_womble].   (Your server name will be different)

If you see a message saying the database is unavailable then you will need to recreate the transaction log to see if that resolves the issue.  To rebuild the transaction log, follow these steps exactly.

 

/usr/openv/netbackup/bin/goodies/netbackup stop

/usr/openv/db/bin/nbdb_admin -auto_start NONE

. /usr/openv/db/vxdbms_env.sh

cd /usr/openv/db/data

mv NBDB.log NBDB.log.bad

/usr/openv/db/bin/dbeng11 -f NBDB

/usr/openv/db/bin/nbdb_admin -auto_start NBDB

( /usr/openv/db/bin/nbdb_admin -auto_start BMRDB  -  If you use BMR)

/usr/openv/db/bin/nbdbms_start_server

/usr/openv/db/bin/nbdb_ping

If it now shows the DB as alive and well, hopefully issue is fixed and you can restart NBU

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
SOLUTION
girish_nair1984's picture

Dear friends

 

I have stop NBU and restarted the server but.

At this point during startup it is stuck showing message

Netbackup SAN Client fibre transport daemon started

It's been 45 mins till now, I am getting this message.

 

Please Help

girish_nair1984's picture

I am not able to log into my Master Server now. I am using Red Hat Linux

Marianne's picture

We have asked this about 3 or 4 times in the last couple of days:

HAVE YOU CHECKED DISK SPACE?

If you cannot login to the system, speak to your Linux support team. NBU nit coming up will not cause this. Sounds like there is something else wron on this server?

Have you tried remote login (using something like putty) ?
Can you ping the server from your desktop or other machines?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

girish_nair1984's picture

Yes, there is space.

I am able to ping onto this server and remote login(putty).

Marianne's picture

Please post output of: 

bpps -x

df -h

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

girish_nair1984's picture

After performing the steps mentioned by mph999  it now shows me that DB as alive.

 

Output of bpps -x

NB Processes
------------
root      4563     1  0 11:55 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv @/us         r/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root      4644     1  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/vnetd -         standalone
root      4647     1  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/bpcd -s         tandalone
root      4695     1  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/nbevtmg         r
root      4731     1  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/nbaudit
root      4893     1  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/nbemm
root      4925     1  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/nbrb
root      4998     1  0 11:56 pts/6    00:00:00 /usr/openv/netbackup/bin/bprd
root      5004     1  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/bpcompa         td
root      5007     1  0 11:56 pts/6    00:00:00 /usr/openv/netbackup/bin/bpdbm
root      5011     1  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/nbjm
root      5013  5007  0 11:56 pts/6    00:00:00 /usr/openv/netbackup/bin/bpjobd
root      5028  5011  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy          dblib nbjm
root      5035     1  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/nbpem
root      5059  5035  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/nbproxy          dblib nbpem
root      5066     1  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/nbstser         v
root      5100     1  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/nbrmms
root      5306     1  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/nbsl
root      5538     1  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/nbars
root      5735     1  0 11:56 ?        00:00:00 /usr/openv/netbackup/bin/nbsvcmo         n
root      6702     1  0 Sep13 ?        00:00:00 /usr/openv/db//bin/NB_dbsrv @/us         r/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn 7
root      6704  6702  0 Sep13 ?        00:00:00 [NB_dbsrv] <defunct>

MM Processes
------------
root      4972     1  0 11:56 pts/6    00:00:00 /usr/openv/volmgr/bin/ltid
root      5747     1  0 11:56 pts/6    00:00:00 vmd
root      5834  4972  0 11:56 pts/6    00:00:00 avrd
root      5837     1  0 11:56 pts/6    00:00:00 tldcd

Shared Symantec Processes
-------------------------
root      3884     1  0 Sep13 ?        00:00:01 /opt/VRTSpbx/bin/pbx_exchange
 

 

Output of df-h

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol01
                      7.6G  6.4G  855M  89% /
/dev/mapper/VolGroup00-LogVol07
                      4.8G  578M  4.0G  13% /or_export
/dev/mapper/VolGroup00-LogVol03
                      4.8G  639M  3.9G  14% /var
/dev/mapper/VolGroup00-LogVol05
                      1.9G  453M  1.4G  25% /home
/dev/mapper/VolGroup00-LogVol04
                      961M   18M  894M   2% /tmp
/dev/mapper/VolGroup00-LogVol06
                      7.6G  5.3G  2.0G  73% /opt/or
/dev/sda1              97M   19M   73M  21% /boot
tmpfs                 2.0G     0  2.0G   0% /dev/shm
/dev/mapper/VolGroup00-LogVol02
                       20G  3.8G   15G  21% /opt/xyz
/dev/mapper/VolGroup00-LogVol08
                      823G  635G  163G  80% /or_data

 

While doing robot inventory I am getting this error

 

unable to find robotic path 201

Marianne's picture

We don't know where NBU is installed, but if it is in / (where /usr/openv is in a default installation), you can see that that you only have 855M free - very little space if you have a busy master server. 

Glad to see that you EMM database is up and running now.

I believe that Martin (mph999) has helped to solve your EMM issue. Please mark his post as solution.

The missing robot is a different issue. You need to start a new discussion for that.

When you start a new thread, please confirm that OS can see all devices - provide output of:

/usr/openv/volmgr/bin/scan
cat /proc/scsi/scsi

As well as NBU device config:
/usr/openv/volmgr/bin/tpconfig -l

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

girish_nair1984's picture

Thank You friends for helping me to solve this error.