Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Strange PBX behavior

Created: 06 Aug 2009 • Updated: 21 May 2010 | 7 comments
Android's picture

OK here goes:
Product Version: NetBackup 6.5.4
Server OS: Windows 2003 Server

About once a day, NetBackup services will hang and cannot be restarted.  The Windows server nust be rebooted in order to get the application running again.  There is some pretty strange behavior associated with this.  For starters you can find the following in the logs:

bpbdm log:
23:29:14.052 [8076.5468] <2> process_request: request complete: exit status 0  ; query type: 78
23:29:56.224 [3024.3028] <2> software_expiration: Expired at Sun Aug 02 23:29:56 2009

23:29:57.396 [3024.3028] <2> mlic_get_nb_features: D:\654\src\licensing\libminlicense\minlicfile.c.401: license.txt reread = <Wed Dec 31 18:00:00 1969
>
23:29:57.396 [3024.3028] <2> software_expiration: New expiration at Mon Jan 18 21:14:07 2038

23:30:02.396 [3024.3028] <2> mlic_get_nb_features: D:\654\src\licensing\libminlicense\minlicfile.c.352: status = <Stat failed>
23:30:02.396 [3024.3028] <2> get_expiration: D:\654\src\licensing\libminlicense\minlicfile.c.513: status = <Stat failed>
23:30:02.396 [3024.3028] <2> software_expiration: Expired at Sun Aug 02 23:30:02 2009

23:30:03.396 [3024.3028] <2> mlic_get_nb_features: D:\654\src\licensing\libminlicense\minlicfile.c.394: status = <No base feature>
23:30:03.396 [3024.3028] <2> get_expiration: D:\654\src\licensing\libminlicense\minlicfile.c.513: status = <No base feature>
23:30:03.396 [3024.3028] <2> db_error_add_to_file: write(D:\Program Files\Veritas\NetBackup\db\error\log_1249275600) failed: Invalid argument (22)
23:30:03.396 [3024.3028] <32> bpdbm: Evaluation software has expired.  See www.veritas.com for ordering information
23:30:03.396 [3024.3028] <2> mlic_get_nb_features: D:\654\src\licensing\libminlicense\minlicfile.c.394: status = <No base feature>
23:30:03.396 [3024.3028] <2> LOCAL CLASS_ATT_DEFS: Unable to read keyfile <D:\Program Files\Veritas\NetBackup\var\license.txt>

23:30:03.396 [3024.3028] <2> get_behavior_rec: Bad MD5 digest for file D:\Program Files\Veritas\NetBackup\db\config\behavior
23:30:03.396 [3024.3028] <2> mlic_get_nb_features: D:\654\src\licensing\libminlicense\minlicfile.c.352: status = <Stat failed>
23:30:03.396 [3024.3028] <2> LOCAL CLASS_ATT_DEFS: Unable to read keyfile <D:\Program Files\Veritas\NetBackup\var\license.txt>

23:30:03.396 [3024.3028] <2> LOCAL CLASS_ATT_DEFS: Unable to open <D:\Program Files\Veritas\NetBackup\share\Class_att_defs.txt>
23:30:03.396 [3024.3028] <2> behavior_write: fopen() D:\Program Files\Veritas\NetBackup\db\config\behavior.3024: Invalid argument (22)
23:30:03.396 [3024.3028] <2> get_behavior: behavior_write failed: system call failed (11)
23:30:03.396 [3024.3028] <2> sendViaScript: attempting to send mail
23:30:03.396 [3024.3028] <2> makeMessageFile: Failed to open temporary file C:\WINDOWS\TEMP\nb_1CC.tmp.
23:30:03.396 [3024.3028] <2> db_error_add_to_file: open(D:\Program Files\Veritas\NetBackup\db\error\log_1249275600) failed: Invalid argument (22)
23:30:03.396 [3024.3028] <32> bpdbm: Unable to send software expiration email notice to administrator
23:30:03.396 [3024.3028] <2> vnet_bind_to_port_addr_extra: vnet_bind.c.112: Function failed: 10 0x0000000a
23:30:03.396 [3024.3028] <2> vnet_async_connect: vnet_vnetd.c.3904: vnet_bind_to_port_addr failed: 10 0x0000000a
23:30:03.396 [3024.3028] <2> vnet_bind_to_port_addr_extra: vnet_bind.c.112: Function failed: 10 0x0000000a
23:30:03.396 [3024.3028] <2> vnet_async_connect: vnet_vnetd.c.3904: vnet_bind_to_port_addr failed: 10 0x0000000a
23:30:03.396 [3024.3028] <2> vnet_connect_to_service_or_vnetd: vnet_vnetd.c.3565: vnet_async_connect failed: 10 0x0000000a
23:30:03.396 [3024.3028] <2> nb_connect_to_vnetd_or_legacy: comm.c.2030: vnet_connect_to_vnetd_or_service failed: 10

I checked the license.txt file first and all licenses are valid with no expiration.  I also checked to be sure there are no connectivity issues with the D:\ drive and there appear to be none.  The files above also do not appear to be corrupt in anyway.

nbemm:
8/4/2009 23:46:56.244 [MachineImpl::QueryMachinesWithFilter] retval - <0>
8/4/2009 23:46:57.306 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBREM>
8/4/2009 23:46:57.306 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBREM>
8/4/2009 23:46:57.306 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBREM>
8/4/2009 23:46:57.306 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBREM>
8/4/2009 23:46:57.588 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
8/4/2009 23:46:57.603 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
8/4/2009 23:46:57.603 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
8/4/2009 23:46:57.603 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
8/4/2009 23:46:57.603 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
8/4/2009 23:46:57.603 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
8/4/2009 23:46:57.619 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
8/4/2009 23:46:57.619 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
8/4/2009 23:46:57.619 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
8/4/2009 23:46:57.650 [EMMTaskBase::run_event_loop] Finished run() in this thread for <EMM>
8/4/2009 23:46:57.822 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
8/4/2009 23:46:57.822 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
8/4/2009 23:46:57.822 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
8/4/2009 23:46:57.822 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
8/4/2009 23:46:57.822 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
8/4/2009 23:46:57.822 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
8/4/2009 23:46:57.822 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
8/4/2009 23:46:57.838 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
8/4/2009 23:46:57.838 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBDSMFSM>
8/4/2009 23:46:59.853 [NBEmmSvc::shutdown_i] Calling shutdown on hosted services
8/4/2009 23:46:59.853 [FATClientORBConfig::shutServant] Calling fini() on FAT Client
8/4/2009 23:46:59.853 [FATClientORBConfig::shutServant] Done fini() on FAT Client
8/4/2009 23:47:00.150 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
8/4/2009 23:47:00.150 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
8/4/2009 23:47:00.150 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
8/4/2009 23:47:00.150 [FATClientORBConfig::shutdown] Waiting for threads to join
8/4/2009 23:47:00.181 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
8/4/2009 23:47:00.197 [EMMTaskBase::run_event_loop] Finished run() in this thread for <NBFSMCLIENT>
8/4/2009 23:47:00.197 [FATClientORBConfig::shutdown] Threads have joined
8/4/2009 23:47:00.197 [REMORBConfig::shutServant] Calling fini() on REM
8/4/2009 23:47:00.431 [REMORBConfig::shutServant] Done fini() on REM
8/4/2009 23:47:00.463 [REMORBConfig::shutdown] Waiting for threads to join
8/4/2009 23:51:20.125 [CSettings::readEmmConfig] EMM.conf configuration file does not exist.
8/4/2009 23:51:20.171 [EMMServer::EMMServer] 
8/4/2009 23:51:20.218 [DSMFSMORBConfig::DSMFSMORBConfig] Setting to default DSM ORB thread count <12>
8/4/2009 23:51:20.296 [EMMMain] <ENTER>
8/4/2009 23:51:20.296 [NBEmmSvc::initialize] Calling controlledRun() on service
8/4/2009 23:51:20.296 [EMMServer::isLocalHostEMMServer] 
8/4/2009 23:51:20.312 [EMMServer::isLocalHostEMMServer] LocalHost = <d3bckp1>, EmmServer = <d3bckp1>
8/4/2009 23:51:20.312 [NBEmmSvc::doShouldStart] Should start
8/4/2009 23:51:20.406 [NBEmmSvc::doInit] <ENTER>
8/4/2009 23:51:20.421 [NBEmmSvc::doInit] NumberOrbThreads = <10>, NumberDbConnections = <4>, NumberDbBrowseConnections = <3>
8/4/2009 23:51:20.484 [EMMServer::SetOrb] Orb is set
8/4/2009 23:51:20.531 [NBEmmSvc::doInit] Extra orb options:  -ORBCollocation no
8/4/2009 23:51:20.531 [EMMServer::init] Server initializing VxEMM
8/4/2009 23:51:20.531 [Info] V-111-1000 [EMMServer::init] EMMServer initializing = VxEMM

23:30:03.396 [3024.3028] <2> job_connect: Can't connect automatically to client localhost status = 21 err = 10055
23:30:03.396 [3024.3028] <2> job_connect: Can't connect to client localhost
23:32:05.038 [5732.3196] <4> bpdbm: INITIATING bpdbm: NetBackup 6.5 0 on d3bckp1 IDIRSTRUCT=2 (VERBOSE = 0)
23:32:05.038 [5732.3196] <2> clean_locks: D:\Program Files\Veritas\NetBackup\db\class_locks is empty or not accessible, No such file or directory (0)
23:32:05.038 [5732.3196] <2> getsockbound: service=bpdbm protocol=tcp port=13721
23:32:05.679 [6668.2492] <2> logconnections: BPDBM ACCEPT FROM 10.67.50.72.3808 TO 10.67.50.72.13724
23:32:05.695 [6668.2492] <2> retention_db: Q_RETENTION_GET
23:32:05.726 [6668.2492] <2> process_request: request complete: exit status 0  ; query type: 55
23:34:54.040 [4508.7432] <2> logconnections: BPDBM ACCEPT FROM 10.67.50.72.4216 TO 10.67.50.72.13724
23:34:54.056 [4508.7432] <2> image_db: Q_IMAGE_BY_BACKUPID
23:34:54.072 [4508.7432] <2> db_get_image_info: Job in progress, found image file D:\Program Files\Veritas\NetBackup\db\images

nbrb:
8/4/2009 23:49:06.167 [StopScanCallbackQueue::evaluateQueue] evaluating 0 allocations to StopScan completed
8/4/2009 23:49:06.167 [StopScanCallbackQueue::evaluateQueue] StopScan evaluation complete. 0 allocations remain, 0 complete, 0 failed
8/4/2009 23:49:06.183 [EMMAccess::getRBCacheDriveList] cannot contact EMM server: message=CORBA exception: TRANSIENT (IDL:omg.org/CORBA/TRANSIENT:1.0) -- system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as '*unknown description*', completed = NO
, file=EMMAccess.cpp, line=1017
8/4/2009 23:49:06.183 V-118-71 [EMMAccess::getRBCacheDriveList] Exception caught while connecting to EMM server: system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as '*unknown description*', completed = NO

8/4/2009 23:49:06.183 V-118-87 [EMMAccess::getRBCacheDriveList] failed to get drive list for RB cache
8/4/2009 23:49:06.183 [Warning] V-118-87 failed to get drive list for RB cache
8/4/2009 23:49:06.183 [isRetryableMDS] MDS returned a retryable error 3000000
8/4/2009 23:49:06.183 [ResBroker_i::evaluateLater] scheduling evaluation to start in 60 seconds
8/4/2009 23:49:07.167 [Warning] V-118-52 Resource Broker optimizations will be partially disabled.
8/4/2009 23:49:07.183 [EMMProvider::touchFileExists] file D:\Program Files\Veritas\NetBackup\db\config\RB_UNLOAD_DUP_WRITER_ON_DEMAND does not exist
8/4/2009 23:49:07.183 [EMMProvider::touchFileExists] file D:\Program Files\Veritas\NetBackup\db\config\RB_UNLOAD_CATALOG_ON_DEMAND does not exist
8/4/2009 23:49:07.183 [EMMProvider::touchFileExists] file D:\Program Files\Veritas\NetBackup\db\config\RB_DISABLE_REAL_UNLOADS_ON_DEMAND does not exist
8/4/2009 23:49:07.183 [EMMProvider::touchFileExists] file D:\Program Files\Veritas\NetBackup\db\config\RB_DISABLE_FAKE_UNLOADS_ON_DEMAND does not exist
8/4/2009 23:49:07.183 [EMMProvider::touchFileExists] file D:\Program Files\Veritas\NetBackup\db\config\RB_RESPECT_UNLOADS_ORDER does not exist
8/4/2009 23:49:07.183 [EMMProvider::touchFileExists] file D:\Program Files\Veritas\NetBackup\db\config\RB_DO_NOT_USE_SHARED_RESERVATIONS does not exist
8/4/2009 23:49:07.183 [EMMProvider::touchFileExists] file D:\Program Files\Veritas\NetBackup\db\config\RB_ITC_DISABLE_DROPOUT_PHASE does not exist
8/4/2009 23:49:07.183 [EMMAccess::fetchEmmServerName] using EMM server XXXXXXXX
8/4/2009 23:49:09.433 [EMMAccess::initialize] CORBA exception: TRANSIENT (IDL:omg.org/CORBA/TRANSIENT:1.0) -- system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as '*unknown description*', completed = NO

8/4/2009 23:49:09.433 [Warning] V-118-5 failed to connect to EMM Server
8/4/2009 23:49:09.433 V-118-127 [EMMAccess::QueryMachine] failed to resolve Machine reference
8/4/2009 23:49:09.433 V-118-82 [EMMAccess::QueryMachine] query of machine information for machine d3bckp1, type 6 returned 3000000
8/4/2009 23:49:09.433 [Warning] V-118-74 failed to obtain information for EMM Server XXXXXXXXX
8/4/2009 23:49:09.433 [ProviderManager::evaluateSanClientLicense] SAN Client license is not installed
8/4/2009 23:49:09.433 V-118-105 [ResBroker_i::doEvaluation] Resource evaluation started

The SAN client license is installed.  I've triple checked.  It's valid, with no expiration.

mds:
8/4/2009 23:45:56.462 [setStuLastUsedTime] spr_tried_key_list is empty
8/4/2009 23:45:56.462 [BEGIN_MDS_DB_TRANSACTION] attempting to get database connection
8/4/2009 23:45:56.462 [BEGIN_MDS_DB_TRANSACTION] got database connection
8/4/2009 23:45:56.462 [BEGIN_MDS_DB_TRANSACTION] attempting to start database transaction
8/4/2009 23:45:56.462 [BEGIN_MDS_DB_TRANSACTION] started database transaction
8/4/2009 23:45:56.462 [sql_get_alloc_info] alloc_key = 14517
8/4/2009 23:46:57.603 [handleMessage] INITIATING:
8/4/2009 23:46:57.603 [handleMessage] got message: DEVICE_STATUS 1 14511 d3bckp1 TX0157 4000745 HP.ULTRIUM3-SCSI.002 2000051 WRITE_ERROR 0 0
8/4/2009 23:46:57.603 [BEGIN_MDS_DB_TRANSACTION] attempting to get database connection
8/4/2009 23:46:57.603 [BEGIN_MDS_DB_TRANSACTION] got database connection
8/4/2009 23:46:57.603 [BEGIN_MDS_DB_TRANSACTION] attempting to start database transaction
8/4/2009 23:46:57.603 [BEGIN_MDS_DB_TRANSACTION] started database transaction
8/4/2009 23:46:57.634 [sql_get_alloc_info] alloc_key = 14511
8/4/2009 23:46:57.634 [sql_get_alloc_info] alloc - alloc_t: key = 14511, job_type = 1, capability_flags = 128, master_server_name = d3bckp1, master_server_key = 1000002, media_server_name = d3bckp1, media_server_key = 1000002, ndmp_ts_name = , ndmp_ts_key = 0, stu_name = d3bckp1, media_robot_num = 0, media_robot_type = 8, drive_robot_num = 0, drive_robot_type = 8, media_key = 4000745, drive_key = 2000051, drive_path = {2,0,2,0}, disk_volume_key = 0, disk_mount_point_key = 0, fat_pipe_key = 0, scsi_res_type = 1, server_state_flags = 1
8/4/2009 23:46:57.634 [sql_update_add_alloc_status_rec] alloc_status_t: key = 14511, med = 4000745, drv = 2000051, code = 8194, op = 0, host = 1000002, data1 = 0, data2 = 0, str1 =
8/4/2009 23:46:57.634 [sql_update_add_alloc_status_rec] added status rec
8/4/2009 23:46:57.634 [handle_device_status_msg] added error, WRITE_ERROR
8/4/2009 23:46:57.634 [sql_get_drive_info] drive_key = 2000051, want_active = 1
8/4/2009 23:46:57.650 [sql_get_drive_info] drv - drive_t: name = HP.ULTRIUM3-SCSI.002, key = 2000051, path = {2,0,2,0}, alloc_state = 1, device_flags = 32779, path_flags = 4096, path_status = 2, active = 1, host_key = 1000002, ndmp_host_key = 0, alloc_host_key = 1000002, scan_host_key = 0, robot_num = 0, robot_type = 8, type_name = NetBackup HCART3, pend_actions = 0, tape_alert2 = 0
8/4/2009 23:46:57.650 [sql_update_set_pending_actions] drive_key = 2000051, act = 1
8/4/2009 23:46:57.650 [sql_update_set_pending_actions] set pending actions
8/4/2009 23:46:57.650 [END_MDS_DB_TRANSACTION] cur_err = 0, m_dbconn_stat = 0, m_dberr_stat = 0, m_closed_db_trx = 0
8/4/2009 23:46:57.650 [END_MDS_DB_TRANSACTION] committed database transaction
8/4/2009 23:46:57.650 [END_MDS_DB_TRANSACTION] cur_err = 0, m_dbconn_stat = 0, m_dberr_stat = 0, m_closed_db_trx = 1
8/4/2009 23:46:57.650 [END_MDS_DB_TRANSACTION] database transaction has already been closed
8/4/2009 23:46:57.650 [handleMessage] EXIT INFO:
8/4/2009 23:46:57.650 [handleMessage] EXIT STATUS = 0 (EMM_ERROR_Success, Success)

8/4/2009 23:51:26.437 [getAllocationList] INITIATING:
8/4/2009 23:51:26.437 [getAllocationList] masterServer =

da:
8/4/2009 23:45:09.024 [DA_Thread_Pool::IsDiskFull] File system for < D:\Program Files\Veritas\\NetBackupDB\data > is 27.525 percent available.
8/4/2009 23:45:18.164 [DA_Thread_Pool::CheckIfVmscdIsNeeded] 
8/4/2009 23:45:18.164 [DA_Thread_Pool::CheckForHeartbeat] Check heartbeats against current time 1249447518
8/4/2009 23:45:44.040 [DeviceAllocatorImpl::helperDriveDeallocated()] DriveKey < 2000049 >
8/4/2009 23:45:44.040 [DeviceAllocatorImpl::helperDriveDeallocated()] Drive < HP.ULTRIUM3-SCSI.000 >, has been deallocated
8/4/2009 23:45:48.165 [DA_Thread_Pool::CheckIfVmscdIsNeeded] 
8/4/2009 23:45:48.165 [DA_Thread_Pool::CheckForHeartbeat] Check heartbeats against current time 1249447548
8/4/2009 23:45:48.602 [DeviceAllocatorImpl::updateMachineState]  - MachineName = < d3bckp1 >, NetBackupVersion = < 0 >, MachineStateOpCode = < 3 >
8/4/2009 23:45:48.602 [DeviceAllocatorImpl::updateMachineState] Found machine with current state < 14 >
8/4/2009 23:45:48.602 [DA_Thread_Pool::SetLastHeartBeat] Hostname < d3bckp1 >, Interval < 300 >
8/4/2009 23:45:48.602 [DA_Thread_Pool::SetLastHeartBeat] Set host d3bckp1 in heartbeat map with last heartbeat time 1249447548, and heartbeat interval is 300
8/4/2009 23:45:56.180 [DeviceAllocatorImpl::helperDriveDeallocated()] DriveKey < 2000058 >
8/4/2009 23:45:56.180 [DeviceAllocatorImpl::helperDriveDeallocated()] Drive < HP.ULTRIUM3-SCSI.005 >, has been deallocated
8/4/2009 23:46:18.165 [DA_Thread_Pool::CheckIfVmscdIsNeeded] 
8/4/2009 23:46:18.165 [DA_Thread_Pool::CheckForHeartbeat] Check heartbeats against current time 1249447578
8/4/2009 23:46:48.166 [DA_Thread_Pool::CheckIfVmscdIsNeeded] 
8/4/2009 23:46:48.166 [DA_Thread_Pool::CheckForHeartbeat] Check heartbeats against current time 1249447608
8/4/2009 23:51:23.421 [DA_Thread_Pool::DA_Thread_Pool] poolSize < 6 >, Hz < 30 >, orb < 12ED80 >
8/4/2009 23:51:23.421 [DA_Thread_Pool::open] Activating threads, PoolSize < 6 >
8/4/2009 23:51:23.421 [DA_Thread_Pool::ResetScanForAllocatedDrives] 
8/4/2009 23:51:23.500 [DA_Thread_Pool::svc] DA thread < 1 > is active
8/4/2009 23:51:23.500 [DA_Thread_Pool::CommandDequeue] 
8/4/2009 23:51:23.500 [DA_Thread_Pool::svc] DA thread < 2 > is active
8/4/2009 23:51:23.500 [DA_Thread_Pool::svc] DA thread < 3 > is active
8/4/2009 23:51:23.500 [DA_Thread_Pool::CommandDequeue] 
8/4/2009 23:51:23.500 [DA_Thread_Pool::svc] DA heartbeat thread is active, Pooling frequency = < 30 >
8/4/2009 23:51:23.500 [DA_Thread_Pool::svc] DA disk check thread is active, Pooling frequency = < 300 > FREE_DISK (1.000 % 0 MB)

OK that's NetBackup.  From the system point of view, I see the following in the system event log:
-The Symantec Private Branch Exchange service terminated with service-specific error 2 (0x2).

-The NetBackup Enterprise Media Manager service terminated unexpectedly.  It has done this 1 time(s).

-The server was unable to allocate from the system nonpaged pool because the pool was empty.

-The NetBackup Service Layer service depends on the Symantec Private Branch Exchange service which failed to start because of the following error:
  The service has returned a service-specific error code.

-The Symantec Private Branch Exchange service terminated with service-specific error 22 (0x16).

-The description for Event ID ( 7039 ) in Source ( Service Control Manager ) cannot be found. The local computer may not have the necessary registry  information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: Symantec Private Branch Exchange, 6076, 3712.

-The NetBackup Enterprise Media Manager service depends on the Symantec Private Branch Exchange service which failed to start because of the following error:  The service has returned a service-specific error code.

-The description for Event ID ( 7039 ) in Source ( Service Control Manager ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: Symantec Private Branch Exchange, 4188, 5196.

This is a real head scratcher.  Any help would be appreciated.  Sorry to overload with log files.....

Comments 7 CommentsJump to latest comment

Amit Karia's picture

Additionally you can also check if your license is not expired

Android's picture

Amit,

As stated int he original post, all licenses are valid with no expiration, the licenses.txt file has been checked and does not appear to be corrupt in anyway, connectivity to the D:\ drive has been verified and there are no issues there.

There is no Storage Foundation software installed on this NetBackup master server. 

rj_nbu's picture

Do you have Mcafee anti virus on that server.

-Rajeev
"Unix IS user friendly... It's just selective about who its friends are"

Android's picture

/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:"";
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin:0in;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:"Times New Roman";
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}

Yesterday I was running some performance counters and bpbkar32 suddenly went nuts.  The handle count jumped up to 2,954,419, page faults was over 2 million, etc.  All the counters were going crazy, then NetBackup Event Manager service went down and suddenly everything went back to normal. 

Anyone with some insight? suggestions?

Android's picture

Yes McAfee is on the server.  We have already gone throught the process of putting the NBU services in the McAfee software following the instructions of the technote.

seer.entsupport.symantec.com/docs/295599.htm

However, since then NBU has been upgraded from 6.5.3 to 6.5.4.  Also Windows patches were applied.  The weird thing is the behavior started about 1.5 weeks after upgrading NBU, and about 1 week before applying the Windows patches. 

Should we go through this process again to be sure that the updates did not undo anything?

Android's picture

OK with no real replies lately I thought I'd give an update to see if any more light is shed.  The problem seemd to be narrowed down to two clients.  Both are virtual machines that were using VCB backup type 3.  Switching to type 1 seems to have eliminated the behavior. 

However, this is merely a work around.  We need to do full VM with file (type 3) for these clients and I cannot seem to figure out what is causing this.  Anyone have any ideas?