A restart of the Enterprise Vault Storage Service may cause the StorageOnlineOpns.exe process to return STORAGE_E_SERVICE_NOT_RUNNING errors.

Article:TECH212999  |  Created: 2013-12-03  |  Updated: 2014-06-16  |  Article URL http://www.symantec.com/docs/TECH212999
Article Type
Technical Solution


Issue



On Enterprise Vault (EV) servers, it is possible for a manual restart of the EV Storage Service to cause the StoreageOnlineOpns.exe process to return ‘STORAGE_E_SERVICE_NOT_RUNNING’ errors.  Event IDs 6654, 6655, and 6656 will be in the Symantec EV event logs, indicating problems with the EV Storage Service.  Retrieval of archived items using the Windows EV Outlook Add-In may return the error "GetOnlineAttachmentfilesize 0xC0041AE9", which will continue to occur until the EV server is rebooted.  The Application Event log may include process crash errors, Event ID 1000 referencing the StoreageOnlineOpns process.

A Dtrace of the StorageOnlineOpns process on the affected EV server will contain repeated lines containing the string:  "The Storage Service is not running      (0xc0041ae9)"


Error



Events logged in Symantec EV event log during a restart of the EV Storage Service:

Type: Error
Event: 6656
Source: Enterprise Vault 
Category: Storage Server
Description:
Error starting process
Reason: The Storage Service is not running      (0xc0041ae9)
Process: Storage Online Process (1)
Reference: RunTheProcess/CCI
 
Type: Warning
Event: 6654
Source: Enterprise Vault 
Category: Storage Server
Description:
Storage Online Process (1)  -  Restarting failed process
PID: 0
Exit Code: Unspecified error  (0x80004005)
The process has failed abnormally and will be restarted automatically
 
Type: Error
Event: 6655
Source: Enterprise Vault 
Category: Storage Server
Description:
Storage Online Process (1)  -  Process failed
PID: 0
Exit Code: Unspecified error  (0x80004005)
The process has failed abnormally and will not be restarted. Either the process has exceeded 
its restart count or automatic restart is disabled. The process can be restarted by stopping and
restarting the Storage Service
 
Events logged in the Application log on the EV server when the StorageOnlineOpns process crashes:
 
Type: Error
Event: 1000
Source: Application Error
Category: Application Crashing Events
Description:
Faulting application name: StorageOnlineOpns.exe, version: 10.0.2.1112, time stamp: <time_stamp>
Faulting module name: RPCRT4.dll, version: 6.1.7601.18205, time stamp: <time_stamp>
Exception code: 0xc00000fd
Fault offset: 0x00033f05
 
Faulting module name may vary between
RPCRT4.dll, ole32.dll, ntdll.dll, KERNELBASE.dll
 
Errors that may occur when using the Windows EV Outlook Add-In when retreieving items from the EV Server:
GetOnlineAttachmentfilesize 0xC0041AE9

Environment



Windows Server 2008 R2
EV 10.0.1/10.0.2


Cause



The StorageOnlineOpns (SOPNs) process is used by the Storage Service and other processes to fetch information about/from Vault Stores.  Remote Procedure Calls (RPC) from other EV processes can start an instance of the SOPNs process when the Storage Service is in a stopped or starting state.  When called, the SOPNs process checks the status of the Storage Service.  If the Storage Service is not in an online state the SOPNs process receives a return value of 0xC0041AE9 or STORAGE_E_SERVICE_NOT_RUNNING.  Calls to the SOPNs process can originate from other EV servers.

The issue reveals itself when the following two conditions occur. 
1. The SOPNs process has been invoked from another EV process and has received the Storage Service status of STORAGE_E_SERVICE_NOT_RUNNING, but has not yet exited.  
2. At he same time the Storage Service is in the "Starting" state creating it's own instances of the SOPNs process.  
 
When the two operations are happening at the same time, all instances of the SOPNs process receive the status value of STORAGE_E_SERVICE_NOT_RUNNING.  This value is then returned for all subsequent instances of the SOPNs process until the EV server is rebooted.

Solution



The Issue has been addressed in EV Versions 10.0.3/10.0.4 and above

Work around to Mitigate the frequency of the issue:
 
1. In large EV environments where Archives are located on different EV servers than the Archive Tasks:
a. Do not manually restart the Storage Service without scheduling a full EV server reboot.
 
2. In large EV environments where Archives are located on the same EV server as the Archive Tasks:
a. Stop all EV services on the affected EV server
b. Stop IIS as follows:
- Open a Command Prompt and type: iisreset /stop
 
Note: This will stop both IIS Admin Service and World Wide Web Publishing Service
c. Start the services in the following order
Enterprise Vault Admin Service
Enterprise Vault Directory Service
Enterprise Vault Storage Service
Start IIS as follows:
- Open a Command Prompt and type: iisreset /start
 
Note: This will start both IIS Admin Service and World Wide Web Publishing Service
Enterprise Vault Shopping Service
Enterprise Vault Indexing Service
Enterprise Vault Task Controller
 

Supplemental Materials

SourceUMI
ValueV-437-6654
Description

 6654 - The process has failed abnormally and will be restarted automatically


SourceUMI
ValueV-437-6655
Description

6655 - The process has failed abnormally and will not be restarted


SourceUMI
ValueV-437-6656
Description

 6656 - Error starting process


SourceETrack
Value3350043
Description

SOPNs crashes and restarts during Storage Service restart with Event ID's 6654,6656,1000



Article URL http://www.symantec.com/docs/TECH212999


Terms of use for this information are found in Legal Notices