Video Screencast Help

Unable to start EV Indexing - configuration error - HR-Access is denied

Created: 02 Aug 2013 • Updated: 07 Aug 2013 | 16 comments
This issue has been solved. See solution.

I am having issues with an Enterprise Vault 10.0.4 server unable to start the "Enterpris Vault Indexing Service". After enabling a "Search and Indexing trace" when starting the service, I found that there is a problem with access, but I am not able to figure out where to set it.

The server was working at 10.0.2, but after the upgrade I am not able to start the indexing. There can also be other changes to the operation system causing this.

Here i a part from the trace:

[8524] (w3wp) <8728> EV:L WinHttpRequest::OnCallback. WINHTTP_CALLBACK_STATUS_HEADERS_AVAILABLE. Status code=401
[8524] (w3wp) <8728> EV:H WinHttpRequest::OnCallback. .  Is the service running?
[8524] (w3wp) <8728> EV:H WinHttpRequest::OnCallback exit. ERROR HR=0x80070005.
[8524] (w3wp) <8728> EV:M WinHttpRequest::Callback from WinHttp. pRequest->OnCallback failed with HR=0x80070005. Calling pRequest->OnResponseComplete.
[8524] (w3wp) <8728> EV:M IndexQueryServerRequest: OnResponseComplete - HR=Access is denied.  (0x80070005)
[8524] (w3wp) <8040> EV:L IndexQueryServerRequest: WaitForSingleObject returned 0.
[8524] (w3wp) <8040> EV:M Response received. Result = Access is denied.  (0x80070005), Status = 401
[8524] (w3wp) <8040> EV:H CIndexQueryServerSearch::Search2. Request FAILED HR=0x80070005.
[8524] (w3wp) <8040> EV:L WinHttpRequest::Shutdown entry. Closing Winhttp handle
[8524] (w3wp) <8040> EV:L WinHttpHandle: Closing handle.
[8524] (w3wp) <8040> EV:L WinHttpRequest::Callback from WinHttp. Code=0x800, context=0x1B48E410
[8524] (w3wp) <8040> EV:L WinHttpRequest::OnCallback. WINHTTP_CALLBACK_STATUS_HANDLE_CLOSING.
[8524] (w3wp) <8040> EV:L WinHttpRequest::Shutdown. Handle closed.  Waiting for callbacks to complete.
[8524] (w3wp) <8040> EV:L WinHttpRequest::WaitForCallbacksFromWinHttpToFinish entry. Max wait = 60000ms
[8524] (w3wp) <8040> EV:L WinHttpRequest::Shutdown exit.
[8524] (w3wp) <8040> EV:L ~WinHttpRequest entry.
[8524] (w3wp) <8040> EV:L ~WinHttpRequest exit.
[8524] (w3wp) <8040> EV:L WinHttpHandle: Closing handle.
[8524] (w3wp) <8040> EV:L WinHttpHandle: Closing handle.
[8524] (w3wp) <8040> EV:M CIndexSearch2::PerformSearch request failed. Attempt:1 ArchiveId:16B537344F3147A4D9AED65EDB6482D771110000fsvaultgjo|hr=Access is denied.  (0x80070005)
[8524] (w3wp) <8040> EV:H CIndexSearch2::PerformSearch Search failed with 'normal' error: hr=Access is denied.  (0x80070005).
[8524] (w3wp) <8040> EV:L CIndexSearch2::FinalRelease
[8524] (w3wp) <8040> EV:L CAuthHelper::Reset Cancel registration? True CancelId: 12077152
[8524] (w3wp) <8040> EV-H {Common.DTraceExceptionAndVEID} Exception: Access is denied.  (0x80070005) Info:VEID:16B537344F3147A4D9AED65EDB6482D771110000fsvaultgjo Diag: Type:System.UnauthorizedAccessException ST:   at KVS.EnterpriseVault.Interop.IndexSearch2Class.Search(String bsQuery, Int32 startResult, Int32 maximumResults, String reserved)|   at DesktopClientCacheWeb.SyncPoint.SimpleIndexQuery(String archiveVEID, Int32 snumToQuery)|   at DesktopClientCacheWeb.SyncPoint.GetSyncPoint(String syncSlot, String archiveVEID, Int32 lastSyncPoint)|   at DesktopClientCacheWeb.SyncPoint.Page_Load(Object sender, EventArgs args) Inner:None
[8524] (w3wp) <1804> EV-L {Slot.Page_Load} Slot.Page_Load
[8524] (w3wp) <1804> EV-L {Slot.ReleaseSyncSlot} Slot.ReleaseSyncSlot - SlotID: 48d8ff9c-6b05-4f0a-b334-32fea7356b20, VEID:16B537344F3147A4D9AED65EDB6482D771110000fsvaultgjo
[8524] (w3wp) <1804> EV:L {CAutoJournalAccessor::ReleaseSyncSlot} (Entry)
[8524] (w3wp) <1804> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
[8524] (w3wp) <1804> EV:L CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
[8524] (w3wp) <1804> EV:L {VaultCoCreateInstanceEx} CLSID [{4EC6FF76-C97A-11D1-90E0-0000F879BE6A}] Server Name [(null)] Used Server Name [(null)] Num of attempts [1] Total elapsed [0.000s] Result [Success  (0)]
[8524] (w3wp) <1804> EV:M CAutoStorageOnline::CreateObject - Calling VaultCoCreateInstanceEx
[8524] (w3wp) <1804> EV:L {VaultCoCreateInstanceEx} CLSID [{957FF4B4-162B-4708-843A-0134868699B4}] Server Name [fsvaultgjo.company.loc] Used Server Name [fsvaultgjo.company.loc] Num of attempts [1] Total elapsed [0.002s] Result [Success  (0)]
[8524] (w3wp) <1804> EV:M CAutoStorageOnline::CreateObject - Checking MULTI_QI.hr
[2412] (EVIndexAdminService) <8888> EV-H {ServiceBaseEx.OnStart} Waiting for some more time for Indexing Service to start. Start time[23:05:40], Log Time[23:06:00], Elapsed Time[00:00:32.2289802 ms]
[8524] (w3wp) <1804> EV:L {CClientAuthenticate::GenAuthString:#361} Generating auth string...
[8524] (w3wp) <1804> EV:M ClientAuthHelperImpl::GenAuthString Authentication Type: Currently impersonated user Client:(null) ==> AuthToken:SRNOGJO048.company.loc xM1R*****
 
I hope someone can help.
 
Regards
Gunnar@Atea
Operating Systems:

Comments 16 CommentsJump to latest comment

GabeV's picture

This line:

[8524] (w3wp) <8728> EV:L WinHttpRequest::OnCallback. WINHTTP_CALLBACK_STATUS_HEADERS_AVAILABLE. Status code=401

seems to be that there is an issue with IIS. You shouldn't receive a 401 error (Access denied) from IIS. Check the permissions on IIS virtual directories or service status. Can you post the IIS logs as well?

“Success is not final, failure is not fatal: it is the courage to continue that counts.”–Winston Churchill

GabeV's picture

This TN will provide a clean dtrace for EV 10 indexing, so you can compare:

http://www.symantec.com/docs/HOWTO56280

I hope this helps

“Success is not final, failure is not fatal: it is the courage to continue that counts.”–Winston Churchill

Pradeep_Papnai's picture

Do you see any error/warning in system log, specially DCOM related?

Peform IISreset & see if you can start EV indexing service also try recyling EV app pool (EnterpriseVaultAppPool, EVindexingPool) from IIS console.

 

Gunnar Vinje's picture

I have now found DCOM errors.

DCOM got error "2147942405" from the computer FSVAULTGJO when attempting to activate the server:
{F4D3EB5B-C7C5-11D1-90DB-0000F879BE6A}
EV_Ajay's picture

Reset the Vault Service Account by following the instructions listed below.

 

1. Logon to the Enterprise Vault (EV) server as the VSA.
2. Open the Vault Administration Console (VAC).
3. On the Directory container, select Properties and click on the Service Account tab.
4. Enter the correct password for the VSA and click Apply.
5. Click OK to acknowledge that the VSA has been reconfigured.
6. Click OK to exit the Directory Properties.
7. Restart all EV services.
 
Note: The Active Directory user account may get locked out if repeated attempts to restart the Enterprise Vault services are made using invalid credentials.  This will only occur if there are policies in Active Directory defined to limit the number of authentication retry attempts.  In this scenario the user account must be reset before any making changes to the Vault Service Account details in the Enterprise Vault Administrators Console.
 

Thanks,

Ajay

Gunnar Vinje's picture

I did reset the Vault Service Account, and restarted all EV services. But it is still the same problem.

Gunnar Vinje's picture

The normal error I get when starting the indexing service is Event ID 41293 - Configuration Error.

I have just discovered an error in the Event Log:

Faulting application name: MigratorServer.exe, version: 10.0.4.1189, time stamp: 0x51dd97c4
Faulting module name: EVPstApi.dll, version: 10.0.4.1189, time stamp: 0x51dd9522
Exception code: 0xc0000005
Fault offset: 0x00010ea9
Faulting process id: 0xb50
Faulting application start time: 0x01ce8f65963a74bf
Faulting application path: C:\Program Files (x86)\Enterprise Vault\MigratorServer.exe
Faulting module path: C:\Program Files (x86)\Enterprise Vault\EVPstApi.dll
Report Id: fb2b1cbf-fb58-11e2-a4fe-0050568c5053
 
This can be related to another problem.
EV_Ajay's picture

Hi,

You can try :

1. Change the identity in the Application Pools for the EVIndexingPool from the AppPoolIdentity to the Network Service. Run IIS reset and  restart the EVIndexingAdminService.

2.  Resetting the Enterprise Vault service account password within the Enterprise Vault console:

    1.  Right click on the Enterprise Vault Directory and select 'Properties'.

    2.  Click the "Service Account" password.

    3.  Re-enter the service account password and click "Apply"

    4.  Click OK and then restart the Enterprise Vault Admin Service

If you share full Description of Event ID : 41293 that will help more.

 

 

Thanks,

Ajay

Gunnar Vinje's picture

Thanks, but it did not help.

First I get 41395 Warning:

Enterprise Vault will now create the indexing metadata. The Indexing service will be unavailable until the metadata has been created. The creation will begin immediately and may take some time to complete. See the documentation for further details. 
 
For more information, see Help and Support Center at http://entced.symantec.com/entt?product=ev&language=english&version=10.0.4.0&build=10.0.4.1189&error=V-437-41395
 
Then a 41293 Error:
Configuration Error 
 
For more information, see Help and Support Center at http://entced.symantec.com/entt?product=ev&language=english&version=10.0.4.0&build=10.0.4.1189&error=V-437-41293
 
- System 
 
  - Provider 
 
   [ Name]  Enterprise Vault  
 
  - EventID 41293 
 
   [ Qualifiers]  0 
 
   Level 2 
 
   Task 132 
 
   Keywords 0x80000000000000 
 
  - TimeCreated 
 
   [ SystemTime]  2013-08-05T10:52:45.000000000Z 
 
   EventRecordID 241979 
 
   Channel Symantec Enterprise Vault 
 
   Computer SRNOGJO048.customer.loc 
 
   Security 
 
 
- EventData 
 
   Configuration Error 
 

 

Pradeep_Papnai's picture

Hi Gunnar,

Can you try to reset indexing engine http://www.symantec.com/docs/HOWTO59060

Regards

EV-C

 

SOLUTION
Gunnar Vinje's picture

I have now followed the document resetting the indexing engine, and this worked! I was not sure what to delete in IIS, as there was nothing listed when selecting Virtual Directories in the right pane. But deleting the listed EVIndexing folder and the EVIndexingApp application pool before installing the msi file made this a success!

Thanks a lot for your help.

Gunnar Vinje @ Atea

Andrew S Jones's picture

Hi Gunnar,

I had a similar issue in one of my lab environments when upgrading to 10.0.3, however this was due to the server previously being 2003 R2 x64 and then upgraded to 2008 R2.  (Issues with IIS 6 to 7.5 I think)

Hopefully reseting the Indexing Engine will solve your issue, as this goes through deleting the EVIndexing virtual directory.  It may however still give you the error with 'Configuration Error'.  We might be able to track down the specific config error.

However if you still come up against the same issue then you might go with a broader approach and uninstall IIS from the EV server, making sure that Windows Process Activation Service is uninstalled after IIS as well (http://forums.iis.net/t/1170286.aspx/1) to ensure that when you re-install IIS it is clean.  Make a note of what other virtual directories exist before uninstalling IIS, so that you know what needs to be added back.  After the re-install of IIS to recreate the 'EnterpriseVault' virtual directory you would run the following command from the EV installation directory:  Configuration.exe ConfigureWebApps install

For EVAnon you would re-run the OWAUser.wsf script, FSAReporting is a different command line parameter.

 

Thanks,

Andrew

 

Gunnar Vinje's picture

I followed the solution to reset the indexing engine provided by EV-Counselor, and that made the indexing service healthy again.

Thnaks a lot for your effort in helping out.

Gunnar Vinje @ Atea

Pradeep_Papnai's picture

Happy to see that your issue is solved by our suggestion :)