Video Screencast Help

Index errors after upgrade to 10.0.3

Created: 14 Mar 2013 • Updated: 14 Apr 2013 | 46 comments
Sarah.Seftel's picture
This issue has been solved. See solution.

Hi

Upgrade of EV from 9.0.2 to 10.0.3 in microsoft cluster (active passive).
Upgrade completed succesfully.

Index administration task created with upgrade to all 32bit indexes.

Upgrade is not working (stays inactive), but it looks like system is having indexing problems, as events are full with the following:

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault 
Date:          14/03/2013 13:40:13
Event ID:      40966
Task Category: Index Admin Service 
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Server
Description:
A program fault has raised an exception.
 
Exception: Failed to get subtask for subtask ID 1DB7E29DCE8D64FEA8701D5EFB860EAED1013b00vaultserver
Diagnostic: 
Type: System.ServiceModel.FaultException`1[[Symantec.EnterpriseVault.Indexing.Common.CommunicationFault, EVSharedManagedInterfaces, Version=10.0.3.0, Culture=neutral, PublicKeyToken=26c5e2ccf2b9267c]]
Reference: An error occurred while invoking Void <StopSubTask>b__f().
Error Context : Stopping sub task for TaskEntryId 17DC08710A3B705409EFC2C9200093CFE1013a00vaultserver and SubTaskEntryId 1DB7E29DCE8D64FEA8701D5EFB860EAED1013b00vaultserver
 
Command Line: "E:\Enterprise Vault\EVIndexAdminService.exe" -EntryID:1EC5115667B050B40B71EFC0504B51FD71710000vaultserver
Application Domain: EVIndexAdminService.exe
Process Id: 8836
Thread Id: 18288
Stack Trace: 
Server stack trace: 
   at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
 
Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Symantec.EnterpriseVault.Indexing.IndexVolumesProcessor.WebService.ITaskProcessorWebService.StopProcessingSubtask(String taskId, String subtaskId, StopReason reasonForStop)
   at Symantec.EnterpriseVault.Indexing.Admin.IndexAdminImpl.Stop64BitSubTasks(String taskEntryId, String subTaskEntryId, StopReason reasonForStop)

   at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.InvokeHelper(Object COMInterface, Action act, String context, LoggingMode loggingMode, Boolean rethrow)

-----------------------------------------------------------------------------

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault 
Date:          14/03/2013 13:44:02
Event ID:      7182
Task Category: Web Application (WP)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Server
Description:
Index Search failed: Unspecified error  (0x80004005)  
Index: 1EB6DC7DA00ED2447BA21981FFB2D908D1110000vaultserver/VolumeSet:866 
Internal reference: SRCI 
------------------------------------------------------------------------------
 

Upgrade went smooth with no special issues, all steps performed by the book, including moving the IndexMetaData to shared location.

 
Any ideas?
 
Also attached dtrace of the following:
EVIndexAdminService
EVIndexVolumesProcessor 
IndexBroker
IndexServer  
 
Thanks,
 
Sarah.

 

Operating Systems:

Comments 46 CommentsJump to latest comment

Rob.Wilcox's picture

I would be tempted to contact Symantec Support on this issue, and discuss with them the internal reference you see in the event log entry.  To be fair though, it does sound like 'something' cluster related.

Sarah.Seftel's picture

Just opened a case.

Will update with the results once we have any.

JesusWept3's picture

Delete and recreate the index rebuild tasks would be the first easiest option

Sarah.Seftel's picture

Hi

can't rebuild, synch or do any job on indexes, they all result with same errors like i attached.

also reinstall of binaries didn't solve the issue.

I have a feeling this will end up in a Symantec hotfix in the end, as I was working on this with Symantec yesterday for 4 hours and still have no idea on how to solve it.

JesusWept3's picture

So just so I understand, did they delete the subtasks from the SubTask table and the index tasks from the vac?

Sarah.Seftel's picture

I deleted subtask and tasks from the ev DB's, as after you create the task, you cannot stop it... you get an error saying : the following subtask could not be stopped [archive name] reason : 0x8013150b, followed with event 40966 Exception: Failed to get subtask for subtask ID 1DB7E29DCE8D64FEA8701D5EFB860EAED1013b00vaultserver

 

Major fun smiley

James Drake's picture

I was wondering if you resolved this problem? I have performed the same version upgrade as you but this is a single server and it was from a 2003 to 2008 R2 server. I have a similar issue.

My rebuild and upgrade index tasks never start and just stay inactive, however I can perform synchronisation tasks. It appears as if new indexing is working ok.

Sarah.Seftel's picture

We figured out that dll are not registered correctly on both nodes.
But reinstall of ev media don't seem to be helpful.
It is still being tested and we are still working on the issue.
Will update when we will have a solution.

JesusWept3's picture

I take it FileReRegister.bat doesnt help any?

Sarah.Seftel's picture

FileReRegister.bat was only suggested today, so it will be tested on Sunday when we are back to work.

Hoping that this will solve the issue...

 

Will update.

James Drake's picture

Since I have the opportunity to work on this until the end of today. I tried the FileReRegister.bat, deleted the subtasks from the database, the task from the VAC, rebooted the server.

After trying a rebuild/upgrade again, the same problem persists with the same errors in the event log.

Although I originally thought new content was being indexed, this was checked against the journal archive. 706 items show in the 64bit index, but this has now ceased to increase despite the constant archiving of journal items.

Following the user archive last night, new 64bit indexes were opened but they contain no indexed content and a check from and end user perspecitve shows I cannot search for any of these newly archived items.

I don't know if this is related, but when I try to stop the index subtasks in the GUI I get a message stating thay there has been a logon failure and to check the account the task runs under has 'logon as a service' rights. I am logged on as the EV service account and it has all these and full admin rights already.

For our environment we don't have a massive amount of archived emails (approx 15 million), so I am considering just wiping all the index data and starting again, however the fact that new content now doesn't appear to be indexing doesn't fill me with confidence. At least I have some index content now, even if it is 'closed legacy 32bit'.

 

 

TonySterling's picture

I would not delete all the index data unless it were a last resort.

You could give this a try first.

How to reset the 64-bit Indexing Engine following corruption of core data files.

Article:HOWTO59060  |  Created: 2011-08-26  |  Updated: 2012-09-13  |  Article URL http://www.symantec.com/docs/HOWTO59060

 

James Drake's picture

Thanks but that didn't work. It seems it is now worse, or better depending upon which way you look at it. EV has decided that at small random amount of 64bit indexes are now officially in a failed state. However, trying a rebuild task just results in the same issue of it never starting.

We have several mailboxes that belong to users no longer with the company. I would like to test a complete index deletion and build from scratch for one of these users. What would be the best way of achieving this?

TonySterling's picture

Did you see these entries?

Event ID:      41372

Indexing Service has started the synchronization of index volume metadata between the indexing engine repository and the Directory database. 

Event ID:      41377

Indexing Service has completed the synchronization of index volume metadata between the indexing engine repository and the Directory database.
Total index volume(s) whose metadata was created: x
Total index volume(s) whose location was updated: y

 

Sarah.Seftel's picture

FileReRegister didn't solve the issue for me too I'm afraid, on both nodes.

however - although Synch is not working, I can stop and delete synch task, upgrade task - cannot be either stopped or deleted.

Pass my files to Symantec.

 

Old Man Jensen's picture

James,

Going back to that logon failure, I believe that's referring to the account that the indexing service itself is running under.

I know that in the past, the password for EV Services' accounts have gone corrupt for some odd reason, and the fix has been to just go into the properties of the service itself (via Services.msc) and re-enter the account's password from there.

May as well verify that it's using the EV service account as well, although I doubt that would've been changed.

May as well check that the EV service account does have 'Log on as a Service' rights, although the EV setup does grant this right to the service account right from the start.  Always worth covering that base, however.

Jason Jensen

Director – Archiving & eDiscovery | Verizon Terremark

YouTube | LinkedIn

James Drake's picture

The account is used to start all EV services, so it wouldn't make any sense if it didn't have those rights. However, just to be sure I have re-entered the details against the index service and checked that the account has 'log on as a service' rights - which it does.

After leaving it over the weekend, I have now got:

 

300 32 bit indexes

127 64bit

87 failed

 

Clearly it has opened 127 64bit indexes total but there is nothing at all indexed in them. The event log continues with its continous index errors.

The latest is event 41329 (frenzyerror). The errors in general leading up to this seem to indicate that the indexer simply cannot read the items from the vault to index them. All EV services use the same service account and all other tasks bar indexing work fine. It doesn't look like this is a basic account rights issue.

Sarah.Seftel's picture

OK

Just got to another customer where I upgraded to 10.0.3

64bit index not working at all.

upgrade of indexes not working.

all items archives since upgrade not indexed.

 

I have the feeling that a microsoft update is causing it.

Want to compare windows updates and installed software from my 2 customers and your server?

James Drake's picture

ok sounds like a good idea. I built this server a week ago and it had all standard and critical updates applied from Windows Update. The server is virtual running on an ESXi 5.0 platform with the version of VMware tools that shipped with v 5.0.

SQL 2008 R2 SP2 is installed on the same virtual server.

EV 10.0.3 has both the cumulative hotfix 1 installed and the other addtional one (fix for Exchange 2007 - 2013 mailbox moves)

Windows 2008 R2 SP1 is installed

Outlook 2007 SP3 is the MAPI provider

The list of Windows hotfixes that are installed are as follows

(generated by powershell

 

HotFixId                                                                                                              
--------                                                                                                              
KB981391                                                                                                              
KB981392                                                                                                              
KB977236                                                                                                              
KB981111                                                                                                              
KB977238                                                                                                              
982861                                                                                                                
KB977239                                                                                                              
KB2670838                                                                                                             
KB2592687                                                                                                             
KB981390                                                                                                              
KB2425227                                                                                                             
KB2506014                                                                                                             
KB2506212                                                                                                             
KB2506928                                                                                                             
KB2509553                                                                                                             
KB2511455                                                                                                             
KB2515325                                                                                                             
KB2530548                                                                                                             
KB2533552                                                                                                             
KB2536275                                                                                                             
KB2536276                                                                                                             
KB2541014                                                                                                             
KB2544893                                                                                                             
KB2545698                                                                                                             
KB2547666                                                                                                             
KB2552343                                                                                                             
KB2560656                                                                                                             
KB2563227                                                                                                             
KB2564958                                                                                                             
KB2570947                                                                                                             
KB2574819                                                                                                             
KB2584146                                                                                                             
KB2585542                                                                                                             
KB2603229                                                                                                             
KB2604115                                                                                                             
KB2607047                                                                                                             
KB2608658                                                                                                             
KB2618451                                                                                                             
KB2620704                                                                                                             
KB2620712                                                                                                             
KB2621440                                                                                                             
KB2631813                                                                                                             
KB2636573                                                                                                             
KB2640148                                                                                                             
KB2643719                                                                                                             
KB2644615                                                                                                             
KB2645640                                                                                                             
KB2647753                                                                                                             
KB2653956                                                                                                             
KB2654428                                                                                                             
KB2656356                                                                                                             
KB2659262                                                                                                             
KB2660075                                                                                                             
KB2661254                                                                                                             
KB2667402                                                                                                             
KB2676562                                                                                                             
KB2685811                                                                                                             
KB2685813                                                                                                             
KB2685939                                                                                                             
KB2690533                                                                                                             
KB2691442                                                                                                             
KB2698365                                                                                                             
KB2699779                                                                                                             
KB2705219                                                                                                             
KB2709630                                                                                                             
KB2709981                                                                                                             
KB2712808                                                                                                             
KB2718704                                                                                                             
KB2719857                                                                                                             
KB2726535                                                                                                             
KB2729094                                                                                                             
KB2729452                                                                                                             
KB2732059                                                                                                             
KB2736233                                                                                                             
KB2739159                                                                                                             
KB2742599                                                                                                             
KB2743555                                                                                                             
KB2749655                                                                                                             
KB2750841                                                                                                             
KB2753842                                                                                                             
KB2757638                                                                                                             
KB2758857                                                                                                             
KB2761217                                                                                                             
KB2763523                                                                                                             
KB2765809                                                                                                             
KB2769369                                                                                                             
KB2770660                                                                                                             
KB2778344                                                                                                             
KB2779562                                                                                                             
KB2785220                                                                                                             
KB2786081                                                                                                             
KB2786400                                                                                                             
KB2789645                                                                                                             
KB2790113                                                                                                             
KB2790655                                                                                                             
KB2791765                                                                                                             
KB2797052                                                                                                             
KB2799494                                                                                                             
KB976902                                                                                                              

 

Sarah.Seftel's picture

 

KB2425227                                                                                                             

KB2446710                                                                                                             

KB2484033                                                                                                              

KB2488113                                                                                                             

KB2492386                                                                                                              

KB2497640                                                                                                             

KB2503658                                                                                                              

KB2503665                                                                                                             

KB2505438                                                                                                             

KB2506014                                                                                                             

KB2506212                                                                                                              

KB2506223                                                                                                             

KB2506928                                                                                                              

KB2507618                                                                                                             

KB2508272                                                                                                              

KB2508429                                                                                                             

KB2509553                                                                                                              

KB2510531                                                                                                             

KB2511250                                                                                                              

KB2511455                                                                                                             

KB2512715                                                                                                              

KB2515325                                                                                                             

KB2518869                                                                                                              

KB2522422                                                                                                             

KB2522766                                                                                                             

KB2524375                                                                                                             

KB2528357                                                                                                             

KB2529073                                                                                                              

KB2533552                                                                                                             

KB2534366                                                                                                              

KB2536275                                                                                                             

KB2536276                                                                                                              

KB2539635                                                                                                             

KB2541014                                                                                                              

KB2544521                                                                                                             

KB2544893                                                                                                             

KB2545698                                                                                                             

KB2547666                                                                                                              

KB2552343                                                                                                             

KB2556532                                                                                                              

KB2560656                                                                                                             

KB2562937                                                                                                              

KB2563227                                                                                                             

KB2564958                                                                                                              

KB2567680                                                                                                             

KB2570791                                                                                                              

KB2570947                                                                                                             

KB2572077                                                                                                              

KB2586448                                                                                                             

KB2588516                                                                                                              

KB2598845                                                                                                             

KB2603229                                                                                                             

KB2607576                                                                                                             

KB2616676                                                                                                             

KB2617657                                                                                                              

KB2620704                                                                                                             

KB2633952                                                                                                              

KB2641690                                                                                                             

KB2794119                                                                                                              

KB976902                                                                                                              

KB976932                                                                                                               

KB982018                                                                                                              

will do the compare later...

GertjanA's picture

Although I am on an EV9 environment, I also am on 2008r2sp2.

A few weeks ago we had an issue, where we found that after a patch-weekend, after the reboot, an additional DotNet patch was installed. That patch also required a reboot, but was not listed anywhere. I assume you have also already rebooted this server, but it might be worth a check to see if there is something in the dotnet folder, that has a later date than the last reboot.

I hope you be able to find the issue soon. These are really difficult to troubleshoot and find...

 

Thank you, Gertjan, MCSE, MCITP,MCTS, SCS, STS
Company: www.t2.nl

www.quadrotech-it.com

www.symantec.com/vision

Sarah.Seftel's picture

yes, servers were rebooted so many times I stopped counting...

James Drake's picture

The server has been rebooted about 20 times since the initial installation and only 10 minutes ago. Unfortunately this is not the fix.

Sarah.Seftel's picture

Another customer with same issue:

 

 
HotFixID                                                                                                               
--------                                                                                                               
KB2536276                                                                                                              
KB2552343                                                                                                              
KB2560656                                                                                                              
KB2564958                                                                                                              
KB2570947                                                                                                              
KB2584146                                                                                                              
KB2585542                                                                                                              
KB2604115                                                                                                              
KB2618451                                                                                                              
KB2620704                                                                                                              
KB2620712                                                                                                              
KB2621440                                                                                                              
KB2631813                                                                                                              
KB2633952                                                                                                              
KB2643719                                                                                                              
KB2644615                                                                                                              
KB2645640                                                                                                              
KB2653956                                                                                                              
KB2654428                                                                                                              
KB2655992                                                                                                              
KB2656356                                                                                                              
KB2656411                                                                                                              
KB2658846                                                                                                              
KB2659262                                                                                                              
KB2661254                                                                                                              
KB2667402                                                                                                              
KB2676562                                                                                                              
KB2685939                                                                                                              
KB2690533                                                                                                              
KB2691442                                                                                                              
KB2698365                                                                                                              
KB2705219                                                                                                              
KB2706045                                                                                                              
KB2712808                                                                                                              
KB2718704                                                                                                              
KB2719033                                                                                                              
KB2729452                                                                                                              
KB2736422                                                                                                              
KB2742599                                                                                                              
KB2743555                                                                                                              
KB2749655                                                                                                              
KB2753842                                                                                                              
KB2756921                                                                                                              
KB2757638                                                                                                              
KB2758857                                                                                                              
KB2765809                                                                                                              
KB2769369                                                                                                              
KB2770660                                                                                                              
KB2785220                                                                                                              
KB2786081                                                                                                              
KB976902                                                                                                               
 
 
James Drake's picture

I have done a comparison but unless I have done something wrong there are only a few similarities between all 3 hotfix lists. I used beyond compare, using my list as the master, comparing it against both of yours and then comparing both outputs against each other.

 

KB2536276
KB2552343
KB2560656
KB2564958
KB2570947
KB2620704
KB976902

 

What operating system and service pack were your servers built with?

James Drake's picture

We are already running cumulative hotfix 1 and it hasn't helped us. Let me know if it helps your installations. However, given the situation, the article doesn't fit the problem anyway because nothing is indexed at all, even items archived 10 seconds ago. To be sure, I reapplied the hotfix, it is still not working for us.

Have you got any customers with 10.0.3 indexing succesfully? The last time I did an install of v10 it was 10.0.2 and that was a clean installation, indexing was working ok then.

Sarah.Seftel's picture

OK, my customers servers:

First:

Microsoft Windows Server 2008 R2 Enterprise, on two nodes with microsoft failover cluster.

Second:

Microsoft Windows Server 2008 R2 Enterprise, one server, not clustered.

 

will do the hotfix tomorrow...

 

Sarah.Seftel's picture

As for your question, these are the only 2 upgrades I did to 10.0.3...

the 10.0.2 ones are ok, only 10.0.3 are not working in indexing issue :(

James Drake's picture

Has anyone else had indexing working against EV 10.0.3? I need to resolve this by the end of this week, so whatever it takes, even a complete destruction and rebuild of all indexes.

Sarah.Seftel's picture

James,

Believe me that I need this to be resolved ASAP as well, as one customer is an insurance company and the other is a hospital, and they all need to search what's archived smiley

And yet - my guess is that even if you will try to rebuild indexes, it Will only rebuild the 32bit ones, and will do nothing on 64bit ones.

My systems are archiving, but are not indexing anything since the upgrade.

It is now escalated to engineering, hoping for the best, but realistic, as I faced big issues like that before, and it takes time to point the exact cause frown

I wish I was able to "downgrade" the system to 10.0.2...

Rob.Wilcox's picture

Very odd.  I've not got an 'upgraded' lab, but my straight install of 10.0.3 appears to be indexing things fine.

James Drake's picture

We actually have a test lab here in which I initially performed a basic upgrade test to mirror the live upgrade. Having managed to get access to it again, I found that the indexer was not upgraded with the cumulative hotfix nor has the operating system had any patches applied. It is Windows 2008 R2 SP1 base release. Only features and software to enable to upgrade to EV 10.0.3 were installed.

The indexing works fine in the lab. This now leads me to believe that the problem is either the cumulative hotfix, a Windows update, or a combination of both. I can upgrade the test lab with the cumulative hotfix and see if it breaks it, but I cannot update Windows as internet access from the lab is virtually impossible.

Sarah.Seftel's picture

a reminder:

I didn't install the cumulative hotfix provided by Symantec yet, so we can narrow it down maybe to a windows update.

I wonder what will happene to the server is I uninstall the updates that you found in common for the 3 servers.

 

Sarah.

James Drake's picture

That would be a good step, however our system is live so unfortunately I cannot perform the uninstall. If you are in a position to do that, then it would be appreciated.

Sarah.Seftel's picture

My systems are live too, but one of the customers is in a cluster mode...

I can uninstall it from the passive node, reboot - and fail over the system to it after reboot.

Will check with my customer tomorrow, at this point - nothing to loose...

Yet's picture

I'm following this up as I'm planning to upgrade few sites from 9.0.2 to 10.0.3 due to requirement for Outlook 2013 & Windows 8. With all these issues, I'm having 2nd thought now...

I highly appreciate if you post latest update on your saga.

I believe lots of us is looking after this post.

Goodluck.

James Drake's picture

As soon as we have updates or a resolution we will update this thread. I too have another customer in the next 4 weeks. In that case it will be a completely new install of EV, but if this isn't resolved by then I will be installing 10.0.2.

Saswata Basu's picture

I had recently Migrated and upgraded 3 sites from 9.0.3 32bit - 10.0.3 64bit on Windows 2008 R2 recently but never found any issues with Indexing. My servers were all non clustered servers hence can't conclude if this is a issue with Clustered installations.

Were there Indexing errors on 9.0.3 prior to upgrading to 10.0.3? Did you check the event log on old server?

 

 

James Drake's picture

My server isn't clustered, its about as simple as you can get. Did you have all available Windows updates installed at the time?

The indexing was working fine before the upgrade. The 32bit indexes continue to be searchable, its just they can't be upgraded and no new content is indexed.

Since this is working in our test lab, I am actually fully updating the lab server with Windows updates now, although this is a slow process because the lab is on old hardware. My test server didn't even have SP1 installed, it was the original release of Windows 2008 R2 with a few hotfixes pre SP1.

Basically, if I can break the lab server, at least I have something 'obvious' to work from.

 

Sarah.Seftel's picture

My server is clustered, but it has nothing to do with that, as issue occures on both nodes.

all the 32 bit index worked fine before the upgrade.

the 32 bit also working after the upgrade.

only 64bit upgrade items and new archiving index is not working.

There were no issues with the server archiving & indexing before the upgrade.

 

James Drake's picture

I performed a dtrace against id 32 and 36. The extract below may be a potential indicator to the problem:

If anyone has any ideas what to look for I would welcome the suggestions.

 

433 11:30:09.859  [5796] (EVIndexVolumesProcessor) <Status Checker Thread for 1C72CE853B0CF3C409CD7FB12BB0C6B4D1110000evserver_415:10164> EV-M {StatusChecker} No Additions acknowledgments received. Will check again in 10 seconds...
434 11:30:09.865  [5796] (EVIndexVolumesProcessor) <Fetcher Thread for 1B5447B888E723A49B5FAAC1E3B5A6BC41110000evserver_471:7512> EV-M {EVAdditionDataReaderFactory} Item could not be fetched. ISN=[138340], SSID=[201304045096491~201302030628260000~Z~B0E21A20FA23192ECC0898DDCF95D071] Error number: 0xC0041923. Going to retry after 1000 milliseconds...
435 11:30:09.876  [5796] (EVIndexVolumesProcessor) <10016> EV-M {VelocityIndex}  Did not retrieve any Acknowledgements
436 11:30:09.894  [5796] (EVIndexVolumesProcessor) <Fetcher Thread for 1EF10105583625648B16AF07EEFFC55281110000evserver_487:10172> EV-H {ItemAdditionsAllocator} Exception: Error encountered whilst trying to fetch item Sequence Number '42173' Item Identifier '201304055183215~201302032255420000~Z~B0FBFEB5593E6CBC9A6F8395500A6241' Info:Could not determine if there are additions to do Diag: Type:Symantec.EnterpriseVault.Indexing.ContentSource.IndexableItemException ST:   at Symantec.EnterpriseVault.Indexing.ContentSource.EVAdditionDataReaderFactory.GetItem(AutoReleaseComObject`1 archiveCrawler, UInt64 nextItemSequenceNumber, UInt64 maxItemSequenceNumber, UInt32 timeoutInSeconds, UInt32 maxRetries)|   at Symantec.EnterpriseVault.Indexing.ContentSource.EVAdditionDataReaderFactory.GetInstance(UInt64 nextItemSequenceNumber, UInt64 maxItemSequenceNumber, UInt32 timeoutInSeconds, UInt32 maxRetries)|   at Symantec.EnterpriseVault.Indexing.IndexVolumesProcessor.ItemAdditionFetcher.GetItem(UInt64 nextItemActionID, UInt64 maxItemID)|   at Symantec.EnterpriseVault.Indexing.IndexVolumesProcessor.ItemAdditionFetcher.GetNextItem(UInt64 nextItemID, UInt64 maxItemID)|   at Symantec.EnterpriseVault.Indexing.IndexVolumesProcessor.ItemAdditionsAllocator.get_HasSomeWorkToDo() Inner:None
437 11:30:09.906  [5796] (EVIndexVolumesProcessor) <Fetcher Thread for 1EF10105583625648B16AF07EEFFC55281110000evserver_487:10172> EV-M {EVAdditionDataReaderFactory} Item could not be fetched. ISN=[42173], SSID=[201304055183215~201302032255420000~Z~B0FBFEB5593E6CBC9A6F8395500A6241] Error number: 0xC0041923. Going to retry after 1000 milliseconds...
438 11:30:09.906  [5796] (EVIndexVolumesProcessor) <Status Checker Thread for 1B5447B888E723A49B5FAAC1E3B5A6BC41110000evserver_471:6120> EV-M {StatusChecker} No Additions acknowledgments received. Will check again in 10 seconds...
439 11:30:09.930  [5796] (EVIndexVolumesProcessor) <10016> EV-M {VelocityIndex}  Did not retrieve any Acknowledgements
 

Sarah.Seftel's picture

PLEASE NOTE : The registry values below are only suitable to version 10.0.3

Finally, Issue resolved. We had to add them manually and re-register all dll's

Missing registry values in FIPS folder under [HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\FIPS].

Those were not present on the servers:

[HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\FIPS]
"LogEventFor"=dword:00000001
"UnManagedDLLHash"=hex:88,17,10,56,cb,cc,a6,02,6f,92,7b,29,93,93,0f,42,94,d7,\
  49,48,00,fc,1c,a4,bf,d6,38,7a,9b,02,e8,1b,f6,10,0f,3a,70,a6,d6,eb,16,9e,ab,\
  97,16,3e,d0,d6,1c,5c,88,47,b0,ac,28,01,1e,c0,99,be,37,05,17,60,aa,6b,cb,93,\
  5b,7f,c8,f4,89,9a,d2,3c,78,34,11,a6,f2,07,bf,15,dc,37,5e,3e,cc,40,f4,b7,33,\
  54,ca,62,4d,6e,28,f5,49,98,8d,45,32,7d,65,f9,b4,49,42,62,09,8c,61,37,c5,39,\
  1a,0e,a8,28,f6,00,5e,00,44,75,7b,80,9d,a9,98,d3,90,58,40,1a,16,61,ff,c7,6d,\
  ea
"ManagedDLLHash"=hex:6d,a2,58,c9,f0,d0,64,53,0f,01,9e,64,33,2c,6c,6a,3f,34,7e,\
  87,13,bb,bf,76,33,43,bc,81,0e,b8,d3,ea,78,f5,86,d9,30,f8,7b,23,10,77,23,66,\
  f4,ee,3c,c6,1b,de,e8,f0,e8,d6,51,3c,30,10,1b,30,c5,46,c6,ff,c7,85,88,b1,c0,\
  4f,5a,90,af,a4,49,85,71,00,f1,1e,58,8b,f7,e5,f6,cd,92,ae,57,b1,6e,df,42,78,\
  40,2f,1c,46,c4,8a,ef,71,bc,60,eb,98,f3,c5,e6,8c,58,e2,da,42,35,32,00,a6,93,\
  e7,23,5b,47,c9,51,77,4e,b2,27,b0,67,9a,06,d8,e5,33,28,cf,8f,2d,33,66,a5,92
"InstallPath"="C:\\Program Files:\\Enterprise Vault\\x64"

 Those were present but won't work properly without the ones above…

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\KVS\Enterprise Vault\FIPS]
"LogEventFor"=dword:00000001
"UnManagedDLLHash"=hex:f1,32,05,6b,91,44,9f,82,ad,33,9f,d8,d7,a3,f1,80,e8,fb,\
  25,95,56,5b,fb,c8,3f,3d,43,56,9d,fa,91,d1,c8,20,98,01,fb,e2,d2,50,e0,fc,fd,\
  2e,92,d3,95,70,4f,83,c4,29,18,bb,ce,84,33,fa,b3,4e,3b,5d,f5,e8,a6,4f,a8,22,\
  9f,68,53,66,57,11,67,36,64,89,45,b8,06,11,48,b4,a5,ad,38,c1,c3,49,ac,e7,05,\
  8b,b0,8a,26,f6,f7,f6,be,63,fe,ed,07,89,b7,0e,b9,8e,1b,5e,22,0c,2c,bb,31,b7,\
  2e,87,62,9b,73,85,3d,74,c3,81,4b,fa,aa,5b,a0,86,fb,6d,4f,ce,c6,8a,4b,33,2f,\
  c2
"ManagedDLLHash"=hex:3f,1f,46,d0,fc,d2,0a,a2,db,f8,d5,5c,57,9b,0d,43,ce,82,15,\
  96,82,4f,71,2c,56,83,d9,e6,cf,3b,2f,56,2e,c9,d7,ba,ae,79,d9,cd,90,eb,a3,f5,\
  cf,cb,6f,0d,e4,22,55,a6,b4,0a,c6,a8,60,92,a4,77,c3,f2,c5,c8,7c,c0,59,75,7c,\
  36,63,b1,e5,97,3c,97,13,61,8e,33,8b,d3,54,cf,a1,7d,e9,7f,f3,c3,16,74,bb,58,\
  33,d5,b4,d9,ec,ab,b8,d5,5c,89,5b,81,4c,3c,ef,fd,a4,80,74,4f,1f,79,29,33,bb,\
  ba,2c,2c,7a,34,cb,67,d5,7b,a5,cc,db,e5,78,4d,36,fb,93,81,7a,37,11,79,89,51
"InstallPath"=" C:\\Program Files:\\Enterprise Vault"

added the registry keys manually, re-registered all dll's and restarted services.

SOLUTION
James Drake's picture

I have tried the same fix for my customer and it has also resolved the issue. Clearly there is some common area that means this key isn't created for some installations, but at least its easy to fix and is now something I will be making a standard check aganist my future installations.

Sarah.Seftel's picture

Happy to know it helped!

I will notify Symantec, so they will know it's got nothing to do with my cluster...