Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Failed Index Volume

Created: 01 Mar 2013 • Updated: 03 Apr 2013 | 8 comments
This issue has been solved. See solution.

We have EV 10.0.3 for Exchange 2007 and recently we ended up with Indexing issue on only Journal mailbox. We cannot search on Journal mailbox from December 2012. Going to Indexing on the EV console we can see 1 failed index. Indexing and searching is fine for all other users.

We recently migrated from EV 8 to 10.0.1 and this is where after a while one of he Journal indexing volume (64-bit) failed and this is the only 64 bit indexing for Journal archive.

We have tried various fixes including:

verifying and Synchronise.

Some of the event logs suggested to upgrade to EV 10.0.3 and we done the upgrade but not luck with the fix.

Symantec support suggested to use EVSVR to locate the missing items. then we applied EVSVR in report mode and then in repair mode it identified the missing items and deleted them from the indexing.

then it was suggested to rebuild the indexes (only for the failed volume) and we done that and completed the rebuilding. the failed index on the Indexing went a way for a short time then restarting the index server it brought back the failed status and showing error id 28998 and 41329 (Error Type: FrenzyErrorDetected).

I am not sure what else we need to do as I believe Symantec is running out of options here.

Any suggestions or ideas please.

thanks

Operating Systems:

Comments 8 CommentsJump to latest comment

TonySterling's picture

From the sounds of things it seems like there might be additional missing items from the journal archive.

Is that what the events indicate?

Advisor's picture

Yes to add on to this, Error type indicates more missing/corrupt items. If you want quick way to get index stable, then you can use poison pill setting to ignore the failed items.

EVRocks's picture

Thank you Guys for your responses.

I did try widening the range but still the same errors and still one index of the Journal is still failing.

As for the poison pill setting... what will actually happened to these missing items. will this change will ignore the failed items means these items will not be indexed and they will not come up when search applied in Journal archive.

Am i right to think that the index rebuilding on that specific parition will remove those items from the sql databases.

Regards,

JesusWept3's picture

Well it really depends what the frenzy failed on and what the event is.
And it also depends on what you did with the EVSVR repair, did it remove the database entries? because indexing won't remove anything from SQL, only the EVSVR will

So what would really help is

1. Paste the Event ID that caused the frenzy
2. Tell us what options you used for the EVSVR repair
3. If you can, attach the Index Rebuild report file for that user

Rob.Wilcox's picture

Fwiw if you use the poison pill setting then the items won't be inthe iindex and won't be used for searching.

If you rebuild then it might be that the archive has the items and indexing can handle them this time, or they could be items which indexing engine still has problems with. Too many of them in succession and the rebuild will fail... Unless you use the poison pill settings.

EVRocks's picture

Guys,

I have now the top Symantec experts looking into this indexing issue and they are struggling to resolve this issue. apparently this is now a known issue with migrating from EV 8 to 10 when indexing house 32 and 64 bit platforms.

We have tried so many fixes and now I am kind of losing the track of what we are doing. but it seems that we are close to fixing this problem.
we identified the missing items. running EVSVR a few times and no change.
We are now started working straight from the sql tables.
While having this issue moving the location of indexing to another drive has complicated the troubleshooting.

- the event that I get now is 28998 - see below
- please see the attached report for the evsvr and the rebuilding index task

Log Name: Symantec Enterprise Vault
Source: Enterprise Vault
Date: 25/03/2013 14:58:04
Event ID: 28998
Task Category: Storage Crawler
Level: Error
Keywords: Classic
User: N/A
Computer: APPSARC001FP.iprod.local
Description:
Retrieval of a saveset failed.

Reason: The system cannot find the path specified. (0x80070003)
Saveset Id: 201212014357667~201212010944210000~Z~615F24B6D2F275A8ED10BD038F25B951
Vault Id: 13694F6308BCD4D4886C68EFFED698AD81110000evault
Index Sequence Number: 24466150
Internal Reference: Fetching Saveset for indexing

For more information, see Help and Support Center at http://entced.symantec.com/entt?product=ev&languag...
Event Xml:

28998
2
113
0x80000000000000

766555
Symantec Enterprise Vault
APPSARC001FP.iprod.local

The system cannot find the path specified. (0x80070003)
201212014357667~201212010944210000~Z~615F24B6D2F275A8ED10BD038F25B951
13694F6308BCD4D4886C68EFFED698AD81110000evault
24466150
Fetching Saveset for indexing

thank you all

AttachmentSize
EVSVR_20130210223318 - Copy.txt 318.22 KB
IndexingSynchronizeReport_Journal Archive IPROD_[22980105 - 24466141].txt 3.5 KB
EVRocks's picture

There were two subfolders that appear to be missing in the Storage area for Enterprise Vault.

EV pointing to missing DVS files in these locations that were attempting to be indexed. The Index Service was hanging on searching for these items causing the JournalArchive table to build up with items ready to be committed to index.

RESOLUTION:

Since there were not any backups to restore the DVS Files back to this folder, the pointers in the Vault Store database were removed using the DeleteSingleSaveset procedure.  Once these pointers were removed, the Journal Index Volume started to update correctly and cleared the items waiting to be indexed from the JournalArchive table of the Vault Store.

it's not known how and why these items gone missing completley despite of having backing up EV every night.

It took over three months to completly fix this issue. I hope this will not happen to anyone elae during the EV migration to 10.0.

SOLUTION