Video Screencast Help
Protect Your POS Environment Against Retail Data Breaches. Learn More.

Index Volume has been marked as failed.

Created: 02 Jun 2009 • Updated: 21 May 2010 | 15 comments
This issue has been solved. See solution.

We are using Enterprise Vault v8.0.1.1560, that was upgraded from v7.0 a few months ago.

The system is Archiving WIndows Exchange Mailboxes and Journal Mailbox.

The User Archives are fine, but the Journal Archive has Failed.

A new Journal Archive was created a Month ago, so we could continue Archiving a new Journal Mailbox and repair or rebuild the old one.

I had to resort to the Rebuild Option and after several days, the process completed 85% of the final update stage and FAILED. I tried to repair
several times, with the same results.

The Event log states Index Failed. Reference: Too many consecutive failed items.

Should I try to restore the Index from a few months ago and repair again ? Or is there another option I should try first.

If I do restore, will it effect the New Index in the same location ?

Thanks,

Comments 15 CommentsJump to latest comment

TonySterling's picture

You will want to look at what is causing the failed items.  This is typically a failure connecting to the storage device.  Make sure that is ok and try to do an update on the index volume.

RBostwick's picture

Thanks Tony for the suggestion.
I am able to Browse the attached Storage device from the Archive Server without a problem and have the other Archives
on this storage device as well.

The Original Index has over a years worth of Indexed Data on it for the Journal Mailbox. The new Journal mailbox has the Index located on the same storage device and can be searched with no problem.

Being that the original has Failed to Rebuild or Update, it is sitting in an unusable state.

I was hoping there would be another solution, without having to restore the Index Folders.

 

 

Liam Finn's picture

What events are created in the Event log when you try to repair the index?

Also do a DTrace of the indexbroker service and the storagebroker service in verbose mode when you attempt the repair and attach the output file so we can see the result

There is a way to set a poison pill setting in the registry to bypass the default number of failures to allow your index to rebuild but you will be missing items in the index after it completes

Here is how to make the setting

HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\Indexing\PoisonPillCount
DWORD
Default: 1
This will set the number of attempts to process an item before it is deemed to be a poison pill. The action taken depends upon what's being processed...
e.g.
After PoisonPillCount consecutive failed attempts to open an index volume it is deemed to be corrupt and is marked as 'failed'
After PoisonPillCount failed attempts to add an item to the index the addition is abandoned and the index update steps onto the next item.

If you do not have the following then please create it, if you do then change the value to '100000' :-

HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\Indexing\MaxConsecutivePoisonPillItems
DWORD
Default: 100
The maximum number of consecutive poison pill items before an index update is deemed to be a poison pill and abandoned. The index update will be restarted on indexing service restart or the hourly check for pending updates.

Remember please backup your registry before making any changes and this setting is to be removed after you rebuild the index

SOLUTION
Maverik's picture

Hi

Basically the issue will be occurring because if an index cannot add 25 ( I believe is the default) items consecutively then it will be marked as failed. Each failure will be written to the event log. You can increase this poison pill limit by use a regards key that I cannot recall at the mo. I will add it tomorrow unless someone beats me to it. What this should allow is say you have 26 consecutive failed items out of 5 million you can rebuild the index at least.

Would this be accepatable for you?

RBostwick's picture

Thanks for the suggestions.

I added the Registry settings, restarted the Index and Storage Service.

The Index Update is running currently. Hopefully it will finish in a day or so and not Fail again.

I will keep you posted.

 

Liam Finn's picture

Make sure you get a DTrace of the indexbroker service and the storagebroker service so we can see the details of the errors you are receiving when trying to rebuild

Liam Finn's picture

This should allow it to rebuild for you but it will be missing items.

You will need to investigate the reason for the failure, correct it and then repair the index to have the missing items repopulated back into the index

Keep us posted.

I would advise you open a support case for this.

The rebuild could be failing because of missing files in the archive
Orphan save sets or missing entries in the archive folder tables in the Enterprise vault Directory database...to name a few

Chances are you will need support to assist in getting the issue corrected

RBostwick's picture

Thanks for all the HELP and suggestions.

The Registry settings allowed the Index Update to continue and finish.

I am able search the Indexed Archives correctly now.

I will back up and remove Registry settings as the next step.

 

 

Deepak W's picture

even i faced the same issue, solution for the same is as mentioned below...

follow below listed steps

1. open the EV console
2. right click on Archives
3. check index volume
4. check mark on the failed index volume 
5. click on search
6. it will show the failed indexes in some time
7. right click those indexes and select Rebuild
8. after some time again perform steps 2-5 steps
9. failed index will be repaired and no error will be there
 

mark this as solution if this resolves your problem...

-- Deepak W (Kindly close the thread if your query is resolved)

Liam Finn's picture

This is fine if you get no errors during an index rebuild. Seeing as in this case there were errors that caused the index rebuild to fail the root cause of the failures needs to be found before the repair can complete sucessfully

Liam Finn's picture

OK good

at least we got you back running

you now have items missing from your index because of the failure. We need to find the reason why they filed

When you did the rebuild did you run a dtrace on the indexbroker and storage services?

If not please setup a dtrace of these and try to repair the index. A repair will try to go out and insert the failed items into the index and the dtrace will show whats happening and give an idea as to why it is failing

Also any event logs you have would be helpful to dientify why the items failed

To see the number of items go to the the folder where the index is and fine a file called indexmissing.log this file will list the savesets that failed to be added to the index. A count of these will tell you how many items you are missing from your index and give you an idea as to how big the issue is

Liam Finn's picture

Any update on this?

Have you managed to get the dtrace or have you worked with support to identify why the last of the items coudld not be indexed?

Please let us know if it is solved so as the thread can be marked as solved

RBostwick's picture

I understand the need to find out the Root cause.

I have never used DTRACE.

What would be the syntax to start it logging the Index and / or storage broker ?

From the messages during the last Update Process, there were just 26 bad records, which was just 1
more then the system default allowed.

Thanks,

Liam Finn's picture

Open a dos prompt
Go to the directory where Enterprise Vault is installed this is normally c:\Program Files\ Enterprise Vault

Type DTrace and press enter

This will change the command prompt window

type view (this will list all services you can trace

look over the list and find the number associated with the indexbroker and storagebroker services
To activate each type SET (the number of the service) then v
then press enter
E.G. Set 43 v

This sets the service number 43 to trace in verbose mode
Next type Log (the directory you want the log to go to)
E.G Log c:\EV\dtrace.log
The logs can get very large so put them on a drive that has plenty of space
Now start the rebuild or let what is currently running to continue This will trap all EV functions for the two services

Watch the log for words like error, failure, failed and post part of the log on here so we can review it with you

To stop dtracing type LOG again and press enter and follow the onscreen prompt

to stop all tracing type quit and enter