Video Screencast Help
Search Video Help Close Back
to help
New in the Rewards Catalog: Vouchers for "Symantec Technical Specialist" and "Symantec Certified Specialist" exams.

Have I just broken an index?

Updated: 21 May 2010 | 36 comments
James Slack's picture
0 0 Votes
Login to vote
This issue has been solved. See solution.

Hi all,

I was getting a few errors about a few users having index errors in the event log so I though I would have a look.

This was the error I was getting for one of the users in question.

-----------------------------------------------------------------------------------------------------------
Event ID 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:239 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Reference: ValidateFileChecksum
-----------------------------------------------------------------------------------------------------------

so, what I did was at this point was check the folder mentioned and looked at the avtrace.log file, which was 0kb.

Then decided that as it was failed it must need a rebuild.

So in the EV Admin Console, I found the mailbox, went into the advanced tab and hit rebuild.

Checking the event log I got these two messages

-----------------------------------------------------------------------------------------------------------
Event 7294
-----------------------------------------------------------------------------------------------------------
Starting index volume update

Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:0 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Job Id: Vol_239
Job Author: MAG\vault.admin
Job Description: Rebuild Index
-----------------------------------------------------------------------------------------------------------

then

-----------------------------------------------------------------------------------------------------------
Event 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:239 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Reference: Open

Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors. The index volume will remain inaccessible until it has been repaired.

For more information, see Help and Support Center at
-----------------------------------------------------------------------------------------------------------

So now, I thought I would check the avtrace file again to see if it had any info - but the folder was gone. Totally gone. That users Index folder has just vaporised.

Have I just made things worse for myslef?

Any ideas anyone, please?

I have about 7 users in this state and would ideally like to sort them all out.

Cheers

Comments

Scanner001's picture
07
May
2009
1 Vote +1
Login to vote

Let me check about the error

Let me check about the error you presently have

Do you have a backup of the index that you can restore to?

The first issue you saw looks like the checksum file in the index folder is out of sync with wjhat the database expects so all you needed to do is to stop the indexing service and i also stop the storage service just for my own level of comfort then i rename the checksum.dat file to something else and then restart the services. This would have recreated the checksum.dat file and the chances are your index would have come back online

If yuo have a backup let me know as there is a way to stop the index rebuilding and then we can restore the backup and restart the services. The index may then show missing items which can be updated or if the same error (first error) shows up you can perform the process i mentioned above

TonySterling's picture
07
May
2009
2 Votes +2
Login to vote

The index folder is gone

The index folder is gone because you choose to do a rebuild.  The first step in a rebuild is to delete the index.

EV is telling you it is having problems accessing the path: Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\

Can you browse to this path from the EV server?

Tony Sterling

TonySterling's picture
07
May
2009
3 Votes +3
Login to vote

Also, for future reference

Also, for future reference always try the Repair or Update option first.  Rebuild should be your last resort.  :)

As scanner said, if you have a back up of the Index restore that to the location.

Cheers,

Tony Sterling

John Chisari's picture
07
May
2009
4 Votes +4
Login to vote

 The initial error you had

 The initial error you had was because of Invalid Checksum - as you can see from the event - ValidateFileChecksum

Technote on how to fix it for the other 6 (if it's the same) - http://support.veritas.com/docs/294109


Scanner001's picture
08
May
2009
1 Vote +1
Login to vote

James, Any update, did you

James,

Any update, did you manage to do the restore?

Did renaming the dat file fix the others that were giving the error

Liam

Nicola Rositani's picture
10
May
2009
0 Votes 0
Login to vote

James, any news? I'm

James, any news?
I'm interested about solution

James Slack's picture
20
May
2009
0 Votes 0
Login to vote

Update

Sorry, the update is no update.

I just got back to work from pig-flu... so lots of catching up to do.

I will have a read through this lots and see what happens, what I will say is thanks to everyone for posting!

Cheers - James 

James Slack's picture
20
May
2009
0 Votes 0
Login to vote

Proper Update

OK, For this one user I mentioned, I had a look in the event viewer and couldnt see any mention of him.

I opened up his mailbox properties and checked the location folder of his index and went there, loads of Archive IDs, had a look at this users Archive ID and found that folder, there are a bunch of files and all certainly looks well now.

So, for this user, I think I am done - does anyone know any differently?

For the other users that are having this issue

-----------------------------------------------------------------------------------------------------------
Event ID 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:239 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Reference: ValidateFileChecksum
-----------------------------------------------------------------------------------------------------------

I will just follow Scanners / Johns instruction to sort the checksum.

I'll try one user and let you know how I get on.

Many thanks

James

James Slack's picture
20
May
2009
0 Votes 0
Login to vote

Scanner001...

Should this work straight away?

I just did exactly as you said for a user and while it has changed the message, it doesn't inspire confidence!

-----------------------------------------------------------------------------------------------------------
OLD event 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1EFEAEC80E474E2439F7835485652E74D1110000vaultsite/Volume:234 (Pannarod Vuddhijoti)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location01\1EFEAEC80E474E2439F7835485652E74D1110000vaultsite
Reference: ValidateFileChecksum

Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors. The index volume will remain inaccessible until it has been repaired.
-----------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------
NEW event 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1EFEAEC80E474E2439F7835485652E74D1110000vaultsite/Volume:234 (Pannarod Vuddhijoti)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location01\1EFEAEC80E474E2439F7835485652E74D1110000vaultsite
Reference: Open

Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors. The index volume will remain inaccessible until it has been repaired.
-----------------------------------------------------------------------------------------------------------

I will have a look through the link John sent to see what I can work out, but is this expected behaviour or does this indicate something more serious?

Many thanks

James

James Slack's picture
20
May
2009
0 Votes 0
Login to vote

OK, rebuilding

Having read through the link John gave, it makes sence that I need to "update" it.

Solution:
To correct the failed index,
1. Open Windows Explorer on the Enterprise Vault (EV) server that controls the failed index.
2. Navigate to the folder listed in the Event ID 7292 description field.
3. Locate and rename the file Checksum.dat.
4. Navigate to a non-index folder, then close Windows Explorer.
5. Open the IndexVolumeReplay utility in EV6.x or EV7.x and search for the failed index, or access the Index Volume tab of the archive in the Vault Admin Console for EV2007.x.
6. Right-click on the failed index or index volume and select the Update option.
7. Check the EV Event Log for new Event ID 7292 errors to ensure the index update does not fail.

Also, I think this is all because of no shutdown process in place on the server. I am going to write a little BAT file to stop the services and restart the box.

Scanner001's picture
20
May
2009
1 Vote +1
Login to vote

If you rename the

If you rename the Checksum.dat file and restart the services it will create a new checksum.dat

BUT

You will not be able to tun an update or repair because the index will show healthy

AND

Next day when you check the system you will see that there are items missing from the index. At this stage you do a repair not an update

James Slack's picture
22
Jun
2009
0 Votes 0
Login to vote

Back to this again

Other dramas over so I can spend the week looking at all my broken indexes.

I have16 users with dodgy indexes at the moment, and i have created a list to work on.

Here is what most are getting.

-----------------------------------------------------------------------------------------------------------
Event 7292

The index volume has been marked as failed.
Index Volume: 10FD1CE1AB9EC0746BE94366DF369C7631110000vaultsite/Volume:3631 (Victorino Ramirez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location06\10FD1CE1AB9EC0746BE94366DF369C7631110000vaultsite
Reference: ValidateFileChecksum
-----------------------------------------------------------------------------------------------------------

As I said, this is what most of the 16 users are getting. What shoudl I do next to fix these issues?

A guy who I tried an Update / repair on a while back however is getting these now:

-----------------------------------------------------------------------------------------------------------
Event 7264

Abnormal error occurred

Error: The AltaVista dataset file is missing. [0xc0041c7d]
Reference: CAltaVistaIndex/OAVI/ie
Index: 14F77730E08A89146A3D6DCD135DA148C1110000vaultsite/Volume:1130 (Jim Mollison)
Info: H:\EVStorage\PRVAULTDB01\Indexes\Location03
14F77730E08A89146A3D6DCD135DA148C1110000vaultsite
-----------------------------------------------------------------------------------------------------------

followed by

-----------------------------------------------------------------------------------------------------------~
Event 7264

Abnormal error occurred

Error: The AltaVista index is not open [0xc0041c64]
Reference: ST/TR/gs
Index: 14F77730E08A89146A3D6DCD135DA148C1110000vaultsite/Volume:1130 (Jim Mollison)
Info:
-----------------------------------------------------------------------------------------------------------

For this user, I did the following:

Found his mailbox in Vault, went into properties, in the Index Volumes tab, just right clicked and chose update / or repair (I dont remember which but this I have tried both settings on different users with the same results).

What I have done now is told this user to rebuild and will see what that yeilds.

If anyone has any advise for me on either of the 2 scenarios that would be great.

Thanks James

James Slack's picture
22
Jun
2009
0 Votes 0
Login to vote

To clarify...

Basically, I have been advised to do this if I get event 7292:

1. Open Windows Explorer on the Enterprise Vault (EV) server that controls the failed index.
2. Navigate to the folder listed in the Event ID 7292 description field.
3. Locate and rename the file Checksum.dat.
4. Navigate to a non-index folder, then close Windows Explorer.
5. Open the IndexVolumeReplay utility in EV6.x or EV7.x and search for the failed index, or access the Index Volume tab of the archive in the Vault Admin Console for EV2007.x.
6. Right-click on the failed index or index volume and select the Update option.
7. Check the EV Event Log for new Event ID 7292 errors to ensure the index update does not fail.

Is this correct? From what you guys have said above you seem to put me off.

If I don't have a backup, do I have any other options?

Cheers

 

Scanner001's picture
22
Jun
2009
1 Vote +1
Login to vote

 James the line item

 James

the line item "Reference: ValidateFileChecksum" basically tells you that the checksum file shows a difference between that is on disk and what it is expecting

To correct this the steps are as follows
1. Stop the index service
2. Open Windows Explorer on the Enterprise Vault (EV) server that controls the failed index.
3. Navigate to the folder listed in the Event ID 7292 description field.
4. Locate and rename the file Checksum.dat.
5. Start the index service
5. Open the IndexVolumeReplay utility in EV6.x or EV7.x and search for the failed index, or access the Index Volume tab of the archive in the Vault Admin Console for EV2007.x.
6. Right-click on the failed index or index volume and select the Repair option.
7. Check the EV Event Log for new Event ID 7292 errors to ensure the index update does not fail. 

Also check the directory where the indexes reside  for a new checksum.dat file which will be created by the index service


As always if possible take a backup of the indexes before making any changes

James Slack's picture
23
Jun
2009
0 Votes 0
Login to vote

Thanks Scanner

I have taken indexing offline, deleted the checksum file and restarted indexing.

When I went to have a look at the first users "Index Volume" tab in EV 2007, it was saying it was NORMAL.

Would I still need to do a REPAIR?

Cheers

James

Scanner001's picture
23
Jun
2009
1 Vote +1
Login to vote

Not yet. What will now happen

Not yet.

What will now happen is next time the services restart (during you backup) you can check the index folder to find a file called indexmissing.log. You can check now if you wish

This file will hold a list of items missing from the indexe

In the VAC you will see the list of indexes with the number of missing items. When you see this then do the repair on the index with the missing items.

EV will then go out, identify the items missing from the index and then it will attempt to add them to the index. The index will show as rebuilding during this time but it is only repairing not rebuilding so no need to panic

Once they are added you will see in the VAC that there are no items missing from the index. When you see this go back to the index directory and delete the indexmissing.log file.

That should bring your indexes up to date and back to a healthy state.

Here is a SQL query that can check the status in SQL if there are items missing from your indexes or not

I presently have SQL reporting services run this query twice a day and email me the results so I know if any items are missing from my indexes daily so I can perform a quick fix

Run this query against the Enterprisevault Directory database

SELECT FolderName AS 'Folder Name', Offline, Failed, Rebuilding, FailedItems, IndexRootPathEntryId
FROM IndexVolume
WHERE (FailedItems <> '') OR
(FailedItems <> '0') OR
(Offline = 'True') OR
(Failed = 'True') OR
(Rebuilding = 'True')



Scanner001's picture
23
Jun
2009
1 Vote +1
Login to vote

 You may also find this one

 You may also find this one more helpful as it provides the name of the archive so it is easier to identify it in the VAC

I prefer this one to the one in the previous post

SELECT ArchiveName AS 'Archive', FolderName AS 'Folder', [Offline] AS 'Index Offline', Failed AS 'Index Failed', Rebuilding AS 'Index Rebuilding',
failedItems AS 'Failed Items'
FROM Archive a, IndexVolume iv
WHERE a.RootIdentity = iv.RootIdentity and
([Offline] = 'True' or Failed = 'True' or Rebuilding = 'True' or failedItems <> '')

James Slack's picture
24
Jun
2009
0 Votes 0
Login to vote

Thanks again Scanner

Thanks for this info.

I am having a read and learn about this now. I am currently struggling to find the indexmissing.log.

While looking where this log could be I stumbled across this item:
http://seer.entsupport.symantec.com/docs/280895.htm  which is "How to verify the integrity of indexes with the IndexCheck utility when using Enterprise Vault (EV)."

Cheers

Scanner001's picture
24
Jun
2009
0 Votes 0
Login to vote

Search your drive where the

Search your drive where the indexes are stored for indexmissing.log

If you check in the VAC for the archives which were failed yesterday...are they now showing a number of items missing from the indexes in the VAC?

James Slack's picture
24
Jun
2009
0 Votes 0
Login to vote

I cant find indexmissing.log

Any idea how I can find it.

I have a H: driver called Indexes and I have searched that whole drive and there is no indexmissing.log file.

I also searched the C: drive and it't not hiding there either. Is there a way to confirgure where this file resides (and therefore find out where I need to look).

Cheers

Scanner001's picture
24
Jun
2009
0 Votes 0
Login to vote

it will always be in the

it will always be in the directory where the index resides.

If it has not created this file it means that you may not be missing any data from the indexes

Did you restart the services again today (maybe early morning for backups)?

Did you run the last SQL script I gave you to check if there were any missing index items?

 

James Slack's picture
24
Jun
2009
0 Votes 0
Login to vote

Not sure how to check for

Not sure how to check for this:
"If you check in the VAC for the archives which were failed yesterday...are they now showing a number of items missing from the indexes in the VAC?"

Any info?

James Slack's picture
24
Jun
2009
0 Votes 0
Login to vote

OK

I will restart the services again manually (maybe use this as an oppertunity to reboot quickly) and then run that script you gave me.

Will report back.

James Slack's picture
24
Jun
2009
0 Votes 0
Login to vote

OK

I will restart the services again manually (maybe use this as an oppertunity to reboot quickly) and then run that script you gave me.

Will report back. Thanks

James Slack's picture
24
Jun
2009
0 Votes 0
Login to vote

OK

I will restart the services again manually (maybe use this as an oppertunity to reboot quickly) and then run that script you gave me.

Will report back. Thanks for

James Slack's picture
24
Jun
2009
0 Votes 0
Login to vote

OK

I will restart the services again manually (maybe use this as an oppertunity to reboot quickly) and then run that script you gave me.

Will report back. Thanks for your

James Slack's picture
24
Jun
2009
0 Votes 0
Login to vote

OK

I will restart the services again manually (maybe use this as an oppertunity to reboot quickly) and then run that script you gave me.

Will report back. Thanks for your help!!

Scanner001's picture
24
Jun
2009
0 Votes 0
Login to vote

in the VAC expand

in the VAC expand Archives

Then expand File System or Exchange Mailbox depending which was failed
in the right side window you will see the list of your archives

Select one of the archives which failed and select Properties
When the properties opens select the Index Volumes tab
If you scroll left on that window you will see the Failed Items Column
This will show you if there are any missing items for the users archives
Each line in this represents an index location
If there are any items missing from any of the specific index locations you can highlight the one with the failed items, right click and select repair

EV will then try to add the missing items to the index.

When a repair is running the index will show as rebuilding, in fact it is just adding the missing items and it is not doing a rebuild


James Slack's picture
24
Jun
2009
0 Votes 0
Login to vote

OK

Had a look at one of the users.

As I said yesterday, the status is NORMAL. If I look at failed items, it shows as NOT SET.

Does this mean all is OK or does it mean that it doesnt know?

Scanner001's picture
24
Jun
2009
0 Votes 0
Login to vote

It looks like all is

It looks like all is OK
Sounds like nothing was missing from the indexes.

Whatever was pending to be added was added sucessfully

I would say your indexes are good


James Slack's picture
24
Jun
2009
0 Votes 0
Login to vote

Thanks man!

I used your script and it came back with 1 item, which is an index I am doing a repair on.

So for me, all I needed to do was delete the checksum.dat files and restart indexing. That's pretty handy to know!

Cheers for all your help on this scanner.

Scanner001's picture
24
Jun
2009
0 Votes 0
Login to vote

Anytime, That what we are

Anytime, That what we are here for

Kristjan's picture
17
Nov
2009
0 Votes 0
Login to vote

Hi, I also have a similar

Hi,
I also have a similar problem, after a bad failover in a EV cluster some indexes started to give me errors in the event log:

Event Type:    Error
Event Source:    Enterprise Vault
Event Category:    Index Server
Event ID:    7292
Date:        17.11.2009
Time:        5:29:53
User:        N/A
Computer:   Machinename
Description:
The index volume has been marked as failed.
Index Volume: 15A3407290727264C8A139148C5DF188F1110000Machinename/Volume:72 (archivename)
Index Volume Path: E:\Indexing\Index01\15A3407290727264C8A139148C5DF188F1110000machinename
Reference: Makestable
           
 
Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors.  The index volume will remain inaccessible until it has been repaired.

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp
----------------------------------
There are no checksum.dat in those folders and when i tried to update index via admin console, it started to rebuild automaticly

EV version is 2007-SP3

Thanks in advance

Scanner001's picture
17
Nov
2009
0 Votes 0
Login to vote

 Please create a new post and

 Please create a new post and we will be more than happy to assist you

GertjanA's picture
17
Nov
2009
0 Votes 0
Login to vote

Make a new post.

Hello Kristjan,

I advise to make a new post. this is a closed thread.

Regards

Thank you, Gertjan
MCSE, MCITP (2x), MCTS, SCS7.5/8.0/9.0, STS9/10
Company: www.t2.nl
The EV-Dashboard. Get it here: http://evdashboard.so