Have I just broken an index?
Hi all,
I was getting a few errors about a few users having index errors in the event log so I though I would have a look.
This was the error I was getting for one of the users in question.
-----------------------------------------------------------------------------------------------------------
Event ID 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:239 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Reference: ValidateFileChecksum
-----------------------------------------------------------------------------------------------------------
so, what I did was at this point was check the folder mentioned and looked at the avtrace.log file, which was 0kb.
Then decided that as it was failed it must need a rebuild.
So in the EV Admin Console, I found the mailbox, went into the advanced tab and hit rebuild.
Checking the event log I got these two messages
-----------------------------------------------------------------------------------------------------------
Event 7294
-----------------------------------------------------------------------------------------------------------
Starting index volume update
Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:0 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Job Id: Vol_239
Job Author: MAG\vault.admin
Job Description: Rebuild Index
-----------------------------------------------------------------------------------------------------------
then
-----------------------------------------------------------------------------------------------------------
Event 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:239 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Reference: Open
Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors. The index volume will remain inaccessible until it has been repaired.
For more information, see Help and Support Center at
-----------------------------------------------------------------------------------------------------------
So now, I thought I would check the avtrace file again to see if it had any info - but the folder was gone. Totally gone. That users Index folder has just vaporised.
Have I just made things worse for myslef?
Any ideas anyone, please?
I have about 7 users in this state and would ideally like to sort them all out.
Cheers
Comments
Let me check about the error
Let me check about the error you presently have
Do you have a backup of the index that you can restore to?
The first issue you saw looks like the checksum file in the index folder is out of sync with wjhat the database expects so all you needed to do is to stop the indexing service and i also stop the storage service just for my own level of comfort then i rename the checksum.dat file to something else and then restart the services. This would have recreated the checksum.dat file and the chances are your index would have come back online
If yuo have a backup let me know as there is a way to stop the index rebuilding and then we can restore the backup and restart the services. The index may then show missing items which can be updated or if the same error (first error) shows up you can perform the process i mentioned above
Liam Finn
http://www.plymouthrocklodge47.com
The index folder is gone
The index folder is gone because you choose to do a rebuild. The first step in a rebuild is to delete the index.
EV is telling you it is having problems accessing the path: Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\
Can you browse to this path from the EV server?
Tony Sterling
Also, for future reference
Also, for future reference always try the Repair or Update option first. Rebuild should be your last resort. :)
As scanner said, if you have a back up of the Index restore that to the location.
Cheers,
Tony Sterling
The initial error you had
The initial error you had was because of Invalid Checksum - as you can see from the event - ValidateFileChecksum
Technote on how to fix it for the other 6 (if it's the same) - http://support.veritas.com/docs/294109
James, Any update, did you
James,
Any update, did you manage to do the restore?
Did renaming the dat file fix the others that were giving the error
Liam
Liam Finn
http://www.plymouthrocklodge47.com
James, any news? I'm
James, any news?
I'm interested about solution
Update
Sorry, the update is no update.
I just got back to work from pig-flu... so lots of catching up to do.
I will have a read through this lots and see what happens, what I will say is thanks to everyone for posting!
Cheers - James
Proper Update
OK, For this one user I mentioned, I had a look in the event viewer and couldnt see any mention of him.
I opened up his mailbox properties and checked the location folder of his index and went there, loads of Archive IDs, had a look at this users Archive ID and found that folder, there are a bunch of files and all certainly looks well now.
So, for this user, I think I am done - does anyone know any differently?
For the other users that are having this issue
-----------------------------------------------------------------------------------------------------------
Event ID 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1F346F1B084E5CF4F8A594A365C7223411110000vaultsite/Volume:239 (Raymond Rodriguez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location04\1F346F1B084E5CF4F8A594A365C7223411110000vaultsite
Reference: ValidateFileChecksum
-----------------------------------------------------------------------------------------------------------
I will just follow Scanners / Johns instruction to sort the checksum.
I'll try one user and let you know how I get on.
Many thanks
James
Scanner001...
Should this work straight away?
I just did exactly as you said for a user and while it has changed the message, it doesn't inspire confidence!
-----------------------------------------------------------------------------------------------------------
OLD event 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1EFEAEC80E474E2439F7835485652E74D1110000vaultsite/Volume:234 (Pannarod Vuddhijoti)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location01\1EFEAEC80E474E2439F7835485652E74D1110000vaultsite
Reference: ValidateFileChecksum
Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors. The index volume will remain inaccessible until it has been repaired.
-----------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------
NEW event 7292
-----------------------------------------------------------------------------------------------------------
The index volume has been marked as failed.
Index Volume: 1EFEAEC80E474E2439F7835485652E74D1110000vaultsite/Volume:234 (Pannarod Vuddhijoti)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location01\1EFEAEC80E474E2439F7835485652E74D1110000vaultsite
Reference: Open
Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors. The index volume will remain inaccessible until it has been repaired.
-----------------------------------------------------------------------------------------------------------
I will have a look through the link John sent to see what I can work out, but is this expected behaviour or does this indicate something more serious?
Many thanks
James
OK, rebuilding
Having read through the link John gave, it makes sence that I need to "update" it.
Solution:
To correct the failed index,
1. Open Windows Explorer on the Enterprise Vault (EV) server that controls the failed index.
2. Navigate to the folder listed in the Event ID 7292 description field.
3. Locate and rename the file Checksum.dat.
4. Navigate to a non-index folder, then close Windows Explorer.
5. Open the IndexVolumeReplay utility in EV6.x or EV7.x and search for the failed index, or access the Index Volume tab of the archive in the Vault Admin Console for EV2007.x.
6. Right-click on the failed index or index volume and select the Update option.
7. Check the EV Event Log for new Event ID 7292 errors to ensure the index update does not fail.
Also, I think this is all because of no shutdown process in place on the server. I am going to write a little BAT file to stop the services and restart the box.
If you rename the
If you rename the Checksum.dat file and restart the services it will create a new checksum.dat
BUT
You will not be able to tun an update or repair because the index will show healthy
AND
Next day when you check the system you will see that there are items missing from the index. At this stage you do a repair not an update
Liam Finn
http://www.plymouthrocklodge47.com
Back to this again
Other dramas over so I can spend the week looking at all my broken indexes.
I have16 users with dodgy indexes at the moment, and i have created a list to work on.
Here is what most are getting.
-----------------------------------------------------------------------------------------------------------
Event 7292
The index volume has been marked as failed.
Index Volume: 10FD1CE1AB9EC0746BE94366DF369C7631110000vaultsite/Volume:3631 (Victorino Ramirez)
Index Volume Path: H:\EVStorage\PRVAULTDB01\Indexes\Location06\10FD1CE1AB9EC0746BE94366DF369C7631110000vaultsite
Reference: ValidateFileChecksum
-----------------------------------------------------------------------------------------------------------
As I said, this is what most of the 16 users are getting. What shoudl I do next to fix these issues?
A guy who I tried an Update / repair on a while back however is getting these now:
-----------------------------------------------------------------------------------------------------------
Event 7264
Abnormal error occurred
Error: The AltaVista dataset file is missing. [0xc0041c7d]
Reference: CAltaVistaIndex/OAVI/ie
Index: 14F77730E08A89146A3D6DCD135DA148C1110000vaultsite/Volume:1130 (Jim Mollison)
Info: H:\EVStorage\PRVAULTDB01\Indexes\Location03
14F77730E08A89146A3D6DCD135DA148C1110000vaultsite
-----------------------------------------------------------------------------------------------------------
followed by
-----------------------------------------------------------------------------------------------------------~
Event 7264
Abnormal error occurred
Error: The AltaVista index is not open [0xc0041c64]
Reference: ST/TR/gs
Index: 14F77730E08A89146A3D6DCD135DA148C1110000vaultsite/Volume:1130 (Jim Mollison)
Info:
-----------------------------------------------------------------------------------------------------------
For this user, I did the following:
Found his mailbox in Vault, went into properties, in the Index Volumes tab, just right clicked and chose update / or repair (I dont remember which but this I have tried both settings on different users with the same results).
What I have done now is told this user to rebuild and will see what that yeilds.
If anyone has any advise for me on either of the 2 scenarios that would be great.
Thanks James
To clarify...
Basically, I have been advised to do this if I get event 7292:
1. Open Windows Explorer on the Enterprise Vault (EV) server that controls the failed index.
2. Navigate to the folder listed in the Event ID 7292 description field.
3. Locate and rename the file Checksum.dat.
4. Navigate to a non-index folder, then close Windows Explorer.
5. Open the IndexVolumeReplay utility in EV6.x or EV7.x and search for the failed index, or access the Index Volume tab of the archive in the Vault Admin Console for EV2007.x.
6. Right-click on the failed index or index volume and select the Update option.
7. Check the EV Event Log for new Event ID 7292 errors to ensure the index update does not fail.
Is this correct? From what you guys have said above you seem to put me off.
If I don't have a backup, do I have any other options?
Cheers
James the line item
James
the line item "Reference: ValidateFileChecksum" basically tells you that the checksum file shows a difference between that is on disk and what it is expecting
To correct this the steps are as follows
1. Stop the index service
2. Open Windows Explorer on the Enterprise Vault (EV) server that controls the failed index.
3. Navigate to the folder listed in the Event ID 7292 description field.
4. Locate and rename the file Checksum.dat.
5. Start the index service
5. Open the IndexVolumeReplay utility in EV6.x or EV7.x and search for the failed index, or access the Index Volume tab of the archive in the Vault Admin Console for EV2007.x.
6. Right-click on the failed index or index volume and select the Repair option.
7. Check the EV Event Log for new Event ID 7292 errors to ensure the index update does not fail.
Also check the directory where the indexes reside for a new checksum.dat file which will be created by the index service
As always if possible take a backup of the indexes before making any changes
Liam Finn
http://www.plymouthrocklodge47.com
Thanks Scanner
I have taken indexing offline, deleted the checksum file and restarted indexing.
When I went to have a look at the first users "Index Volume" tab in EV 2007, it was saying it was NORMAL.
Would I still need to do a REPAIR?
Cheers
James
Not yet. What will now happen
Not yet.
What will now happen is next time the services restart (during you backup) you can check the index folder to find a file called indexmissing.log. You can check now if you wish
This file will hold a list of items missing from the indexe
In the VAC you will see the list of indexes with the number of missing items. When you see this then do the repair on the index with the missing items.
EV will then go out, identify the items missing from the index and then it will attempt to add them to the index. The index will show as rebuilding during this time but it is only repairing not rebuilding so no need to panic
Once they are added you will see in the VAC that there are no items missing from the index. When you see this go back to the index directory and delete the indexmissing.log file.
That should bring your indexes up to date and back to a healthy state.
Here is a SQL query that can check the status in SQL if there are items missing from your indexes or not
I presently have SQL reporting services run this query twice a day and email me the results so I know if any items are missing from my indexes daily so I can perform a quick fix
Run this query against the Enterprisevault Directory database
SELECT FolderName AS 'Folder Name', Offline, Failed, Rebuilding, FailedItems, IndexRootPathEntryId
FROM IndexVolume
WHERE (FailedItems <> '') OR
(FailedItems <> '0') OR
(Offline = 'True') OR
(Failed = 'True') OR
(Rebuilding = 'True')
Liam Finn
http://www.plymouthrocklodge47.com
You may also find this one
You may also find this one more helpful as it provides the name of the archive so it is easier to identify it in the VAC
I prefer this one to the one in the previous post
SELECT ArchiveName AS 'Archive', FolderName AS 'Folder', [Offline] AS 'Index Offline', Failed AS 'Index Failed', Rebuilding AS 'Index Rebuilding',
failedItems AS 'Failed Items'
FROM Archive a, IndexVolume iv
WHERE a.RootIdentity = iv.RootIdentity and
([Offline] = 'True' or Failed = 'True' or Rebuilding = 'True' or failedItems <> '')
Liam Finn
http://www.plymouthrocklodge47.com
Thanks again Scanner
Thanks for this info.
I am having a read and learn about this now. I am currently struggling to find the indexmissing.log.
While looking where this log could be I stumbled across this item:
http://seer.entsupport.symantec.com/docs/280895.htm which is "How to verify the integrity of indexes with the IndexCheck utility when using Enterprise Vault (EV)."
Cheers
This is also pretty useful info
Indexing Best practice Guide
ftp://ftp.support.veritas.com/pub/support/products/Exchange_Mailbox_Archiving_Unit/293428.pdf
Search your drive where the
Search your drive where the indexes are stored for indexmissing.log
If you check in the VAC for the archives which were failed yesterday...are they now showing a number of items missing from the indexes in the VAC?
Liam Finn
http://www.plymouthrocklodge47.com
I cant find indexmissing.log
Any idea how I can find it.
I have a H: driver called Indexes and I have searched that whole drive and there is no indexmissing.log file.
I also searched the C: drive and it't not hiding there either. Is there a way to confirgure where this file resides (and therefore find out where I need to look).
Cheers
it will always be in the
it will always be in the directory where the index resides.
If it has not created this file it means that you may not be missing any data from the indexes
Did you restart the services again today (maybe early morning for backups)?
Did you run the last SQL script I gave you to check if there were any missing index items?
Liam Finn
http://www.plymouthrocklodge47.com
Not sure how to check for
Not sure how to check for this:
"If you check in the VAC for the archives which were failed yesterday...are they now showing a number of items missing from the indexes in the VAC?"
Any info?
OK
I will restart the services again manually (maybe use this as an oppertunity to reboot quickly) and then run that script you gave me.
Will report back.
OK
I will restart the services again manually (maybe use this as an oppertunity to reboot quickly) and then run that script you gave me.
Will report back. Thanks
OK
I will restart the services again manually (maybe use this as an oppertunity to reboot quickly) and then run that script you gave me.
Will report back. Thanks for
OK
I will restart the services again manually (maybe use this as an oppertunity to reboot quickly) and then run that script you gave me.
Will report back. Thanks for your
OK
I will restart the services again manually (maybe use this as an oppertunity to reboot quickly) and then run that script you gave me.
Will report back. Thanks for your help!!
in the VAC expand
in the VAC expand Archives
Then expand File System or Exchange Mailbox depending which was failed
in the right side window you will see the list of your archives
Select one of the archives which failed and select Properties
When the properties opens select the Index Volumes tab
If you scroll left on that window you will see the Failed Items Column
This will show you if there are any missing items for the users archives
Each line in this represents an index location
If there are any items missing from any of the specific index locations you can highlight the one with the failed items, right click and select repair
EV will then try to add the missing items to the index.
When a repair is running the index will show as rebuilding, in fact it is just adding the missing items and it is not doing a rebuild
Liam Finn
http://www.plymouthrocklodge47.com
OK
Had a look at one of the users.
As I said yesterday, the status is NORMAL. If I look at failed items, it shows as NOT SET.
Does this mean all is OK or does it mean that it doesnt know?
It looks like all is
It looks like all is OK
Sounds like nothing was missing from the indexes.
Whatever was pending to be added was added sucessfully
I would say your indexes are good
Liam Finn
http://www.plymouthrocklodge47.com
Thanks man!
I used your script and it came back with 1 item, which is an index I am doing a repair on.
So for me, all I needed to do was delete the checksum.dat files and restart indexing. That's pretty handy to know!
Cheers for all your help on this scanner.
Anytime, That what we are
Anytime, That what we are here for
Liam Finn
http://www.plymouthrocklodge47.com
Hi, I also have a similar
Hi,
I also have a similar problem, after a bad failover in a EV cluster some indexes started to give me errors in the event log:
Event Type: Error
Event Source: Enterprise Vault
Event Category: Index Server
Event ID: 7292
Date: 17.11.2009
Time: 5:29:53
User: N/A
Computer: Machinename
Description:
The index volume has been marked as failed.
Index Volume: 15A3407290727264C8A139148C5DF188F1110000Machinename/Volume:72 (archivename)
Index Volume Path: E:\Indexing\Index01\15A3407290727264C8A139148C5DF188F1110000machinename
Reference: Makestable
Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors. The index volume will remain inaccessible until it has been repaired.
For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp
----------------------------------
There are no checksum.dat in those folders and when i tried to update index via admin console, it started to rebuild automaticly
EV version is 2007-SP3
Thanks in advance
Please create a new post and
Please create a new post and we will be more than happy to assist you
Liam Finn
http://www.plymouthrocklodge47.com
Make a new post.
Hello Kristjan,
I advise to make a new post. this is a closed thread.
Regards
Thank you, Gertjan
MCSE, MCITP (2x), MCTS, SCS7.5/8.0/9.0, STS9/10
Company: www.t2.nl
The EV-Dashboard. Get it here: http://evdashboard.so
Would you like to reply?
Login or Register to post your comment.