What can cause the index to fail

Article:HOWTO56273  |  Created: 2011-07-27  |  Updated: 2014-04-04  |  Article URL http://www.symantec.com/docs/HOWTO56273
Article Type
How To




Enterprise Vault Index Admin service (IAS) has been implemented to manage the legacy indexbroker.exe and the new EVIndexAdminService.EXE, it is responsible for monitoring the health of these processes and, if required, will attempt to restart them. As part of the responsibility of this service is to monitor the index volumes for their current state, The StoragePendingWorkDiscovery checks on a regular basis whether there are any index volumes that require synchronization. This is an hourly check that only selects index volumes that have their WorkPending flag set to 1 or when the service is first started, ignores any failed flags. The following event is logged:

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          05/07/2011 12:00:07
Event ID:      3197
Task Category: Archive Task
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      D3X.server.LOCAL
Description:
Starting synchronization run of all mailboxes on Exchange server D700

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp?EvtID=3197
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Enterprise Vault " />
    <EventID Qualifiers="16388">3197</EventID>
    <Level>4</Level>
    <Task>19</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2011-07-05T11:00:07.000000000Z" />
    <EventRecordID>119284</EventRecordID>
    <Channel>Symantec Enterprise Vault</Channel>
    <Computer>D3X.server.LOCAL</Computer>
    <Security />
  </System>
  <EventData>
    <Data>D700</Data>
    <Data>all</Data>
  </EventData>
</Event>

If an error is returned from the index engine indicating a potentially corrupt index volume it will attempt a search against the index volume to confirm whether it is failed.  The 64-engine could potentially have resolved the cause of the error on its own by the time the IAS has processed the error hence this procedure before marking a healthy index volume as failed.

The frequency of checks for index volumes to process is by default set to 1 hour, this is based on the work pending flag.

  • Ignores any currently failed volumes.

The frequency of checks can be controlled using configurable settings

  • VAC -> Server Properties -> Advanced -> Indexing
    • Frequency of checks for index volumes to process
      The time interval between the indexing service’s check for index volumes to process (Hours)


The frequency checks for failed items to process is by default set to run every 6 hours, based on the failed flag and Failed Volume Reason.

  • Includes failed volumes where the reason is one of the failed volume ID’s:
    • 1,  2,  8,  9,  13,  14,  16

This can be controlled using configurable settings

  • VAC -> Server Properties -> Advanced -> Indexing
    • Frequency of checks for failed volumes. 

The time between the indexing service’s attempts to process failed index volumes (hours).
 

Every  ten hours a Full Check occurs and gets a list of index volumes owned by the Indexing service from the Directory database (stored procedure: EnumVolumesForPendingWorkCheck).

  • Processes the a list index volumes obtained to extract the Index Volumes for the appropriate vault store database(s) and then determines if there are any pending items in the ItemsadditionstatusLog, ItemsUpdateStatusLog and ItemsDeletionStatusLog. An Index volume has pending work if there are any Journal JournalArchive, JournalDelete, JournalUpdate records in the volume’s archive and within the its sequence number range with indexcommited=0.(stored procedure: CheckIsPendingRevisions) for Index Volumes belonging to the respective Archives in the Vault Store Databases.
  • The presence of items that are neither marked successful nor poison-pill with any of the three index status log tables will instigate a notification of work to be issued.
  • Any volumes found to have pending work, are queued for submission to the relevant Index Volumes Processor on their respective servers (64bit) or IndexServer (32bit).
    The Frequency of full checks for index volumes to process by default occurs every 10 hours for a full vault store database check.  This can be controlled using configurable settings.
  • VAC -> Server Properties -> Advanced -> Indexing
    o Frequency of full checks for index volumes to process.
    The time interval between the indexing service’s full checks for index volumes to process (Hours).


An information event is raised at the ten hour interval check as follows:

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          04/07/2011 01:52:32
Event ID:      41365
Task Category: Index Admin Service
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      D100.server.LOCAL
Description:
Summary of the current index volume check:

Volumes synchronized: 1
Failed volumes synchronized: 0

The next synchronization checks will take place as follows:

Volumes having pending work in 0 hours and 59 minutes
Failed volumes in 3 hours and 59 minutes
All volumes in 9 hours and 59 minutes.

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp?EvtID=41365
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Enterprise Vault " />
    <EventID Qualifiers="0">41365</EventID>
    <Level>4</Level>
    <Task>132</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2011-07-04T00:52:32.000000000Z" />
    <EventRecordID>10650</EventRecordID>
    <Channel>Symantec Enterprise Vault</Channel>
    <Computer>D100.server.LOCAL</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Summary of the current index volume check:

Volumes synchronized: 1
Failed volumes synchronized: 0

The next synchronization checks will take place as follows:

Volumes having pending work in 0 hours and 59 minutes
Failed volumes in 3 hours and 59 minutes
All volumes in 9 hours and 59 minutes.</Data>
  </EventData>
</Event>
 

General Troubleshooting Information

View the properties of an archive, under the index tab a warning may appear that one or more index volumes are marked as failed and a reason set on the index volume.  Enterprise Vault will mark an index as ‘Failed’ when a serious error occurs while processing the index that EV cannot handle.

Clicking on the index volume shows the failed volume error reason; this can be used later when repairing the index volume.  These index volumes are unavailable until they are repaired; this is to prevent any further changes to the indexes or potentially incorrect search results being returned.

There are several reasons why the index volume may fail, it does not necessarily mean that there is a physical issue with the index and does not necessarily mean the index is corrupt, it simply may need to be updated once the underlying issue has been resolved.

The problem could be any of the following; this is identified by the volume reason ID and Error reason. The volume Reason ID can be found in the Index volume table in the Enterprise vault Directory Database.

Volume Reason ID:
Error Reason:
1
The Index Volume Cannot be found.
2
There are too many Consecutive Items that have failed.
3
There is a sequence number mismatch between the index and storage
4
The Index Volume has been partially Deleted.
5
The dataset file is missing.
6
The dataset file is empty.
7
The dataset file is invalid.
8
Unable to open the index.
9
The checksum file is invalid
10
An error occurred while saving the index data to disk.
11
An error occurred while saving dynamic attribute indexing data to disk
12
The index compaction has failed.
13
There has been an indexing engine error.
14
Failed to Audit Items
15
There are duplicate items in the index.
16
The State file is missing from the index.
17
There was an error accessing or updating the index engine metadata.

 

1. The index volume cannot be found.
The Index Volume Processor (IVP) fails to find the index volumes under the index root path stored in the database.

2. There are too many consecutive items that have failed.
If the IVP fails to fetch 25 separate items from storage at a time, it  will trigger a frenzy error and report “There are too many consecutive items that have failed” and a list of the failed items.
An event will also get logged and list the failed items:

Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault
Date:          6/21/2011 3:40:56 PM
Event ID:      41329
Task Category: Index Volumes Processor
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      IndexServer1.server.local
Description:
Index volume is marked as failed.

Archive Name: DomTest3/twix
Index Volume ID: 128140EE76EDF61489B88550159726C5F1110000TwixAgents.twix.local_7611
Reason: None
Error Type: FrenzyErrorDetected
Description:
Error 1: Error encountered whilst trying to fetch item Sequence Number '28806' Item Identifier '201106208563747~201106200945100000~Z~60DF4C6D4EFBB564BD069207A0453011'

3. There is a sequence number mismatch between the index and storage.
If a disaster recovery of storage has taken place this error indicates that indexing is ahead of storage therefore the Index sequence numbers being out of synch.  A typical cause of this error would be when a user restores the Enterprise Vault databases but leaves the index alone. In this case the user must take some corrective action ( ie.. restore the index to match or rebuild) in order for the failed status to be cleared.

4. The index volume has been partially deleted.
Each 32-bit and 64-bit index contains many files. If something went wrong while EV was attempting to delete the physical index file, the index volume is marked as failed with error number 4. Within each 32-bit index volume any errors are written to the avtrace.log and would report an error with a list of failed items.

5. The dataset file is missing.
Within each 32 bit index there is a file called dataset. The dataset contains configuration information about the index and so it is the most critical file with in the index. This message is displayed if the IAS service fails to locate the dataset file.

6. The dataset file is empty
This message is specific to 32-bit index volumes and would only be encountered if the dataset file is empty. The dataset contains configuration information about the index and so it is the most critical file in the index, if this file becomes empty possibly due to editing then the index cannot open, and a search result will return informational message showing 0 items found.

7. The dataset file is invalid.
Within each 32 bit index there is a file called dataset. The dataset contains configuration information about the index and so it is the most critical file with in the index. If it becomes corrupt, then the index cannot be opened.  Sometimes the dataset file becomes blank and if opened may contain just one line of text rather than all the configuration data it should (perhaps 30 lines or more). The most common cause of this corruption is if the EV server has the /3gb switch set in the boot.ini file. It is strongly recommended that the /3gb switch is never set on EV servers. If the /3gb switch is set, then that setting should be removed.

8 Unable to open the Index.
If there is a low level issue with the index data or location at the point access to the index is requested (ie. Search request), the volume will be marked as failed with error number 8

9 The checksum file is invalid.
This message is specific to 32-bit index volumes and would only be encountered if the index file checksum validation is enabled in the registry. For a search or an update; the checksum is calculated again and compared to the value that has been previously saved. If they do not match, then the volume is marked as failed. This insures that CA and DA searches are running against complete and healthy indexes.

10 An error occurred while saving indexing data to disk.
This is an AV (AltaVista) only error, set when there are issues writing in-memory changes to disk.

11 An error occurred while saving dynamic attribute indexing data to disk
Altavista does not require a pre-configured index schema in order to index and search fields. Fields can be generated dynamically at indexing time, although some types such as numeric values have to be defined when opening the index for fast sorting and relevance ranking.

12 The index compaction has failed.
AltaVista does not keep a copy of the index for live searching whilst updating a secondary copy and then merge the two; this simplifies merges and reduces required disk footprint, but can result in searches being blocked whilst updates and file compactions take place.

13 There has been an indexing engine error.
Enterprise Vault will communicate with the 64-Bit Indexing engine via SOAP interfaces. The physical on-disk indexing operations are performed asynchronously to the indexing requests. Enterprise Vault will push documents through the SOAP API; these documents are then queued by the indexing engine internally and eventually committed to the underlying index files. if there is a low level error during this process, the request will be retried a predefined number of times, if this also fails this message will be returned to the index volume.

14 Failed to Audit Items
Usually indicates there was a problem committing changes in the index to the IndeAdditionStatusLog, ItemDeletionStatusLog or ItemUpdateStatusLog tables.

15 There are duplicate items in the index.
This message is displayed if duplicate items are found in the index during verifaction.

16 The State file is missing from the index.
Within each of the 64-bit index there is a file called state.dat. The State file contains pertinent information regarding the index in XML format:
<?xml version="1.0"?>
<IndexVolumeInternalState Id="19534D0A7AC5FA146842E57572DFCC8A61110000d3x.server.LOCAL_20428">
  <Actions>
    <Additions AcknowledgedCount="14" HighestSubmittedSequenceNumber="10113" HighestAcknowledgedSequenceNumber="10113" LastAcknowledged="2011-06-30T14:22:13.2587108Z" />
    <Deletions AcknowledgedCount="0" HighestSubmittedSequenceNumber="0" HighestAcknowledgedSequenceNumber="0" LastAcknowledged="2011-06-30T13:45:48.5909412Z" />
    <Updates AcknowledgedCount="0" HighestSubmittedSequenceNumber="0" HighestAcknowledgedSequenceNumber="0" LastAcknowledged="2011-06-30T13:45:48.5909412Z" />
  </Actions>
</IndexVolumeInternalState>

This message is displayed if the IAS is unable to locate the State.dat file with in the index volume or data corruption possibly caused by miss-editing.

17 There was an error accessing or updating the index engine metadata
When the Indexing service is started it performs some basic checks to ensure the 64bit indexing engine metadata matches EV’s metadata and that the number of index volumes recorded in the Directory database matches the number of volumes recorded in the indexing engine’s internal database.
If the numbers don’t add up, the IAS will attempt to recreate missing collections within the engine. If something goes wrong whilst a volume is being recreated, the volume will be marked as failed with error number 17.
 




Article URL http://www.symantec.com/docs/HOWTO56273


Terms of use for this information are found in Legal Notices