Searches remain in 'Not Started' or 'Searching' but never complete.

Article:TECH53400  |  Created: 2007-01-10  |  Updated: 2014-08-13  |  Article URL http://www.symantec.com/docs/TECH53400
Article Type
Technical Solution

Product(s)

Issue



Searches remain in 'Not Started' or 'Searching' but never complete.

 


Cause



Searches that do not start can be caused by several factors, such as but not limited to:

  • Transaction log restrictions
  • Licensing
  • Older searches that have never completed
  • Incorrectly configured searches
  • Searches looking for index volumes that no longer exist (i.e., due to rebuilding all index volumes of an archive to a difference size than before).

Searches that never complete are normally waiting on a previous search to release resources before the current search in question progresses.

 


Solution



Step 1: Verify there are no SQL database issues.
 
  • Tempdb database Data and Transaction logs are not full
  • Customer or Configuration database Data and Transaction logs are not full
  • Customer database indexes for the tables are not over 50% fragmented


Run the following SQL queries (if the CADA_Quick Check or CADA_ConfigCheck utility has been provided by EV Support and has already been run, look at the results)

DBCC SQLPERF(LOGSPACE)
DBCC SHOWCONTIG (tblIntDiscoveredItems) WITH ALL_INDEXES
DBCC SHOWCONTIG (tblProductionToDiscoveredItem) WITH ALL_INDEXES
DBCC SHOWCONTIG (tblDiscoveredItemToSearch) WITH ALL_INDEXES
DBCC SHOWCONTIG (tblSearchItems) WITH ALL_INDEXES


As an example, in the results below examine the Bold items.

master                        0.4921875        68.25397           0
tempdb                      1.242188          92.20126           0
model                         0.4921875        55.55556           0
msdb                          0.7421875        56.31579           0
EVDAConfiguration      9.992188          17.26935           0
EVBADiscovery          299.9922          95.42372           0
EVMailboxVS2            79.99219          5.009034           0
EVJournalItems           79.99219          2.987963           0
EVCAConfiguration      9.992188          26.04574           0
EVBACompliance        299.9922          4.311799           0

 

As another example, in the results below examine the Bold items for tables that have over 1000 pages scanned.

DBCC SHOWCONTIG scanning 'tblIntDiscoveredItems' table...
Table: 'tblIntDiscoveredItems' (450100644); index ID: 1, database ID: 13
TABLE level scan performed.
- Pages Scanned...........................................: 54089
- Extents Scanned.........................................: 6802
- Extent Switches..........................................: 6994
- Avg. Pages per Extent.................................: 4.2
- Scan Density [Best Count:Actual Count].......: 38.89% [7:18]
- Logical Scan Fragmentation ....................: 72.22%
- Extent Scan Fragmentation .........................: 46.08%
- Avg. Bytes Free per Page............................: 1188.4
- Avg. Page Density (full)................................: 65.32%


The results from the "DBCC SQLPERF" query for LOGSPACE indicates that both transaction logs' space for the tempDB and the customer database EVBADiscovery are over 90%.  Effectively, no more items will be added to these transaction logs and thus must be truncated or allocated additional space.

The results from the first "DBCC SHOWCONTIG" query has a logical scan fragmentation of over 50% for the primary index on the tblIntDiscoveredItems table which causes the SQL server to require more resources to perform the index scan, thus slowing the processing of requests for data to be loaded into or accessed from the table.  This means only slightly better performance than a top to bottom, row by row scan to insert the new items into the table.  If the search has a large hit count, this could take numerous hours to complete.  If this issue the issue, run the Symantec recommended SQL maintenance on the customer database (see TECH74666 in related articles below for details):  
  

  • Shrink Databases - note that this is an optional action that, if done, must be done before the remaining actions.
  • Rebuild Indexes
  • Update Statistics

 
STEP 2:  Determine the cause of failure or delay (if not SQL related)

1. Verify there is not a licensing issue.  Examine the Event logs on the Accelerator server for Event ID: 201.  If you are receiving a 201 event, please contact your Symantec sales representative to obtain a correct license.

2. Determine the oldest search that never completed by running the following queries against the Compliance Accelerator or Discovery Accelerator Customer database.

SELECT TOP 10 c.Name AS 'Case/Dept Name', ints.Name AS 'Search Name', ints.SearchID, ints.RunDate
FROM tblIntSearches ints
INNER JOIN tblCase c ON ints.CaseID = c.CaseID
WHERE ints.StatusID NOT IN (1, 3, 4, 7, 8, 9, 10, 173)
ORDER BY ints.RunDate ASC

-- Status Codes
--      1 = Accepted
--      3 = Pending Acceptance
--      4 = Rejected
--      7 = In Review
--      8 = Abandon
--      9 = Pausing
--    10 = Pending Acceptance (Errors)
--  173 = No Hits
 
Paused Accelerator searches can prevent the release of worker threads for use by active searches, thus causing slow or no search processing.  The following SQL script can be run against each of the Accelerator customer databases to show 'Paused' searches:
 
SELECT c.Name AS 'Case/Dept Name', ints.Name AS 'Search Name', ints.SearchID, ints.RunDate
FROM tblIntSearches ints
INNER JOIN tblCase c ON ints.CaseID = c.CaseID
WHERE ints.StatusID IN (175, 176, 177, 178, 179)
ORDER BY ints.RunDate ASC
 
-- Status Codes
-- 175 = Paused
-- 176 = To Be Paused
-- 177 = Pausing
-- 178 = To Be Submitted
-- 179 = Resubmitting
 

Each paused search needs to be investigated and resolved. 

Note: The top row listed will be the oldest search.

3. Examine the oldest search using the information obtained from step 2.

4. In the CA or DA Client with the Department or Case and the oldest search returned in Step 2 selected, Inspect the Information column of each index volume for possible causes for the search to have never completed.
 
5. Resolve the issue that stopped the oldest search from completing.
  • If the resolution is not clear from the Information column, then run the following queries against the Accelerator Customer database.  Replace the XX with the correct Search ID number from the results of the queries in Step 2.
SELECT SearchID, [Name], NumIndexVolsSearch, NumIndexVolsTotal
FROM tblIntSearches WHERE SearchID = XX

SELECT SearchID, SearchVaultID, NumHits, ExpectedNumHits, CurrentHitCount
FROM tblSearchVault WHERE StatusID IN (12, 18) AND SearchID = XX 
 
  • The first query numbers should match.  This indicates that all volumes that are expected to be searched have been searched.
  • The second query's set of three hit numbers should match.  This indicates that the expected number of hits to be accepted into the review set table match the current and expected hit counts.

6. In most cases restarting the Enterprise Vault Accelerator Manager Service (EVAMS) will resolve the searches stuck in 'searching' status.

7.  If the searches are 'Failed' or do not complete after restarting EVAMS, they will need to be rejected or stopped and run again.
 
a. From the Search Details from step 2, navigate to the search, click the Stop Search button or Reject button on the search actions toolbar.
 
b. If the Reject button was clicked,
     1) The Browsing searches dialog box will be displayed with the question "Are you sure you want to reject the selected search?" with Reject or Cancel buttons present.
     2) Click the Reject button in the dialog box to reject the search.  Note: Clicking the Cancel button will allow the search to return to its previous state.
     3) The next panel displayed will be the Search criteria page.
     4) In the lower right of the panel, click the Save button to run the search again or click the Cancel button in the lower right of the panel.
     5) If the Cancel button was clicked, the Creating a search dialog box will be displayed.
     6) Click the Discard button in the dialog box to discard the search.
 
c. If the Stop button was clicked,
     1) The Browsing searches dialog box will be displayed with the question "Are you sure you want to abandon 1 searches?" with Abandon and Cancel buttons present.
     2) Click the Abandon button in the dialog box to stop the search.  Note: Clicking the Cancel button will allow the search to return to its previous state.
  
 
8. Run the SQL query again to verify the searches are no longer producing errors or stuck.


Step 3: Verify there isn't missing information in the configuration database

Verify that there is a configuration ID of "218" for each customer database. Run the following SQL Query on the Configuration database

SELECT * FROM tblConfiguration WHERE ConfigurationID = 218

There should be one for each customer and the Value must be 0.
If not, do the following.

1. Open the CA/DA client.

2. Navigate to  the Configuration tab, then to the Settings sub-tab.

3. Hold the <CTRL> key and click on "Configuration Settings" in the banner area to display hidden settings.

4. Expand the Vault Directory Synchronization folder.

5. Look for the Upgrade Vault v5 Searches option.  See Figure 1.

Figure 1
 

If the last modification column is blank, the System did not properly insert the value '0' into the configuration table.

6. Enable the option (select the Value check box), then disable the option (remove the check mark in the Value check box)

7. Select the Save button in the lower right of the panel, then close the CA/DA Client.

8. Re-start the Enterprise Vault Accelerator Manager Service (EVAMS) on the CA or DA server.

9. Run the SQL Query again to see if there is a configuration ID for each customer, if not, the Configuration database is corrupt and a new Configuration database will need to be created.  See Creating a new Configuration database below Step 10.

10. Test the search again, if it is still not working, create a new Configuration database
 

Creating a new Configuration database

1. Stop the Enterprise Vault Accelerator Manager Service (EVAMS) on the CA or DA server.

2. In the following files located in "C:\Program Files\Enterprise Vault Business Accelerator\" (for 32-bit OS) or "C:\Program Files (x86)\Enterprise Vault Business Accelerator" (for 64-bit OS) edit these files..

AcceleratorManager.exe.config
AcceleratorManagerConsole.exe.config
AcceleratorService.exe.config
ADSynchroniser.exe.config

change the following line:

<add key="State" value="Installed" />

To:

<add key="State" value="Uninstalled" />

3. Start the EVAMS.

4. Open the EVBAAdmin webpage and create a new configuration database.

5. Connect the existing Customer databases to the new Configuration database by 'Create new customer' and 'connect to the existing database'.
 
 



Legacy ID



291019


Article URL http://www.symantec.com/docs/TECH53400


Terms of use for this information are found in Legal Notices