Video Screencast Help

Compliance Acclerator Searches goes in hung state.

Created: 22 May 2013 • Updated: 28 Oct 2013 | 10 comments
This issue has been solved. See solution.

Hi Guys,

I have come across a small issue.

My compliance team is trying to perofrm a search on each dept having 5 searches in each dept and 1000 hotwords in it.

CA server version 9.0.2
CA Client Version 9.0.2
No of Dept           8
No of searches in each dept at a same time 5

So when the compliance guys try to perfrom a search in the system the searches go in never ending loop and does not complete at all.

Is there a limit on number of hotwords used in a search criteria, secondly is there a way i can fine tune the server in order that the searches compelte.

Operating Systems:

Comments 10 CommentsJump to latest comment

Kenneth Adams's picture

Hello, Cdee;

If you've not already done so, please go through TECH56172, "Recommended steps to optimize performance on Enterprise Vault (EV), Compliance Accelerator (CA), Discovery Accelerator (DA), and SQL Servers in an EV environment", available at http://www.symantec.com/docs/TECH56172.  This is for all versions of EV, CA, and DA, but there are notes that some settings are not appropriate for 64-bit OS, so note those as appropriate.

Technically, there is no programmed limit to the number of hot words or hot phrases a search criteria can hold.  There is, however, a pratical limit based on the available resources of the DA, EV, and SQL servers.  Wild cards in the hotwords add a lot of complexity to the criteria, too, to the use of wild cards should be kept to the mimimun necessary.

Even after implementing the optimizations recommended in TECH56172, you may still have to break the searches into smaller sets of criteria.  Each index volume is searched by an instance of the IndexServer process.  Each IndexServer process is allocated a maximum of 2 GB  of RAM for its use.  Into this 2 GB of RAM must fit the search criteria, the index volume contents, and the search hits.  That's a lot to fit into 2 GB of RAM, which also tends to become unstable after reaching 1.5 GB used.

If the physical size on disk of your index volumes is over 3 GB, you may need to implement the AVSMaxLoc registry value (set to 500 million decimal) and rebuild the indexes of the archives being searched.  Hopefully, you don't have to do this as the rebuild would prevent any searches against the archives as they are rebuilding.

I hope you find this information helpful.

 

Ken Adams

Backline Support for CA, DA, ACE, UCE, PSTD, ARMS, EVDC
US Support Region

Scott _Hastings's picture

At the end of the day, I think you'll always have searches in Searching, Accepting, and, Pending Acceptance (Errors).

 

This is a really problem because searches that are "stuck" don't go into review sets on a timely basis.

 

I would do 2 things.

 

1) Run the following query against the CA Customer database to check for these items"

SELECT tc.CaseID, tc.Name 'Case/Dept Name'

, tis.SearchID, tis.Name 'Search Name', tis.StatusID, tis.RunDate

, ts1.Name 'Search Status'

FROM tblIntSearches tis

JOIN tblCase tc ON tis.CaseID = tc.CaseID

JOIN tblStatus ts1 ON tis.StatusID = ts1.StatusID

WHERE tis.StatusID IN (2, 6, 10)

 

2) Create a Task that restarts the Accelerator Service each day. This will clear any searches that are Searching or Accepting. Pending Acceptance w Errors need to be resubmitted.

Kenneth Adams's picture

While Scott's SQL query will report on all searches in a 'Searching', 'Accepting...', or 'Pending Acceptance (Errors)' status, just restarting the Enterprise Vault Accelerator Manager Service (EVAMS) nightly may not cause stuck searches to complete.  It may, but it also may not depending on where the issue causing the stuck condition resides.

In your CA environment (just like in any other CA or DA 9 environment), there are resources on the EV Indexing Server that may cause a search to become hung.  Just restarting EVAMS won't clear those resources.  Restarting the EV Indexing Service may clear them, or a reboot of the EV Indexing Service server may be needed.  It really depends on the actual root cause of the 'stuck' condition.

There are also other issues in the CA environment that could cause searches to become 'stuck' that restarting EVAMS won't clear.  For example. if an archive has had an adjustment made to its index volumes - say the AVSMaxLoc registry value was set to a higher value than previously set and the archive's index rebuilt to result in fewer index volumes - EV could have a different number of Index Volumes for an archive than CA has.  In such cases, a CA search would be looking for more index volumes to search than actually exists.  Such a condition would result in CA showing all index volumes as being finished, but the search never completes because it is awaiting the return from non-existent index volumes.  (Note that this condition requires stopping EVAMS, truncating the tblIndexVolumeSet table in the CA Customer database, starting EVAMS and letting the archive synchronization process repopulate the tblIndexVolumeSet table with the current index volumes in EV.)

If you've optimized your environment per TECH56172 as I recommended in my first post AND you're still having CA searches become stuck, please open a support case so we can look into why those searches are becoming stuck.

 

Ken Adams

Backline Support for CA, DA, ACE, UCE, PSTD, ARMS, EVDC
US Support Region

SOLUTION
EV_Ajay's picture

Hi,

I totaly agree with Ken explanation. He provided very good information.

You can refer following articles which will help you :

Guidelines on conducting effective searches
http://www.symantec.com/docs/HOWTO58574

About searching with Compliance Accelerator
http://www.symantec.com/docs/HOWTO31528

 

Thanks,

Ajay

Scott _Hastings's picture

Kenneth, I optimized my environment as per TECH56172 and still had trouble. I was simply sharing my  experience and how I was able to resolve it. I certainly didn't intend to contradict you.

Scott

 

Kenneth Adams's picture

Hi, Scott;

I understand you were not trying to contradict me.  I just wanted Cdee to understand that just restarting EVAMS may not allow the hung searches to complete.  In your environment it does, but you should not have to restart EVAMS nightly just to get searches to complete unless your SQL Server is loosing network connectivity with your CA server.  We must have continuous connectivity between the CA and SQL servers in order to update the search with hits and status changes.  If that connection breaks, it would throw Event ID 158 errors in the CA server's Symantec Enterprise Vault Event Log every 10 minutes for each CA Customer (including Custodian Manager) that you have running.

Ken Adams

Backline Support for CA, DA, ACE, UCE, PSTD, ARMS, EVDC
US Support Region

Scott _Hastings's picture

Right Kenneth.. I have a hard time getting SQL DBA's to do proper maintenance, and SQL Servers are usually a bit overloaded. Sometime back  (at a very, very large company) you guys advised me to disable TCP Chimney and TCPIP Offload Engine.

The customer was very resistant to this, but when I finally convinced them, the issue was immediately resolved.

 

I was just tossing in a little "real world"

 

Thanks!

 

Kenneth Adams's picture

Ah.  Another example of following our advice to see the issue resolved.  ;-)

Thank you for sharing that information, Scott.

Ken Adams

Backline Support for CA, DA, ACE, UCE, PSTD, ARMS, EVDC
US Support Region

EV_Ajay's picture

Hi,

Do you have any updates on this thread? Do you need more assistance regarding this topic? Please mark the post that best solves your problem as the answer to this thread.

 

Thanks,

Ajay

Cdee's picture

Thanks for the post issue has been resolved.

Sr Messaging Engg.