Video Screencast Help

Archive Explorer shows "search has exeeded the allowed time"

Created: 13 Aug 2013 • Updated: 08 Oct 2013 | 18 comments
STHN's picture
This issue has been solved. See solution.

Hi folks!

in a customer environment (EV 10.0.2, they cannot upgrade) I saw this when selecting a folder in archive explorer.

archiveexplorersearch.jpg

Did anyone see this, also? Any solutions around?

I found this, but I'm not sure it's the same issue here.

Enterprise Vault (EV) Search applications show "Unable to list the contents of the whole archive
http://www.symantec.com/docs/TECH63922

Thanks in advance!

 

 

Operating Systems:

Comments 18 CommentsJump to latest comment

GabeV's picture

Hi Sebana,

That technote is talking about federated search which is basically searching accross multiple index volumes. For this archive, how many index volumes do you see in the VAC? Also, when you clicked in the AE folder, how long it took to display the error message? do you see any errors on the event viewer regarding indexing or web applications? Have you seen this issue just in this archive?

“Success is not final, failure is not fatal: it is the courage to continue that counts.”–Winston Churchill

STHN's picture

Hi GabeV,

they have 3 EV-Servers, all for separate exchange servers. No building blocks.

It's observed only on one EV server in every archive and every folder. Even if it contained almost nothing. I didn't count how long it took but 60 sec. is more or less the time we waited for results. There are no errors on the eventlogs on the EV server. I would like to take a DTrace but I'm not sure about the processes that are involved here. I would be grateful for any hint.

Thanks!

PMCS GmbH & Co. KG - Consulting und Support für Altiris/SEP/EV und andere Symantec Produkte.
Please take the time and mark this post as solution if it solved your problem - thanks!

Ben Watts's picture

Just a quick question, does this happen even if you open Archive Explorer via IE on the actual problem EV Server so that you are not going over the network to access it?

Where are the indexes located, are they local to all 3 EV Servers or are they remote to the problem EV Server and local to the 2 EV Servers that are ok?

 

Are you on 10.0.2 or wanting to upgrade to 10.0.2?

 

Pradeep_Papnai's picture

I think this problem is mostly around EMPTY folders, you should check tech note below.  

How to remove the empty folders from Archive Explorer in Enterprise Vault (EV) for Microsoft Exchange
http://www.symantec.com/docs/TECH47918

If you are searching agaist 32 bit index then use "IndexBroker, IndexServer, EVIndexAdminService" for Dtrace & Use "EVIndexQueryServer, EVIndexAdminService" for 64 bit index.
 

Ben Watts's picture

Hi Counselor,

 

I am not disagreeing with you but could you let me know WHY you think that as I am interested to know for future customers in the same situation.

 

STHN's picture

Hi Ben,

sorry, I missed your question. They are on 10.0.2 already and cannot upgrade to SP3 or 4.

PMCS GmbH & Co. KG - Consulting und Support für Altiris/SEP/EV und andere Symantec Produkte.
Please take the time and mark this post as solution if it solved your problem - thanks!

STHN's picture

Me too.

PMCS GmbH & Co. KG - Consulting und Support für Altiris/SEP/EV und andere Symantec Produkte.
Please take the time and mark this post as solution if it solved your problem - thanks!

Pradeep_Papnai's picture

This problem I saw long before in version 7 or 8, could not recall that incident at this moment. Lots of shortcut were moved to other folder & left empty folder hierarchy with few archived in archive. I would like to say that this problem "could be" same but not 100% sure. Let’s take d-trace to see if get any solid lead to diagnose this issue further.

A_J's picture

Hi,

 

You can Dtrace following process..

 

For 32bit.

IndexBroker

IndexServer

w3wp

For 64bit.

EVIndexQueryServer

EVIndexAdminService

W3wp.

For more information on how to take a Dtrace, please check the below article..

http://www.symantec.com/docs/TECH38122

Ben Watts's picture

Hi Sebaha,

 

Have you tried accessing the Archives for those affected users via a different server:-

e.g. http://evserver1/enterprisevault/archiveexplorerui.asp........ gives the error message above but if you try http://evserver2/enterprisevault/archiveexplorerui.asp........ do you get the same error or does it work?

If it works on the second EV Server I would look at the IIS settings on the affected EV Server to see if any of them are different compared to the working Servers.

 

If you could get a definite 'time' where we timeout, do it a couple of times to get an average, this will help us to round down which timeout is affecting you.

 

If you could get a definite 'time' where we timeout, do it a couple of times to get an average, this will help us to round down which timeout is affecting you.

 

 

STHN's picture

Hi Ben,

I cannot test the access from another EV-Server right now, but I will consider it on the next opportunity.

From the DTrace I can see it times out exactly after 120 seconds.

I see that it does a federated search because they havn't upgraded their indexes, yet. Upgrade from EV 9.0.2 to 10.0.2 has been around 10 July. Search results go back to 26 May for the user I used for troubleshooting.

Here is what I can see in the DTrace:

476    14:46:25.835     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::DoFederatedSearch: VolumeSet=54650.
477    14:46:25.835     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::DoFederatedSearch: Start async search on VolumeSet=54650, Type=64bit, Computer=vaultserver01.domain.fqdn
490    14:46:25.835     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::DoFederatedSearch: Successfully launched async search on VolumeSet=54650
491    14:46:25.835     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::DoFederatedSearch: VolumeSet=8298.
492    14:46:25.835     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::DoFederatedSearch: Start async search on VolumeSet=8298, Type=32bit, Computer=vaultserver01.domain.fqdn
994    14:46:32.060     [9108]    (EVIndexQueryServer)    <9792>    EV-M    {SearchService} Returning 100 results (665 matches) for source 19927A0AB4381F24B94E233A22C1B4BF91110000blahblah.domain.fqdn/54650

27949    14:48:25.098     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::WaitForResults - Elapsed: 120 (sec)|Active: 1 ;|Failed: 0|
27950    14:48:25.098     [19464]    (w3wp)    <10032>    EV:H    CFederatedSearchBroker::WaitForResults: TIMED-OUT: Number of timed-out searches: 1;|
27951    14:48:25.098     [19464]    (w3wp)    <10032>    EV:H    CFederatedSearchItem::CancelSearch: Seconds=0; AEID=19927A0AB4381F24B94E233A22C1B4BF91110000blahblah.domain.fqdn - VolSetId=8298
27952    14:48:25.098     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::WaitForResults: Federated Search Results Info:|Searched Volumes: 2|Successful: 1|Failed: 1|HR=Success  (0)|
27953    14:48:25.098     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::DoFederatedSearch Process XML Results.
27954    14:48:25.098     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::ProcessResults - One result set only. hr=Success  (0)
27955    14:48:25.098     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::ProcessResults Update RESULTS attribute values. hr=Success  (0)
27956    14:48:25.098     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::ProcessResults - Format xml results as Variant
27957    14:48:25.098     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::IStreamToSafeArray - HR=Success  (0)
27958    14:48:25.098     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::ProcessResults (exit) HR=Success  (0)
27959    14:48:25.098     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::DoFederatedSearch - Exit (hr=Success  (0))
27960    14:48:25.098     [19464]    (w3wp)    <10032>    EV:L    ~CFederatedSearchBroker() - Destructor
27961    14:48:25.098     [19464]    (w3wp)    <10032>    EV:L    ##### CFederatedSearchBroker::FederatedSearchThread - (exit). HR=Success  (0) #####|
27962    14:48:25.098     [19464]    (w3wp)    <10032>    EV:L    CAutoRefPtr - Release referenced pointer.
27963    14:48:25.098     [19464]    (w3wp)    <10032>    EV:M    |##### SComModuleLock - UNLOCK #####|
27964    14:48:25.098     [19464]    (w3wp)    <28648>    EV:M    #### CFederatedSearchBroker::Search - Search thread completed in 120 seconds  HR=Success  (0). ####

 

So I suggested them to upgrade one index to 64 bit and see if the issue is solved by that.

PMCS GmbH & Co. KG - Consulting und Support für Altiris/SEP/EV und andere Symantec Produkte.
Please take the time and mark this post as solution if it solved your problem - thanks!

Ben Watts's picture

I still believe this timeout value is coming from IIS somewhere, just do not have access to my labs to confirm where we set limits or have them applied to us.

I know there is a timeout limit on the Default Website level of 120 seconds on the connection timeout setting.

You could try raising this as a test (I am not 100% this is where the limit we are failing but it is the only place I can think of at this time that a limit of 120 seconds is applied)  to see if an increased time allows for searches to complete, once that is working you could attempt to find the root cause of the time difference between this EV Server and the others.

I would suggest trying this even if the setting is the same on all EV Servers, the connection between this EV Server and the Indexes could be hitting a bottleneck somewhere whilst the others are fine.

Default Website - Advanced Settings - Connection Limits

 

I will have a proper look tomorrow once in and let you know if I find anything different.

 

Ben Watts's picture

Just to update you, I tried altering that setting in my lab but it had no effect at all so I do not think that is where we are seeing our timeouts.

 

Like you have seen yourself it appears to be hanging on one volume on this search, below we see the search of the 64bit volume complete happily with 665 results:-
994    14:46:32.060     [9108]    (EVIndexQueryServer)    <9792>    EV-M    {SearchService} Returning 100 results (665 matches) for source 19927A0AB4381F24B94E233A22C1B4BF91110000blahblah.domain.fqdn/54650

Bu it appears to be hanging on another volume as the below line states that we have been successful on one but failed on the other:-

27952    14:48:25.098     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::WaitForResults: Federated Search Results Info:|Searched Volumes: 2|Successful: 1|Failed: 1|HR=Success  (0)|
 

and this line shows that we start searching the 32bit volume but then time out 2mins later whilst we are still trying to search it:-

492    14:46:25.835     [19464]    (w3wp)    <10032>    EV:L    CFederatedSearchBroker::DoFederatedSearch: Start async search on VolumeSet=8298, Type=32bit, Computer=vaultserver01.domain.fqdn

27951    14:48:25.098     [19464]    (w3wp)    <10032>    EV:H    CFederatedSearchItem::CancelSearch: Seconds=0; AEID=19927A0AB4381F24B94E233A22C1B4BF91110000blahblah.domain.fqdn - VolSetId=8298

 

Think you may well have found the cause yourself.

Let us know how the upgraded Volume searches.

STHN's picture

Thanks for sorting that out. I saw that it's failing on the 32 bit index. I wonder why.

Accessing the archive explorer through another EV-Server (vaultserver02) shows the same issue. I think that is because the index search is still handled by vaultserver01.

Is there any point in performing a verify index on the 32 bit index to see if there are any issues with that? I'm not expecting to find any.

Thanks in advance!

PMCS GmbH & Co. KG - Consulting und Support für Altiris/SEP/EV und andere Symantec Produkte.
Please take the time and mark this post as solution if it solved your problem - thanks!

Ben Watts's picture

No I dont think there is, I think upgrade it to 64bit and see if we are still seeing the same issue.

A verify only confirms that the index is 'good', if it isn't already marked as failed then I wouldnt expect a verify to help too much.

SOLUTION
STHN's picture

I don't expect any helpful results of the verify, either. But maybe the process of verifying is particularly slow or something like that.

I will post the results when I get them. Thanks.

PMCS GmbH & Co. KG - Consulting und Support für Altiris/SEP/EV und andere Symantec Produkte.
Please take the time and mark this post as solution if it solved your problem - thanks!

DKumar's picture

Hi ,

I had Similar issue , after migrating to 64-bit index now are able to see as below

 

Capture.JPG

 

With exclamatary mark ( After page number).

STHN's picture

Upgrading to 64-bit indexes solved the issue.

Thank you Ben for helping resolve it. Thanks anyone else for giving advice!

PMCS GmbH & Co. KG - Consulting und Support für Altiris/SEP/EV und andere Symantec Produkte.
Please take the time and mark this post as solution if it solved your problem - thanks!