Video Screencast Help

Yesterday we had EV server queue pile up issue for all EV servers,5. As a workaround we performed cluster SQL database failover and failed back and the issue got fixed. Again we had same issue EV issue today.

Created: 19 Oct 2012 • Updated: 15 Nov 2012 | 4 comments
Mohankumar K's picture
This issue has been solved. See solution.

 

Yesterday we had EV server queue pile up issue for all EV servers, As we checked the server and confirmed the below information.

 

1.      Confirmed that the MSMQ A5 queues are piled up.

2.      Verified all the EV services are running fine.

3.      Checked the server event log and confirmed that there was SQL database instance connectivity issue. Please find below event logs

 

Log Name:      Symantec Enterprise Vault

Source:        Enterprise Vault

Date:          10/19/2012 5:03:51 PM

Event ID:      13397

Task Category: Storage Online

Level:         Warning

Keywords:      Classic

User:          N/A

Computer:      ev03.stf.nus.edu.sg

Description:

The connection 'Provider=SQLOLEDB;Server=evdb01;Database=EVStudentVaultStore1;Trusted_Connection=Yes' was lost and the system is waiting to reconnect (Thread Id: 11336)

 

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp?EvtID=13397

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="Enterprise Vault " />

    <EventID Qualifiers="32772">13397</EventID>

    <Level>3</Level>

    <Task>47</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2012-10-19T09:03:51.000000000Z" />

    <EventRecordID>1974692</EventRecordID>

    <Channel>Symantec Enterprise Vault</Channel>

    <Computer>ev03.stf.nus.edu.sg</Computer>

    <Security />

  </System>

  <EventData>

    <Data>Provider=SQLOLEDB;Server=evdb01;Database=EVStudentVaultStore1;Trusted_Connection=Yes</Data>

    <Data>11336</Data>

  </EventData>

</Event>

 

 

Log Name:      Symantec Enterprise Vault

Source:        Enterprise Vault

Date:          10/19/2012 5:04:06 PM

Event ID:      13395

Task Category: Directory Service

Level:         Warning

Keywords:      Classic

User:          N/A

Computer:      ev03.stf.nus.edu.sg

Description:

The connection 'Provider=SQLOLEDB;Server=evdb01;Database=EnterpriseVaultDirectory;Trusted_Connection=yes' was lost and the system failed to reconnect (Thread Id: 5492)

 

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp?EvtID=13395

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="Enterprise Vault " />

    <EventID Qualifiers="32772">13395</EventID>

    <Level>3</Level>

    <Task>21</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2012-10-19T09:04:06.000000000Z" />

    <EventRecordID>1974697</EventRecordID>

    <Channel>Symantec Enterprise Vault</Channel>

    <Computer>ev03.stf.nus.edu.sg</Computer>

    <Security />

  </System>

  <EventData>

    <Data>Provider=SQLOLEDB;Server=evdb01;Database=EnterpriseVaultDirectory;Trusted_Connection=yes</Data>

    <Data>5492</Data>

  </EventData>

</Event>

 

 

Log Name:      Symantec Enterprise Vault

Source:        Enterprise Vault

Date:          10/19/2012 5:04:22 PM

Event ID:      6578

Task Category: Migrator Server

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      ev03.stf.nus.edu.sg

Description:

Abnormal error occurred

 

Object:    CSSASCache

Reference: RE(1)/fe

 

For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp?EvtID=6578

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="Enterprise Vault " />

    <EventID Qualifiers="49156">6578</EventID>

    <Level>2</Level>

    <Task>29</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2012-10-19T09:04:22.000000000Z" />

    <EventRecordID>1974726</EventRecordID>

    <Channel>Symantec Enterprise Vault</Channel>

    <Computer>ev03.stf.nus.edu.sg</Computer>

    <Security />

  </System>

  <EventData>

    <Data>CSSASCache</Data>

    <Data>RE(1)/fe</Data>

    <Binary>5468652053514C2064617461626173652073657276657220666F72202332206973206E6F7420617661696C61626C653A202020202020436865636B2074686174207468652053514C205365727665722069732076616C696420616E642069732072756E6E696E67202020202020202020496E7465726E616C207265666572656E6365282338292020204465736372697074696F6E3A202020202020233120202020204164646974696F6E616C204D6963726F736F667420737570706C69656420696E666F726D6174696F6E3A2020202020536F757263653A2020202020202023332020204E756D6265723A20202020202020233420202053514C2053746174653A2020202023352020204E6174697665204572726F723A20233620202048524553554C54202023372020202020202020283078383030343334346129</Binary>

  </EventData>

</Event

 

4.      As we verified EV database cluster services and confirmed that the cluster resource are up.

5.      As a workaround we performed cluster SQL database failover and failed back and the issue got fixed. Again we had same issue EV issue today.

 

 

Kindly find the below update.

 

n  What type of mail archiving Domino or Exchange used?

Exchange 2007

n  Version of Enterprise vault used??

EV 8.0

n  EV server platform?

All EV Server are running windows 2008 R2

Comments 4 CommentsJump to latest comment

GertjanA's picture

I assume you are on EV8SP4? lower versions are not supported on W2008R2.

Are you using DNS Alias for SQL server (evdb01)

Can you check: http://www.symantec.com/docs/TECH66826

Can you confirm there are no issues on the SQL-cluster (like quorum getting lost etc). Backup's maybe?

 

Thank you, Gertjan, MCSE, MCITP,MCTS, SCS, STS
Company: www.t2.nl

www.quadrotech-it.com

www.symantec.com/vision

JesusWept3's picture

another thing, i take it restarting the Admin service didn't resolve the issue either?
You haven't limited the amount of sql connections that could be made by EV have you?

JesusWept3's picture

Well Gertjan started you off nicely i think, check the SQL Servers for any intermittment communication issues or cluster issues, so it could be that the SQL Servers are experiencing a "blip" in communications , and EV just isn't handling the disconnect well.

So check with the SQL Server, look at the event logs and any other sql logs that may be showing it disconnecting from the network, or any long procedures that may be disconnecting the EV Services

Also check out the technote he stated: http://www.symantec.com/docs/TECH66826

And from my side I asked whether restarting the admin service would have resolved the issue (without having to resort to failing over SQL Server) and whether you have limited any connections on the SQL Server itself, could it be that EV is exhausting the amount of connections given to it?

Also when this happens, do you see anything abnormal on the SQL Servers? 100% CPU usage, higher than normal disk usage etc

If a restart of the EV Services does not work but failing the SQL Server *does* work, then you would have to assume its an issue with the SQL Service or the node at that particular time.

Also look to see if anything has changed in the environment
have you recently updated EV? updated SQL? changed Storage for the SQL Databases?
Enabled a new bunch of users? started any big PST Migrations? Vault Cache builds etc?
 

If it becomes a much larger issue then your best bet is to get DTraces of the directoryService on the EV Server, get the application, system and Enterprise Vault logs from the Event Viewer, and also Application and System logs from the SQL Server as well as any other Error logs and a snapshot of the activity monitor...then open a case with symantec to help troubleshoot the issue

SOLUTION