Video Screencast Help
Scheduled Maintenance: Symantec Connect is scheduled to be down Saturday, April 19 from 10am to 2pm Pacific Standard Time (GMT: 5pm to 9pm) for server migration and upgrades.
Please accept our apologies in advance for any inconvenience this might cause.

EV9 poor (journal) archiving performance

Created: 08 Feb 2013 • Updated: 13 Feb 2013 | 23 comments
This issue has been solved. See solution.

Hi

Environment in EV9.0.2 and Exchange 2003.

We have ~2300 users being journaled to a single mailbox, this mailbox is targeted by journaling archiving in EV - single journaling task.

I ran the numbers over 3 months and we're getting an average of 6891 items being archived per hour by this task, obviously a way off the best practise numbers quoted in the EV9 performance guide (20k-60k). Servers are beefy, plenty of RAM, CPU, storage are (new) Centeras with collections enabled.

The mailbox targeted by the journal task always has a big backlog, we can't seem to archive quickly enough.

Questions:

1. Is it a good idea to up the "number of concurrent connections to exchange server" from 5 to 10 as per the performance guide? What about the "Maximum number of items per target per pass"  (currently set to 1000)

2. Is it beneficial to set up, say, an additional 4 journaling tasks, and split the 2300 users so they journal into 4 mailboxes instead of 1? Any downsides to doing this? I can't find any recommendation in the performance guide around number of journal tasks vs users.

3. Can you have separate journal tasks all archiving to the same archive, or are new archives needed for the new journal tasks?

Thanks

 

Comments 23 CommentsJump to latest comment

JesusWept3's picture

concurrent connections isnt going to make much of a difference
neither will messages per pass really

You can have different tasks and targets going to the same archive

You can split up the mailboxes if you want, that should give you better performance
But honestly, something is desperately wrong here, the numbers are really bad

I know we had a similar situation where the throughput was worthless, and we found that the EV Monitoring Agent had a significant handle leak and was consuming a LOT of resources, we killed that and then journaling flew

So somethings you may want to check
Handles and memory counts of all processes on the machine
Have a look at disk queue lengths on the EV Server
Have a look at disk queue lengths on the exchange server
Have a look at disk queue lengths on the SQL Server

Also on the SQL Server, check and see if there are any processes being blocked or deadlocked, and that it isnt SQL causing the slow down.

And if you're on Exchange 2003, it may be worth using Outlook 2003 instead of 2007 as 2007 is needed for Exchange 2010, its recommended to use outlook 2003 for Exchange 2003 and 2007 due to the fact that MAPI is more performant on Outlook 2003

Also do a cursory check of Application and System Logs on EV/Exchange/SQL to make sure we're not getting any issues from hardware.

And its always a good idea to look an make sure that the NIC's and Network are in good shape, that you're not dropping packets or anything like that

Also you can do a dtrace of JournalTask and StorageArchive and see if there are any real slow deltas
 

goatboy's picture

Already on Outlook 2003.

Monitoring agent is a big consumer of memory, and has ~15K handles.

Is it safe to kill? What is it and does it need to be disabled permanently?

JesusWept3's picture

thats the monitoring agent, it writes all the statistics to the EV Monitoring Database that you need to view some of the SQL Reports or up to date information in the Enterprise Vault Operations Manager (EVOM)

Also what are your backlogs like in journaling at the moment?

JesusWept3's picture

oh and end task it, EV will start it back up

 

goatboy's picture

is 15K a significant number of handles? Will try this anyway...

goatboy's picture

 238552 backlogged items in the journal mailbox. 

I ran the Dtrace, what should I look for? I am seeing these sorts of numbers, not sure what is (ab)normal:

800762 13:15:04.259  [4172] (JournalTask) <4160> EV:H CArchivingAgent::ProcessItemInternal Time Taken to Process Item - C026EF85FDA3FAC6F343BBF6D443F171: 1.200357 secs

844408 13:15:22.963  [5100] (StorageArchive) <496> EV:H SavesetReceive Time taken to store item - C03D7F08D0CD7C3E1F8FDCA4083829F1: 0.241217 secs

840189 13:15:22.556  [3736] (StorageArchive) <5052> EV:H SavesetReceive Time taken to store item - C03D788ECB1A023B41C8B119C42E79B1: 0.185905 secs

789694 13:15:02.947  [4172] (JournalTask) <5580> EV:M CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter]: [0.010175] seconds

825483 13:15:08.072  [1200] (JournalTask) <8968> EV:M CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter]: [5.138734] seconds

812493 13:15:05.963  [4172] (JournalTask) <3636> EV:M CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter]: [1.715702] seconds

796078 13:15:03.916  [1200] (JournalTask) <4632> EV:M CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter]: [22.294424] seconds

For the 22 second external filter event, it's taking 22 seconds to perform a small part of the whole process to archive a single email? That is painful. A few of these and that would explain such poor archiving performance?

Our custom filtering does a DL lookup to tag messages with custom retentions based on DL membership.

Thanks

 
JesusWept3's picture

Wow,
Well kill the MonitoringAgent then do another dtrace to see if that helps any with the custom filter
but you have hit the nail right on the head, if it doesn't get any better then you may need to call support to find out why its taking so long

Is that your own custom filter or is it something like orchestria or another third party piece that is applying the filters?

goatboy's picture

It's our own filter:

<RULE NAME="10 Years RECIPIENT" RETENTION="10 Years">
      <RECIPIENTS>
           <DL>Retention 10 Years</DL>
      </RECIPIENTS>
</RULE>

<RULE NAME="10 Years AUTHOR" RETENTION="10 Years">
      <AUTHOR>
           <DL>Retention 10 Years</DL>
      </AUTHOR>
</RULE>

There's an additional 18 rules covering other DLs (9 AUTHOR, 9 RECIPIENT). Then 2 more rules at the end that do SUBJECT matching.

6891 items archived on average per hour, that's 116 per minute, and ~2 per second. I'll get some numbers around how long the custom filter is taking on average shortly.

JesusWept3's picture

Thing about averages, you have to account for weekends, holidays , downtime due to backup modes etc, so the number can change dramatically

 

But still, 22 seconds for one of those messages is *really* bad

TonySterling's picture

Do you have a lot of Internal SMTP Domains?

This issue is resolve in EV 9 SP4:

Environments with a very large number of internal SMTP domains experienced delays in journal archiving [Ref 13024, E2678932]

In some environments, delays occurred during journal archiving. This happened when custom filtering was enabled, and a very large number of internal SMTP domains was specified in the InternalSMTPDomains string value under the following registry key:

HKEY_LOCAL_MACHINE
\SOFTWARE
 \KVS
  \Enterprise Vault
   \Agents

This has been fixed.

goatboy's picture

InternalSMTPDomains - this value is not present.

Backup mode - generally about 1.5 hrs a night for 5 nights when incrementals run, longer for fulls.

I pulled the "Time Taken to Process External Filter" values from an ~10 minutes DTRACE before and after killing the monitoring agent process.

Before - 3530 entries, average time 1.42 seconds

After (this DTRACE was shorter) - 2466 entries, 1.38 seconds.

Here are the top 20 numbers.

Before:

 

CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 103  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 82  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter 33  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter 33  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter 33  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter 32  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter 32  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 29  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 29  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 27  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 27  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 26  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 24  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 24  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 24  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 24  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 24  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 23  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 22  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 22  seconds

 

After:

 

CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 165  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 163  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 106  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 72  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 30  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 28  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 27  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 26  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 26  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter 24  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 24  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter 24  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter 24  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter 24  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter 22  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 22  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter 22  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [KVS.Accelerator.PlugIn.Filter 21  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 21  seconds
CEVFilterController::ProcessAllFilters - Time Taken to Process External Filter [EnterpriseVault.CustomFilter 21  seconds
goatboy's picture

This looks like the issue we're having: http://www.symantec.com/business/support/index?page=content&id=TECH191331

Problem

When using several Custom Filter rules based upon large distribution lists, the Exchange Journaling task is slow to process items.

But the fix is for 9.0.3 only! sad

JesusWept3's picture

Then really your best bet is to to upgrade to 9 SP4+

Nathan Clark 2's picture

just a fyi the handle consumption is significant and should be fixed in the next available release.

 

goatboy's picture

So we had some nested DLs in the DLs that were being looked up by the custom filter, that wasn't helping things. But removing them didn't help throughout that much. Created 4 new journal mailboxes and finally the backlog is going down, albeit slowly. Question, is Exchange server hardware a big factor in journal archiving rates? The Exchange server hosting the journal mailboxes is pretty old.

JesusWept3's picture

It all really depends on how old the hardware is, and how many items per day you are journaling
The biggest issue you are going to face is disk queue lengths

But really the issue you are seeing is all AD lookups and thats where the slow down is coming from
Considering you're on 9 SP2 and there are significant number of custom filter improvements, your best bet really is to upgrade, may have made all the journal splits and multiple targets unnecessary

SOLUTION
goatboy's picture

Thanks. Are you talking disk queue lengths on EV server or Exchange? If Exchange, the disks containing the information stores or smtp queues or both? The numbers are good on the new journal mailboxes so I am pleased with the change. Agree with upgrade, except we have CA and DA version dependencies, so upgrade will need to cover these as well...

JesusWept3's picture

Avg Disk Queue Length on the Exchange servers that host the Information stores
If it gets too backed up then you will see a rapid decrease in performance

Typically you will see this when users getting the yellow balloon popups on their desktop stating that items are attempting to be downloaded, and you could see that in outlook when connected to the journal mailbox on the EV Server.

Anywho! EV9 SP4 server is compatible with all CA and DA versions of 9
So like DA9 SP2 can work with EV9 SP4, or EV9 SP2 with DA 9 SP4
You just have to ensure that the EV API installed on those servers (or the EV Binaries) match the SP version of the main EV Server

So if you have EV9 SP4 server and DA9 SP2, you should install EV 9 SP4 binaries on the DA server so that the EV binaries match the server

You can check the compatibility chart here on page 87
http://www.symantec.com/business/support/index?page=content&id=TECH38537

TonySterling's picture

Minor correction, the version of DA must be equal or greater than the version of EV. From the compatibility doc.

If the major version of Compliance Accelerator or Discovery Accelerator is the same as the version of Enterprise Vault, the minor version (Service Pack) of Compliance Accelerator or Discovery Accelerator must be the same as, or later than, the minor version of Enterprise Vault.

For example:

You can run Compliance Accelerator 10.0 with Enterprise Vault 9.0.2 servers, but not Compliance Accelerator 9.0.2 with Enterprise Vault 10.0.

You can run Discovery Accelerator 9.0.2 with Enterprise Vault 9.0.1 servers, but not Discovery Accelerator 9.0.1 with Enterprise Vault 9.0.2.

JesusWept3's picture

Huh, where in the doc does it say that? It contradicts that chart then :/

TonySterling's picture

It is in the intro and yes it does contradict the chart!  :)

goatboy's picture

Thanks Tony, I was just about to clarify that. And thanks Jesus, your advice is much appreciated! I'll run some disk queue stats, but for now, EV performance is acceptable.

goatboy's picture

Yesterday's stats - 640254 = 26677 per hour! Finally in a range (albeit the lower range) documented in the EV9 performance guide.