Video Screencast Help
Search Video Help Close Back
to help
Not able to make it to Vision this year? Get a sampling in the Best of Vision on Demand group.

EV synchrinization & archiving very slow

Updated: 04 Apr 2012 | 16 comments
Leonardo Fernandez 2's picture
0 0 Votes
Login to vote
This issue has been solved. See solution.

Running EV 9.0.2 on Windows 2003 SP2 for Exchange 2007. I have about 2500 mailboxes. Synchronization is scheduled twice daily at 11am and 11pm. Sync kicked off at 11 am today, its now 12 noon and there are still 2300+ entries in the A7 msmq queue.

My EV usually processes 1000 mailboxes nightly, but lately it has only procesed a handfull.

Does sync slow down or completely halt archiving?

I kicked off a "run now" against a mailbox right before sync started and the a3 queue still shows 1 entry.

I did receive two 3196 errors but that's just for two users which i think were deleted before synchronization so ev did not know about them.

Comments

JesusWept2's picture
12
Jul
2011
0 Votes 0
Login to vote

What are teh 3196 errors? I

What are teh 3196 errors?

I saw something similar where EV wouldn't do anything for a task because it had mailboxes that no longer existed lingering in the A3 queue, as from an archiving stand point that would be the highest priroity, so anything in A3 should take presedence over anything else.

So once i found out the user didn't exist, purged the queue and then it ran, but in that instance (EV9 SP1) it would actually fail the task, the task wouldn't remain running.

What i would suggest if you can is just to Dtrace the Archive Task for 30 minutes and upload it if you can.

But you may have bigger issues than this, you may want to check your SQL Server for any locking or blocking, as if SQL gets bottlenecked, then EV will remain unresponsive, and theres an olidish issue that still lingers regarding the ArchiveFolder table, and if it meets a ceterain looping condition then it brings EV to its knees

Leonardo Fernandez 2's picture
12
Jul
2011
0 Votes 0
Login to vote

I'll check with my sql

I'll check with my sql engineer. maybe he can run some diags o the sql server. I'll be back with I find. thanks.

Rob Wilcox's picture
12
Jul
2011
0 Votes 0
Login to vote

Can you post the event's

Can you post the event's related to the 3196 event ID?

 

Thanks
Rob Wilcox
Enterprise Vault Engineering
Certification Team
http://www.symantec.com/connect/blogs/rob-wilcox
Leonardo Fernandez 2's picture
12
Jul
2011
0 Votes 0
Login to vote

It was just 2 events and it

It was just 2 events and it was for two mailboxes that were removed. The events are not related as I've been having this probem for about two weeks wthout those events.

Rob Wilcox's picture
12
Jul
2011
0 Votes 0
Login to vote

hmm okay.

hmm okay.

 

Thanks
Rob Wilcox
Enterprise Vault Engineering
Certification Team
http://www.symantec.com/connect/blogs/rob-wilcox
Leonardo Fernandez 2's picture
13
Jul
2011
0 Votes 0
Login to vote

I rescheduled mailbox

I rescheduled mailbox synchronization to run at 9 am, once a day only. The A7 queue was empty last night, yet the Items Archived per Hour report shows only 12 archives processed between midnight and 8 am. EV archives 1000 items in a mailbox before moving on to the next one. it have processed well over 1000 mailboxes alst night. i used to process that many when i was archivign 5000 items per mailbox.

Not only are there no errors in the Symantec Enterprise Vault event log, but there are no entries period between midnight, info event 3156 (Start to process the Exchange system) and 9:05 am, warning event 41261

It is more than 2 days since backup or replication was detected for one or more open partitions.
Open partitions not scanned: 1
Total open partitions: 1
List of partitions:

but, the a5 queue has over 2000 entries in it.

Leonardo Fernandez 2's picture
13
Jul
2011
0 Votes 0
Login to vote

I meant to say EV should have

I meant to say EV should have processed at least 1000 archives last night, but did not..

MarkBarefoot's picture
13
Jul
2011
0 Votes 0
Login to vote

I would suggest logging a

I would suggest logging a case with Support as there are a lot of things that need looking at, something that cannot easily been done via forum posts.

Thanks,

 

Mark

Supportability Analysis Engineer for Enterprise Vault products.

 

**REMEMBER TO VOTE IF THIS HELPS AND MARK IT AS RESOLVED TOO IF IT IS!!**

Leonardo Fernandez 2's picture
18
Jul
2011
0 Votes 0
Login to vote

I've been working with

I've been working with Symantec Support since last week. We still can't figure out what is wrong. Have check EV in and out and it all looks ok. So far the only possible problem is the hard drives on the sql server. Tney are heavily fragmented. some was suppsoed to defrag them this past weekend. Still waiting for an update on that. Other than the we can't figure out why EV is archiving and synchronzing so slowly.

Leonardo Fernandez 2's picture
25
Jul
2011
0 Votes 0
Login to vote

Well, after spending a little

Well, after spending a little over a week working with symantec it turns out that the performance issues must be related to hardware. We are going to defrag the 900+ GB Index drive this weekend as well as the 1 TB drive that contains the active partition. Hopefully this will increase performance until we can move the EV partitions to something faster than a sata drive.

If anyone can think of anything else I can do to increase performance, please let me know.

Thank you all for you help.

JesusWept2's picture
25
Jul
2011
0 Votes 0
Login to vote

If you're talking about doing

If you're talking about doing defrags on the SQL drives, then great, be sure to do all your maintenance plans too (rebuilding SQL indexes and the like) however defrag of your EV indexes and EV vault store data wouldn't cause any issues or slow downs with synchronizations.

Leonardo Fernandez 2's picture
25
Jul
2011
0 Votes 0
Login to vote

We've already defragged the

We've already defragged the sql drives and the sql databases are properly maintained,  but that did not increase performance. Tthe symantec tech I spoke too said defragging the drives that the vault index and partition is on might help. What else can I do. There are no errors in EV. We ran dtaces, recreated the archive task and even ran a DBCC SHOWCONTIG command in sql and all looked ok.

JesusWept2's picture
25
Jul
2011
0 Votes 0
Login to vote

You can try it, but it won't

You can try it, but it won't help, i mean, i'll give you a years worth of my salary if it works.

So first question is, do yu still have the case open with Symantec or did they close it?
Did they ever look in to the SQL activity monitor when its at its slowest to determine whether there are any long running queries or anything locking or blocking other sql activities?
Did they get a dtrace of ArchiveTask etc to look at whether it was connectivity between exchange and the EV Server?
Did they check the CPU and Memory usage of SQL Server to make sure that the CPU is not gong 100% and simply not responding to requests?
Did they check the Exchange server logs to see if there were any errors that exchange was throwing out?
Did they get any network monitoring, either from wireshark or AppCritical (Apparanet) trace to see if there are any network issues between the EV Server, SQL Server and Exchange server?

I mean it simply comes down to one of a few things 
 - SQL Server - if SQL server is too busy to respond, or a massive query or a query stuck in an almost infinite loop is encountered, then simply you will never get much of a response from anything, if SQL is hosed it will affect the entire environment

 - Network - Sometimes it can be things like a network binding where your server is using the 10mb cable connection instead of your dual bonded 1GB connection, or that you have TCP Chimney enabled on the EV Server and the SQL Server..... it could even be a damaged wire or an out of date driver for the NIC

 - Exchange - It could be that Exchange itself is having issues, you might go on the exchange server and find that Exchange is literally just kicking you out and disconnecting you, or that its own drives are so heavily fragmented and destroyed that the amount of requests EV is polling is causing it to become unresponsive

make sure to the check the System and application logs of the EV Server, SQL server and Exchange server, find out whether this is just one Exchange server thats slow or all of them, try even something as simple as copying a large file between servers and see how long it takes to copy

Also try creating a mail profile in outlook and see if theres anything unusually slow, unresponsive or erroneous happening when you connet 

Leonardo Fernandez 2's picture
25
Jul
2011
0 Votes 0
Login to vote

The case was closed today but

The case was closed today but can be reopened within 10 days.
I have an SQL "expert" and he tells me that SQL is ok, but I will ask him the same question you asked me.
We did dtrace the archivetask and I sent it to Symantec. They told me everything looked normal.
CPU and memory of SQL was confirmed ok by my SQL engineer.
I've checked the exchange server logs and there are no errors.
One of my server engineers ran a performance monitor during my midnight to 8am schedueld archive run and thaere were high disk drive i/o rates
I've checked Exchange inside and out and its ok.
I have gone through all the logs on the ev server and there are no errors. As a matter of fact no events are registered in the EV log between midnight and 8am, which I find strange to begin with.

Access to archvied email is just fine.

Everything in Exchange and Enterprise vaylt works fine, except that archiving has slowed down tremendously.

JesusWept2's picture
25
Jul
2011
0 Votes 0
Login to vote

Dont suppose you have Moved

Dont suppose you have Moved Items enabled in your Archiving Policy do you?

Leonardo Fernandez 2's picture
25
Jul
2011
0 Votes 0
Login to vote

I am attching a screenshot of

I am attching a screenshot of our SQL sentry screen showing sql activity. My sql engineer says all the wait time is from the front end, meaning the EV server.

SQL Activity.jpg