Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Why are messages still queued to be archived?

Created: 14 Feb 2013 • Updated: 21 Mar 2013 | 15 comments
This issue has been solved. See solution.

Hello,

According to our monitoring system, we archive upto 50K messages every night. However, the same system reports that when our archiving run completes, there are still a couple of thousand messages in the queue waiting to be archived.

Can someone tell me please if that's normal? Otherwise, what are the main reasons for it?

We have about 4,000 users with a 6-hour archiving window. EV904/Exchange 2003.

Thanks,

- Alan.

Comments 15 CommentsJump to latest comment

TonySterling's picture

Which queue specifically?  If it is a5 that is mailboxes to be processed and it is normal to have some items in there at the end of the schedule.  It will pick up where it left off.

When you upgrade to EV 10 sp 2 or later you will have a nice archiving report created at the end of the run that you can reference.

TS

JesusWept3's picture

As tony said if its A5, don't worry about it,
One thing you can do is open up the queue message, scroll all the way down to the bottom and double click the last message, this will be your oldest message, look at the submit time for that message

If your archiving window kicks off at something like midnight, and it has a submit time of 2am, then it simply means that it re-queued the user for archiving but then ran out of time to get to that mailbox again.

So the way that it works is you have Number of Messages Per Pass set on the task, by default 1000 messages.....

1. The Schedule starts at midnight
2. Posts a message to A5 to Process Exchange Server
3. EV looks up all the enabled users for that exchange server
4. Posts a message in the queue for each enabled user
5. Each of those messages will have a submit time of midnight
6. EV will then take several at a time (default is 5 users or 5 concurrent connection)
7. EV then scans the mailbox and archives up to 1000 messages
8. EV then moves on to the next mailbox in the queue and so on and so forth until the queue is empty
9. Any mailboxes with more items to archive past those 1000 items will be re-added to the A5 queue
10. So if it took 1 hour to process all mailboxes at 1000 messages per pass, you may have additional mailboxes queued, and the submit time be somethng like 3:50am

Then your archiving schedule stops at 4am
Well the message is still in the A5 queue, because EV only had ten minutes to attempt to get to that mailbox and simply ran out of time

The A6 queue would be Moved/Updated Shortcut processing, and that will run throughout the day regardless of schedule to update the metadata in the ev database, so although you might come in in the morning and see 3000 messages in A6 queue, it will go down throughout the day.

Storage Archive and Storage Restore queues will also go down regardless of the schedule.
I believe A1 and A2 will still go down regardless of the schedule.....
None of the queues will move though if EV is in backup mode

BruGuy's picture

Thanks guys.

Yes indeed, it's the A5 queue for each Exchange server's (we have four) archiving task. There are between 300 - 600 mesages in each. Seems to be exactly 4 messages in the A4 queues too. All others are 0. 

The number of messages per pass is set to 2000 and concurrent connections to 5.

What's stange is that the A5 contains mailboxes which don't even have 2000 messages in total in them (some only have 5 or 6 messages in total), so why would they be requeued? There are about 20 queue messages just called [1] too. 

All the messages have the same time of 4:20am, our archiving run is from 9pm - 6am, backup is afterwards.

Is that all normal?

GertjanA's picture

Jumping in here: and, what about [2]?

Thanks JW for your (as always) thorough and clear explanation. But, this does also mean that at the start of the new scheduled run, these existing entries will first have to be processed, and then the newly added are being handled right? It is a sequential run if I am not mistaken.

Question of Alan on [1] and mine on [2], I believe it has something to do with a retry, but I am not sure. If Tony or JW know, please enlighten us :-)

Thank you, Gertjan, MCSE, MCITP,MCTS, SCS, STS
Company: www.t2.nl

www.quadrotech-it.com

www.symantec.com/vision

JesusWept3's picture

You're right Gertjan, [1] [2] etc is a retry on fail count
Once it gets to three the mailbox is discarded and won't be archived against that archiving run

BruGuy's picture

I still can't figure out why the queue contains mailboxes which only contain 5 or 10 mail items in total? If it can archive 2000 messages in one pass, why wouldn't they be picked up on the first pass?

Thanks for the help anyways. Much appreciated.

JesusWept3's picture

Because they're erroring out

BruGuy's picture

So it Is something to worry about after all, if a few hundred normal mailboxes "error out" every single night?

Why would enabled, under-quota mailboxes with the correct perms not be archived? Is there a tech note explaining the reasons to look for please?

JesusWept3's picture

To be honest you'd have to DTrace it to see what's going on if nothing is being thrown in the event log

Typically they're soft or recoverable errors
Other times it can be that they're mailboxes moved from on exchange server to another but provisioning hasn't been run so it doesn't reflect the change, so you end up with these mailboxes in the wrong MSMQ etc

BruGuy's picture
Thanks.
 
Well the event log says that its failing to sync some mailboxes because they're over-quota. Not many though. Does that mean syncing uses the A5 queue too?
 
The event log has a lot of strange errors for a few mailboxes, or a specific message sent to many mailboxes:
 
Event Type: Warning
Event Source: Enterprise Vault 
Event Category: Archive Task 
Event ID: 2270
Date: 16/02/2013
Time: 06:54:35
User: N/A
Computer: EV1
Description:
A queued operation exceeded the retry count and has been discarded
 
Exchange Mailbox Archiving Task for EX1
 
Updating moved items in a folder. Folder path: [-Inbox-Mission-Mike-family 31.03.-20.04.2012], Exchange mbx DN: [/O=Company/OU=Site/cn=Recipients/cn=x], archive ID: [15D058FF179E5C08C0838B2D7F1110ev1].
 
Archive folder ID: [1DF62C4944B4CAC63E32B4E5111ev1], folder retention category: [], EVPM folder: [False], compressed item request XML length: [2616].
 
Error: [0x80004005]
 
 
Event Type: Error
Event Source: Enterprise Vault 
Event Category: Storage Online 
Event ID: 6578
Date: 16/02/2013
Time: 06:54:35
User: N/A
Computer: EV1
Description:
Abnormal error occurred 
 
Object:    CRetentionCategoryCache 
Reference: RE(1)/fe 
 
For more information, see Help and Support Center at 
evevent.symantec.com/rosetta/showevent.asp?EvtID=6578
Data:
0000: 54 68 65 20 70 61 72 61   The para
0008: 6d 65 74 65 72 20 69 73   meter is
0010: 20 69 6e 63 6f 72 72 65    incorre
0018: 63 74 2e 20 20 28 30 78   ct.  (0x
0020: 38 30 30 37 30 30 35 37   80070057
0028: 29                        )       
 
And a couple of these:
 
Event Type: Error
Event Source: Enterprise Vault 
Event Category: Archive Task 
Event ID: 3432
Date: 16/02/2013
Time: 06:26:26
User: N/A
Computer: EV1
Description:
One or more errors occurred during the creation of a profile to connect to an Exchange Server. 
 
Targeted Exchange Server: EX1
Mailbox: SMTP:SRV-APP-EX1@company.com
 
ConfigureMsgService failed with the following errors: 
 
Out of memory error has occurred for the following connection points:
EX1
dc1
 
This indicates that the maximum number of connections has been consumed on this Enterprise Vault Server.
 
And these:
 
Event Type: Error
Event Source: Enterprise Vault 
Event Category: Archive Task 
Event ID: 3433
Date: 13/02/2013
Time: 01:46:50
User: N/A
Computer: EV1
Description:
Could not build a saveset for the message titled [Photos nov 2012] in Mailbox [/O=Company/OU=Site/cn=Recipients/cn=SIM], function [CArchivingAgent::PI_BuildSaveset] failed. 
 
Error: 0x8004011B 
Description: Unknown error 0x8004011B 
 
For more information, see Help and Support Center at 
evevent.symantec.com/rosetta/showevent.asp?EvtID=3433
 
Event Type: Error
Event Source: Enterprise Vault 
Event Category: Storage Archive 
Event ID: 6592
Date: 13/02/2013
Time: 01:33:13
User: N/A
Computer: EV1
Description:
Abnormal error occurred 
 
VaultId:   122E420426A10B8C5200D0978B001110090ev1
Reference: CStore::GetOpenPartitionVolume()/Invalid device type 
 
For more information, see Help and Support Center at 
evevent.symantec.com/rosetta/showevent.asp?EvtID=6592
Datta:
0000: 43 61 74 61 73 74 72 6f   Catastro
0008: 70 68 69 63 20 66 61 69   phic fai
0010: 6c 75 72 65 20 20 28 30   lure  (0
0018: 78 38 30 30 30 66 66 66   x8000fff
0020: 66 29                     f)      
 
And similarly:
 
Event Type: Error
Event Source: Enterprise Vault 
Event Category: Archive Task 
Event ID: 2244
Date: 13/02/2013
Time: 01:07:49
User: N/A
Computer: EV1
Description:
An error occurred whilst processing the message titled: Photos nov 2012 for mailbox: /O=Company/OU=Site/cn=Recipients/cn=sim, the message will not be archived and could not be changed back from the Archive Pending state. 
 
 
Seems to be a corrupt message in many different mailboxes....
 
And the usual end-user problem warning:
 
The folder has Default permissions set that grant all users access to this folder. By default, this has not been synchronized to the users archive. 
 
And lastly:
 
The Exchange mailbox provisioning task found enabled mailboxes that are not part of a provisioning group. These mailboxes cannot be archived. If you do not want these mailboxes to be archived, use the Administration Console to disable them. See the provisioning task report for further details (full reports only).
 
The converter logs have this type of entry often:
 
Event Type: Warning
Event Source: Enterprise Vault Converters
Event Category: Storage Archive 
Event ID: 28993
Date: 16/02/2013
Time: 03:54:48
User: N/A
Computer: EV1
Description:
Unable to convert item content 
 
Reason: Unspecified error  (0x80004005) 
Supplementary Info: file could not be opened (7) [CS2D] 
 
Item: Saveset Message for Mailbox /O=Company/OU=Site/cn=Recipients/cn=simx, 
Attachment:  
Type: zip 
Vault ID: 187BE50D844E850DB134AFA643DB1100ev1 
SaveSet ID: 201302163215~2013011600000~Z~11497076C19E9947AE067367041 
Attachment ID: 1 
 
This item will be archived without a preview being available to the web application and the  content will not be indexed. It is not possible to search on the content but the item can be restored  as normal 
 
 
Thanks for any advice if anything is really strange here. There are so many but it's probably normal with a few thousand mailboxes.
 
Chris Warren's picture

Gonna chime in here.  You have a variety of different errors..  I'll comment on a few of these generally:

Those top 2270's (Updating moved items in a folder.) and the 6578's are related.  Basically because something during the scan to determine if shortcuts were moved didn't pull the Retention category right (Note the folder retention category: []).

For these, look at the following:

http://www.symantec.com/docs/TECH156217
During Shortcut Processing to update Moved items, Event Ids 6578 and 2270 occur when 'Archive Exchange Managed Folders' is set to 'Off'

http://www.symantec.com/docs/TECH188048
Exchange Shortcut Processing is incorrectly identifying all mailbox folders as being moved

http://www.symantec.com/docs/TECH71905
Moved shortcuts are not updated correctly during archive task

For the 3432's, see the following forum post:

https://www-secure.symantec.com/connect/forums/eve...

The 6592's, you may want to open a case with Support on.

Finally, the last events you posted seem pretty self-explanitory due to corrupt or unreadable messages.

On a side, if a mailbox is over quota, that will affect Synch's as well as archiving, since EV must be able to update exchange objects to change messages class and such.  If Reads and Writes are denied, EV will have issues.

I hope this helps.

BruGuy's picture

Thank you very much indeed Chris.

- For the 2270 and 6578 errors: I'm not quite clear. We only have the default "Business" retention category in EV and we don't use Managed Folders in Exchange (2003). We did create a "DoNotArchive" top-level folder in most users' mailboxes using EVPM - if users move or rename that folder, does it cause an error?

- The 3432 error doesn't appear to be happening anymore but I'll keep an eye on it and apply the reg hack if needed.

- The corrupt message error is being caused by a single message which was sent to all users. It has little text but the TO field contains a few thousand recipients. That means the message envelope is almost 1MB and EV can't seem to handle that. Should we tick the "set failed messages - do not archive" in the archiving policy? Does that need an associated reg hack? I think that's why our A5 queues are so long.

Thanks again.

Chris Warren's picture

For the 2270's and 6578's, I'd recommend applying ClearFailedMoveUpdate (TECH71905)

As for the 'Corrupt' messages, that can be related to the maximum number of Recipients being hit by Outlook.  Check the following:

http://www.symantec.com/docs/HOWTO74999
Maximum Outlook attachments and recipients: AttachmentMax and RecipientMax

I believe MS lowered the max to 2000 at one point, which can make EV and Outlook consider items 'corrupt' if they go above this.

I hope this helps.

SOLUTION
Chris Warren's picture

Heya - Are you still experiencing errors from above?

BruGuy's picture

Hi Chris,

Happy to report that setting the RecipientMax fixed practically all of the errors, after a few days.

We still get some event log errors like "Object: CRetentionCategoryCache / Reference: RE(1)/fe" but no noticeable side-effects.

Thanks for your help,

- Alan.