Video Screencast Help

Check to see if users mailbox really does exist on exchange to prevent 3310 errors

Created: 30 Jun 2010 • Updated: 06 Jun 2011 | 5 comments
JesusWept3's picture
8 Agree
0 Disagree
+8 8 Votes
Login to vote
Status: Implemented

I just wanted to float a product enhancement request (it's on the border between broken and being by design, needing enhancement)
The issue is regarding Tasks failing due to people either storing or restoring from their vault, however their exchange server has changed and the request goes to the wrong server.
For instance
1. UserA is on ExchangeServer1 and they get moved to ExchangeServer2
2. For whatever reason, provisioning and synchronization does not update the users hidden message
3. User selects an item and then does a Store In Vault
4. The Archive Task posts the request to “Enterprise Vault Exchange Mailbox Task for ExchangeServer1 A2” queue
5. Archive Task picks up that request, verifies the calling user has permission to the archive
6. The Archive Task then logs on to ExchangeServer1 throws a MAPI_E_NETWORKERROR because the user no longer exists on that exchange server
7. A 3310 error is thrown by EV and the task stops for 10 minutes
8. The task starts up, immediately tries to process the same A2 message and fails
This then becomes a rolling problem because if the issue is not seen for some time, then synchronization will not occur for that exchange server because the task is failed
And because its failed and it can’t synchronize, more users fall in to the similar situation.
This also presents us with a number of other challenges
1. To get around the issue you have to clear out the message from the MSMQ
     But MSMQ is all or nothing, you either purge all the items or nothing, meaning you will  purge legitimate items
     OR you can buy an excellent product named QueueExplorer by Cogin ($49 with the amount of servers we have is over $4,000)
2. Even with QueueExplorer, deleting messages can be time consuming because in the instance I had today , there were 24 unique users waiting for items to restore, but 8 of them had been moved
    So at first it was confirm the user blocking the queue and delete his/her messages, and then restart the task, but after that I went through each user ID and performed AD lookups to find their home each server
3. When a task is in a 3310 State, it is in a Running state, not failed or stopped, so EVOM will show that the tasks are A-OK, the VAC Health pad says it’s all A-OK
What I’m requesting is that EV do one simple lookup before processing a user and that is to just check the exchange server the user belongs to as opposed to purely trusting what the queue says
Also post a more descriptive error as to why the 3310 is occurring
Event Type:                     Error
Event Source:                Enterprise Vault
Event Category:            Retrieval Task
Event ID:   3310
Date:                                    6/30/2010
Time:                                   5:36:35 PM
User:                                    N/A
Computer:                       EVSERVER1
There was a problem accessing a network service or resource.  The dispenser will re-queue the current item and sleep for 5 minute(s).
Task: Exchange Mailbox Archiving Task for EXCHANGESERVER1(Retrieval)
The Dtrace:
10133         15:10:32.588                 [9952]        (ArchiveTask)                  <9636>      EV:M             EX2KMBXPERM::cmr[/o=Domain/ou=myOrg/cn=Recipients/cn=myUser] - Caller = DOMAIN\myUser
10134         15:10:32.588                 [9952]        (ArchiveTask)                  <9636>      EV:M             EX2KMBXPERM::cmr[/o=Domain/ou=myOrg/cn=Recipients/cn=myUser] - Caller = DOMAIN\myUser is the mailbox owner
10135         15:10:32.588                 [9952]        (ArchiveTask)                  <9636>      EV:M             MBXPERM::CCA[/o=Domain/ou=myOrg/cn=Recipients/cn=myUser] - caller (DOMAIN\myUser) has rights to mailbox
10136         15:10:32.588                 [9952]        (ArchiveTask)                  <9636>      EV:H              :CArchivingAgent::ArchiveItem() |Return the MAPI session to the session pool |
10137         15:10:32.588                 [9952]        (ArchiveTask)                  <9636>      EV:H              :CArchivingAgent::ArchiveItem() |Get the MessageEntryId from the item being archived |
10138         15:10:32.588                 [9952]        (ArchiveTask)                  <9636>      EV:L               CRetentionCategoryCache::ReLoad (Entry) |
10140         15:10:32.588                 [9952]        (ArchiveTask)                  <9636>      EV:M             CBaseDirectoryServiceWrapper::CreateDirectoryService() - Entry [m_nNumTries = 40]
10141         15:10:32.588                 [9952]        (ArchiveTask)                  <9636>      EV:L               CBaseDirectoryServiceWrapper::CreateDirectoryService() - Successfully communicated with an EV Directory Service on the local machine
10142         15:10:32.588                 [9952]        (ArchiveTask)                  <9636>      EV:L               CRetentionCategoryCache::ReLoad (Exit) |Success  [0] |
10143         15:10:32.588                 [9952]        (ArchiveTask)                  <9636>      EV:H              :CArchivingAgent::ArchiveItem() |Everything has been successful up to this point, so we are going ahead and calling ProcessItem. |
10144         15:10:32.588                 [9952]        (ArchiveTask)                  <9636>      EV:M             CArchivingAgent::ProcessItemInternal ArchiveId: 16C2ADFB29F0C634A91038E49A8BD642A1110000myEVSite | VaultId: 1ED6CAE1A8B456B4288FAD77B320F437F1110000myEVSite
10145         15:10:32.588                 [9952]        (ArchiveTask)                  <9636>      EV:M             CArchivingAgent::ProcessItemInternal [queued:T][PropertySet:(null)]
10146         15:10:32.588                 [9952]        (ArchiveTask)                  <9636>      EV:M             EPC::GDP - Returning Default Policy : [Default Exchange Mailbox Policy][1A5C9C61102B8844EB22E823AEFB07B9E1012700myEVSite]
10186         15:10:32.619                 [9952]        (ArchiveTask)                  <9636>      EV:L               :CArchivingAgent::ProcessItem() |Getting a MAPI session from the session pool |
10187         15:10:32.619                 [9952]        (ArchiveTask)                  <9636>      EV:M             CMAPISession::GetMapiSessionFromPool(AdditionalFlags = 0)
10188         15:10:32.619                 [9952]        (ArchiveTask)                  <9636>      EV:M             CMAPISession::GetMapiSessionFromPool(): Exit status: 0x0
10189         15:10:32.619                 [9952]        (ArchiveTask)                  <9636>      EV:M             CArchivingAgent::ProcessItemInternal - Call to PI_GetMAPISession Failed RetryCount[1] [0x8004011D]
10190         15:10:32.619                 [9952]        (ArchiveTask)                  <9636>      EV:L               :CArchivingAgent::ProcessItem() |Exiting routine |

Comments 5 CommentsJump to latest comment

TonyD's picture

This is a daily problem in our very large environment. It would be greatly appreciated if this suggestion could be given some serious thought!

Login to vote
jprknight-oldaxatech's picture

It looks as though we have also run into this problem as well; and could very well continue to in the future as we also have a larger environment and need to load balance our Exchange mailboxes evenly across out Exchange servers.

Login to vote
Alex Brown's picture

This sounds more like something that should be escalated via support rather than a product enhancement request. It may be that the changes we made to the Exchange agent for Exchange 2010 in EV 9.0 can help  with this matter but ultimately I don't think this should exist as a new feature product enhancement rather fixing up existing functionality.

Login to vote
JesusWept3's picture

sorry, yeah its been fixed in EV9, it simply throws the item out, generates an error saying no such object on server and carries on which is perfect

But to be honest, we went the support route, and CFT told us that it was an enhancement request, so it was caught up in a quasi enhancement bug purgatory where no one could agree on what it was, other than it was an annoyance for us

Login to vote
Alex Brown's picture

ok thanks for letting us know and am pleased this scenario is handled more elegantly in 9.0.

I'll close this idea off for now.

Login to vote