Event 3419 MAPI error detected for an Exchange 2010 or later target (possible DAG failover) - 0x80040115 OR Task failing with Event 3305 and 2258

Article:TECH169970  |  Created: 2011-09-20  |  Updated: 2014-02-20  |  Article URL http://www.symantec.com/docs/TECH169970
Article Type
Technical Solution

Product(s)

Environment

Issue



The Enterprise Vault Event log contains Events 3419 and 2217. 
The HRESULT 0x800400115 (MAPI_E_NETWORK_ERROR) is the cause of the event. These events start occurring 5-7 minutes after the tasks have started and are typically repetitive and occur approximately every 5 to 7 minutes once the initial issue is encountered. This can lower over all performance when archiving an Exchange 2010 environment.

OR

The task failing with generic Event 3305 and 2258

The HRESULT 0x8004011d (MAPI_E_FAILONEPROVIDER) is the cause of the event.

 


Error



The following is an example of the error message:

Event ID 3419: 
There was a problem accessing an Exchange 2010 or later server. The dispenser will re-queue the current item and sleep for 1 minute(s).

Followed by the Informational Event:
Event ID: 2217: 
Message dispenser will resume processing

When performing a Dtrace of JournalTask or Archive Task, this will report HRESULT: 0x80040115
 
For the Journal Task:
EV:M    CExchangeShortcutAccessor::SetSession - Call to OpenEntry Failed [0x80040115] - Retrying

For the Archive Task:
EV:M    MAPI error detected for an Exchange 2010 or later target (possible DAG failover) - 0x80040115

 

OR

Dtrace shows

------------------------

{CMailboxHelper::CreateProfileAndSessionEx:#995} MAPI Profile [VaultMbxAgent-MAIL02-7804-1392716766-0-588-0] successfully created.
{CMailboxHelper::OpenMailbox:#289} Reading message store table.
{CMailboxHelper::OpenMailbox:#325} Opening message store...
{CMailboxHelper::OpenMailbox:#334} Could not open message store: [0x8004011d]
{IsCurrentUserADomainAdmin:#4541} User is domain admin: [False]
Event ID: 3305 The Task 'Exchange Journaling Task for MAIL02' failed to log on to Exchange server 'EUMAIL02' using mailbox 'SMTP:EU-IT-EVMAIL02@group.com'. Please ensure the mailbox has not been hidden, that the server is running and that the Vault account has sufficient permissions on the server.
Event ID: 2258 The Journal Task 'Exchange Journaling Task for MAIL02' could not be started due to startup errors.

Netmon Log Shows

------------------------------ 

  Frame: Number = 649, Captured Frame Length = 86, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[B4-B5-2F-55-3E-64],SourceAddress:[00-25-90-51-80-08]
+ Ipv4: Src = 10.202.170.10, Dest = 10.202.170.4, Next Protocol = TCP, Packet ID = 36692, Total IP Length = 72
+ Tcp: Flags=...AP..., SrcPort=59534, DstPort=15346, PayloadLen=32, Seq=3662234255 - 3662234287, Ack=2148205286, Win=183 (scale factor 0x7) = 23424
- MSRPC: c/o Fault:  Call=0x4  Context=0x0  Status=0x1C010014  Cancels=0x0
  - Fault:
     RpcVers: 5 (0x5)
     RpcVersMinor: 0 (0x0)
     PType: 0x03 - Fault
   + PfcFlags: 35 (0x23)
   + PackedDrep: 0x10
     FragLength: 32 (0x20)
     AuthLength: 0 (0x0)
     CallId: 4 (0x4)
     AllocHint: 32 (0x20)
     PContId: 0 (0x0)
     CancelCount: 0 (0x0)
     Reserved: 0 (0x0)
     Status: 0x1C010014 nca_server_too_busy - The server is too busy to handle the call.
     Reserved2: 0 (0x0)

Environment




 


Cause



Hardware and software load balancing appliances actively harden TCP traffic overheads in an effort to stop all potential high bandwidth traffic from causing network delays. All communication that occurs between the Enterprise Vault Server is TCP traffic. These connections between Microsoft Exchange and the Enterprise Vault Server are usually open for extended periods of time. When a load balancer is implemented with a Microsoft Exchange Server, all TCP traffic is given a timeout value. This timeout value actively ceases all extended TCP sessions between the two communication points.

The 5 to 7 minute frequency in which error messages appear is caused by the closing of TCP sessions between the Enterprise Vault Server and the Microsoft Exchange Server. When these connections are dropped, the Enterprise Vault Server loses connectivity to the Microsoft Exchange Server and erratic behavior occurs. The load balancing appliance follows TCP Stack Hardening guidelines that force any open TCP traffic that is passing through it and is over the 5-minute threshold to be closed.
 
It has also been found that the CAS server is Throttling RPC connections from different service accounts, thereby causing contention with EV connections.  This is a known issue and referenced in the following Microsoft article: support.microsoft.com/kb/2668900

Solution



Workaround:
Bypass the hardware load balancer with a hosts file entry so that the Enterprise Vault Server may communicate directly with a single Client Access Server (CAS).

1. Browse to c:\Windows\System32\drivers\etc and edit the hosts file.
2. Create an entry with the IP address of one of the CAS nodes and the network name of the hardware load balancer. The name of the hardware load balancer will match the Exchange 2010 CAS Array name.

Example:
10.10.10.34 (IP address of the CAS Node) CASArray.domain.com (Fully Qualified Domain Name of the CAS Array)

3. Save the file.
4. Open a Command Prompt and run ipconfig /flushdns
5. Restart the Enterprise Vault Task Controller Service.

Solution:
Increase the default TCP Session Timeout on the applicable Load Balancing Appliance.

Note: For further assistance to increase this timeout value, please contact the appliance vender.

Acknowledgements:

www.blackberry.com/btsc/KB28704




Article URL http://www.symantec.com/docs/TECH169970


Terms of use for this information are found in Legal Notices