Data Loss Prevention

 View Only
  • 1.  Exchange 2010, IronMail, and Email Prevent unhandled exception

    Posted May 09, 2012 08:40 AM

    Hello all,

    I have a problem with my email prevent and Im having difficulty coming to a solution. My setup is Exchange 2003/2010 (upgrading), to email prevent 11.5 (in forwarding mode w/ no mx lookup, no TLS), to Ironmail, to internet. Whats happening is that the connection is being closed due to an error of some sort, the exchange queue gets backed up, then when the connection is restored, the queue is flushed and mail is delivered as expected. It does this every 5 minutes or so.

    The error in the logs are as follows:

    RequestProcessor Log: SEVERE: RPT(2c): Returning fatal response and terminating connections due to unhandled exception.            java.lang.NullPointerException

    SMTPOperational Log: 07/May/12:12:10:44:622-0400 [INFO] (SMTP_CONNECTION.5203) Forward connection error (tid=2c cid=2 mta=<> reason=<>)07/May/12:12:10:44:622-0400 [SEVERE] (SMTP_CONNECTION.5204) Peer disconnected unexpectedly (tid=2c cid=2 local=<> remote=<> reason=java.lang.NullPointerException)

    It will usually write the above errors 4 or 5 times, then, it will finally reconnect, at which point it will send all the email that has accumulated in the queue. It will work for a while, at which point it will start failing again. Emails pile up in the queue and we go through the process again.

    We have tried several things, but nothing seems to work. From a packet capture I can see that its failing when the exchange sends an EHLO command, so I think the error is between Exchange and Email prevent, but Im not sure if the IronMail device plays any role. We have turned off senderID as I seen a KB article that said that could cause the issue, and made sure TLS is not being used. None of that had any effect.

    If anyone has any ideas I would greatly appreciate them, Im really not sure what to do next.

    Thanks in Advance!!

    D

     

     



  • 2.  RE: Exchange 2010, IronMail, and Email Prevent unhandled exception

    Posted May 09, 2012 04:33 PM

    dkscriv, have you attempted to open a support case with Symantec? I think in this case it may be the best thing to do so they can try and walk through troubleshooting the error. Offhand it looks like there must be some sort of connection error as you said. With the Email Prevent functions provided and using Forward mode, I know the Symantec server never truly takes ownership, so if there is a problem opening both connections on both sides properly at the same time, that may play into the issue. Honestly I think you should get with support though to have them troubleshoot this kind of error.



  • 3.  RE: Exchange 2010, IronMail, and Email Prevent unhandled exception

    Posted May 10, 2012 07:34 AM

    I do have a case open but looks like they might be stumped as well.

    Interestingly, I have 2 email prevents, one goes to another Exchange org. It doesnt have the Ironmail device in its path. It would be pretty easy to say that Ironmail is the issue, but I dont really have any proof thats it. From my packet captures and logs it's exchange thats dropping the connection. But I get it could be dropping based on a response from either the DLP or Ironmail device.

    Has anyone out there implemented DLP w/ Ironmail? I thought it was SMTP compliant.

    D



  • 4.  RE: Exchange 2010, IronMail, and Email Prevent unhandled exception

    Posted May 10, 2012 10:39 AM

    dkscriv,

    I would suggest a next course of action, to reach out to your local Symantec account team and see if they can't escalate the case for you. Support being stumped sounds a little off especially around the DLP product. Your configuration is not very custom and I'm sure there are others using your same configuration. We have some really smart guys that work on the higher tiers of support and I'm sure if it is handed to them they'd be able to get the right resources to identify the issue.

    Unfortunately I haven't setup EMail Prevent with IronMail before so I don't have any knowledge of what the process looked like or some potential hiccups I may have remembered hitting. Sorry I couldn't be of more help.



  • 5.  RE: Exchange 2010, IronMail, and Email Prevent unhandled exception

    Posted May 10, 2012 04:52 PM

    Dkscriv,

    The biggest thing to remember when troubleshooting SMTP Prevent issues, especially regarding connections, is that it's an SMTP proxy, not an MTA. So when a connection request comes in from Exchange, DLP will attempt to connect to Ironmail FIRST and only once Ironmail has accepted it's connection request will DLP turn around and accept the connection request from Exchange. The rest of the SMTP conversation follows the exact same pattern: receives from Exchange, passes to Ironmail, once confirmed responds to Exchange.

    You mentioned in your post that it shows up when Exchange sends an EHLO command, but if Ironmail refuses that command from DLP, DLP will turn around and refuse it to Exchange.

    The SMTPOperational log entry you referenced doesn't have the IP of the remote server it's referencing, and that's normal for most entries other than the initial connection log entry, but in order to figure out what is closing the connection we need to track down which server it's referencing.

    In the log you showed there is a reference to which connection the error is for: Peer disconnected unexpectedly (tid=2c cid=2 local=<> remote=<>. The 'tid=2c cid=2' are references to a particular connection. Look in that same log, before the error entry, and find where that connection is initially established (will have the same tid and cid). That initial log entry will have IP's for Local and Remote - the Local IP will be the SMTP Prevent server and the Remote IP will be either the upstream Exchange server or the downstream Ironmail server.

    Once we know which server is disconnecting unexpectedly, we can look into why it's happening. Some other contextual information that would be helpful:

    1. How many concurrent connections are you allowing? (System > Overview > [server] > Config > Inline SMTP > Maximum Number of Connections

    2. How many CPU cores on the Prevent server?

    3. Is there a loadbalancer between Exchange and DLP or DLP and Ironmail? Is Ironmail clustered? If there's a loadbalancer, what kind is it?

    Let us know what you find...

    - Tim



  • 6.  RE: Exchange 2010, IronMail, and Email Prevent unhandled exception

    Posted May 14, 2012 08:57 AM

    Thanks for the reply, sorry Ive been out and been unable to reply until this morning.

    12 concurrent connections

    2 procs - VMWare

    No load balancing, no clustering

    Im not 100% sure on the logs, from the requestprocessor log: 

    NFO: RPT(2e) Waiting for new connection
    May 14, 2012 8:08:02 AM com.vontu.mta.rp.ESMTPRequestProcessorThread _acceptPeer
    INFO: (SMTP_CONNECTION.1201) Connection accepted (tid=2c cid=1,433 local=EXCHANGE:25 remote=DLP:40778)
    May 14, 2012 8:08:02 AM com.vontu.mta.rp.ESMTPRequestProcessorThread connectNextHop
    INFO: (SMTP_CONNECTION.1203) Forward connection established (tid=2c cid=1,434 local=DLP:60967 remote=IRONMAIL:25)
    May 14, 2012 8:08:02 AM com.vontu.mta.rp.ESMTPRequestProcessorThread _process
    SEVERE: RPT(2c): Returning fatal response and terminating connections due to unhandled exception.
    java.lang.NullPointerException
    May 14, 2012 8:08:02 AM com.vontu.mta.rp.ESMTPPeer close

    For whatever reason, the smtpprevent log is not showing the connection as closing now?

    From a wireshark capture, I see the ACKS from Exchange to DLP, and DLP to Ironmail, and then SMTP Proxy Server Ready, then the EHLO command, which leads to:

    421 4.3.0 Fatal Processing Error. Closing Connection.

    Im at a loss. The server is running on VMWare 5, so Im wondering if this is somehow the issue. Could lowering the concurrent queue possibly help? We're not really sending a lot of mail, so perf shouldnt be an issue.

     

    Thanks again for the reply!!


    Daryl



  • 7.  RE: Exchange 2010, IronMail, and Email Prevent unhandled exception
    Best Answer

    Posted May 24, 2012 10:50 AM

    The solution was to add more processor cores (2 cores, 2 procs) in VMWare. The odd thing is my other exchange to exchange monitor, which has basically the same load, doesnt have this issue with the exact same hardware. I guess its just at its limit and we havnt seen this behavior yet, or Ironmail is changing the connection in some way.