Sorry for the belated response here. Certificates are always one of those thorn in the foot type of issues.

So I guess what I'm trying to wrap my head around here is you said at one point that these are outbound emails correct?

When sending messages outbound it is really up to the receiving server to supply the sending server with it's public certificate in order to encrypt the messages. If this is a self signed certificate, we obviously would not have them as a trusted authority and so the certificate would fail.

So to be honest I'm not sure if we can "fix" this issue because it is not really anything we have control over. I was under the assumption that we would fail open for outbound if we are allowed to, but I could be wrong on that one.

Please let me know if I am misunderstanding the issue or if there is anything that I can clarify.

Thanks, and I hope this helps!
Tom 

It sounds like your interpretation is mostly correct.  Let me see if I can clarify at all.

Upon opening the SMTP connection to the remote domain in question, our SBG sends (as it is configured to) the STARTTLS command.  After TLS begins, the handshake fails - based on what I can see in the MTA logs (set at the debug level).  I can't seem to find an exact reason for the handshake failure, but I'm assuming that it's because the remote domain is offering a self-signed cert, and, of course, as you mentioned, our SBG is not configured to trust that cert.  On our side, we're using a Thawte cert, so there shouldn't be any problems there. 

Once the TLS handshake fails, the entire connection fails and the message we were attempting to send goes into the delivery queue with an "error on HELO/EHLO" status.

As I mentioned, in the advanced SMTP configuration, I've got what I'm calling "opportunistic TLS" enabled globally for outbound messages by checking off "Attempt TLS encryption for delivery of all messages"

So, since our SBG is attempting TLS with the remote server by sending the STARTTLS command, we have the failure scenario.  If I uncheck "Attempt TLS encryption for delivery of all messages", our SBG does not send the STARTTLS command to the remote domain in question and simply sends the message in clear SMTP.  What I'm looking for is a way to prevent our SBG from sending STARTTLS to that specific domain only, so that we can continue to do opportunistic TLS with other remote servers where applicable.

I apologize if I'm not explaining this very well - it does appear to be a quite complicated situation.

Thanks for your help,

You are explaining it perfectly. So no worries there.

 I also stand corrected. We changed the domains section so that it is for both local and non-local domains. I am getting some of this clarified so I will have a response here in a few.

Thanks! And sorry for incorrectly informing you.

*I have edited my post so as not to mislead*

Ok, so from what we have so far I am not sure whether we can address this or not.

Basically what it sounds like is that the receiving domain is requiring TLS encryption in order to accept mail. I find this odd if in fact they are using a self signed certificate because this would cause issues with mail flow across the board for them. Not just with you.

What you have done so far should allow us to fail open if the STARTTLS command fails. Either unchecking or having "Attempt TLS encryption" should keep this in an open state. You may also try "Require and don't verify" and see if this works.

Other than that most likely you would need to clarify what is happening with the receiving domain. Which may need to be done anyways.

If there is more needed to be done on our side I would suggest giving us a call and opening a case. Maybe having a little more personal interaction would help with getting to a resolution.

Thanks for your time! I hope this gets sorted out.
Tom

Just to clear up a couple remarks:

Basically what it sounds like is that the receiving domain is requiring TLS encryption in order to accept mail. I find this odd if in fact they are using a self signed certificate because this would cause issues with mail flow across the board for them. Not just with you.

That's not actually the case.  They CAN receive cleartext SMTP, and they probably are receiving cleartext SMTP from everyone else in the world except us, as we're configured for opportunistic TLS globally (that is, if ANY remote server responds with STARTTLS as one of the available commands after a HELO/EHLO, then we will send the STARTTLS)

What you have done so far should allow us to fail open if the STARTTLS command fails. Either unchecking or having "Attempt TLS encryption" should keep this in an open state. You may also try "Require and don't verify" and see if this works.

I've tried both of those suggestions to no avail.  The only way to configure our SBG to send cleartext SMTP to the remote domain is to disable TLS globally in the SMTP advanced configuration.

In order to rectify this problem, I've had to go a step further back into our Exchange server and set up a custom non-opportunistic TLS connector to this specific domain.

Basically, I've tried all the combinations available in the picture below, none of them will disable TLS and allow the SBG to "fail open"

I may have to abandon my efforts on this one soon, because it sounds like this simply isn't an available feature in SBG.

I'm sorry this is causing so many issues for you. What I'm suggesting at this point is for you to call technical support because this "should" work. If it is not, we are either missing something or there is something wrong with the Brightmail Gateway.

What should happen is that we will issue the STARTTLS command but if this fails for whatever reason, we should fall back to a clear text email as long as the receiving server can do so. Since the receiving server can, then we should be working. But we aren't.

So I would highly suggest that you give us a call.

Thanks for your time, and I'm sorry this wasn't an easy resolution.

Tom

No problem.  Thanks for your help anyways... I'll open a support case :)

Submitted as Case # 320-205-248 in case anyone comes across this thread in the future encountering the same problem :)

I just had a notification about an upgrade available for SBG.  We're currently running 8.0.1-7.  The upgrade available is for 8.0.2-12.

From the release notes:

Email TLS encryption now working properly

--In Symantec Brightmail Gateway 8.0.1, TLS encryption of email messages was
not working properly. This issue has now been fixed.

I'm going to install this update tonight and see if the problem still exists

So this issued had to do with the TLS cert bundle offered with the TLS connection. Hopefully this will address the issue you are seeing, but I'm not 100% that it will. Keep us posted.

-Tom 

Well, as predicted Tom, the update did not solve the problem.  I've sent a packet capture and diagnostics logs off to support.