Video Screencast Help

Symantec Messaging Gateway / Delivery Queue Continually Filling

Created: 09 May 2013 • Updated: 10 May 2013 | 6 comments
This issue has been solved. See solution.

Greetings Symantec Support Forum,

We have a Symantec Messaging Gateway currently running version 9.5.3-3,  the gateway forwards mail to our Exchange Server 2010

We had a scare the other day when external email stopped flowing in.  We found that this was being caused by the SMG's Delivery Queue being filled with tens of thousands of what seem like SPAM messages.  After clearing the delivery queue external mail started to flow back in.

The issue now is that the delivery queue continues to accumulate about 100 of these per hour. They all have similar errors listed:

421 Closing Connection

451 4.4.2 [internal] send BODY failed

451 4.4.2 [internal] send RSET failed

451 4.4.2 [internal] connection closed by remote host

They are all To: a valid internal address and From: an external address (some dont seem valid), the route is listed as Control-Center

 

Is something misconfigured?  I can continue to clear the delivery queue but why are these messages getting stuck in the first place?

Any help or insight you can offer would be greatly appreciated,

Thank you,

 

 

 

Operating Systems:

Comments 6 CommentsJump to latest comment

toby's picture

Hello Rhombus,

first of all I would recommend you to move to SMG version 10 as there have been a couple of improvements in terms of Antispam.

The problem you describe is a bit to generic to narrow down to what exactly it is.

From a mail flow perspective the question is whether your mail flow is correct and you dont have any loops. I would also say, to have a couple of messages in the delivery queue is quite common in a real world production environment, but if most recipients are valid internal recipients and you route from a scanner just to Exchange, I would check whether there is any restrictions that the SMG cant route mails to Exchange properly what creates this huge queue. (In case you can set an alert for queues size that might help you getting a notification before the situation gets critical)

If all the messages are spam I would be interested whether you take advantage of all reputation and spam filters the SMG provides by default as to have this scenario only by inbound spam looks like a spam attack or not using common filters we provide, but in that case you should check as well whether you see any anormality in terms of IP connections or Internet volume what will give a good understanding whether you were targeted by a spam attack(In that case you may want to take advantage of the customer specific rules in SMG v. 10)

Hope this helps

toby

------------------------------------------------------------------

Best regards!

toby

CISSP / STS / MCP 

TSE-JDavis's picture

It sounds like there is a firewall between the SMG and yoru email server and it is performing some kind of email scanning. If the SMG missed a bunch of spam it might think the SMG is a spammer and start throttling it.

RHOMBUS's picture

TSE-JDavis, we do have a firewall between the appliance and the exchange server, I'll check that for SMTP inspection.

Toby, I'll try to upgrade and set alerts.  There shouldn't be any loops going on.

Could this have anything to do with TLS?  I get this error in Exchange about TLS

Event 12014, MSExchangeTransport
Microsoft Exchange could not find a certificate that contains the domain name mail.companyx.org in the personal store on the local computer. Therefore, it is unable to support the STARTTLS SMTP verb for the connector Default with a FQDN parameter of mail.companyx.org. If the connector's FQDN is not specified, the computer's FQDN is used. Verify the connector configuration and the installed certificates to make sure that there is a certificate with a domain name for that FQDN. If this certificate exists, run Enable-ExchangeCertificate -Services SMTP to make sure that the Microsoft Exchange Transport service has access to the certificate key.

I've tried several suggestions to clear that error with no avail.  We do have valid certificates.

Just for kicks today I ran:  Enable-ExchangeCertificate -Services SMTP

When it promtped me for the thumbprint I pasted the certificates thumbprint

I'll monitor both Exchange and the SMG Delivery Queue and see if anything changes.

Thanks guys,

RHOMBUS's picture

After upgrading to 10.0.1-2  all mail flow stopped.  Any ideas?

Art_P's picture

Based on what you originally wrote, I would say this probably doesn't have anything to do with your Exchange server or its TLS configuration.

The information that I understand is that:

1. Delivery queue fills up, many of the items are destined for the Control Center.

2. Clearing the queue corrects the issue for a while, but then messages start building up again and again many are destined to be delivered to the Control Center.

The primary culprit(s) will likely be quarantine and possibly DDS. I would review your DDS logs and BrightmailLog.log files for more direct errors.

Status > Logs > Component: Control Center

Status > Logs > Component: Directory Data Service

You may also want to do a health check on the database. Depending on the size of the data in the database, the command may take some time. However, an overly large database can also cause performance issues, so if it is too large, you may want to prune data and then run the health check. You woul use the command line, logged in with the "admin" account, and issue the following command:

cc-config database --repair

You should see most tables return as OK with a few showing "note     : The storage engine for the table doesn't support repair", which are appropriate responses. If you see anything different, you may need to run the repair command again. Up to 3 times at most. If there are problems beyond the third run, then there may be deeper issues with the database.

If you see any errors related to the Directory Data Service, you should be able to find an answer by searching for the error found.

If you find a useful post, please use the "thumbs up" feature to mark the post as helpful. If your question has been answered, please select the "Mark as solution" for the post that best answered your question. Thank you.

SOLUTION
RHOMBUS's picture

Update:

I was able to get mail flowing again by setting the DNS to Use Internet Root servers

Thank you Art_P 

I didn't run the health check yet, I'll do that after hours but I did look at the logs and found:

May 10 2013 13:46:27 [SmtpWorker-12] ERROR - Quarantine failed to resolve email address janedoe@companyx.org. DDS has encountered an error. 

 

com.symantec.smg.controlcenter.BrightmailException: null ; nested exception is: 
com.symantec.smg.controlcenter.directoryintegration.DDSClientException: Authentication to LDAP server unsuccessful
 
That lead me to the LDAP settings under Directory Integration.  Long story short the account used for the LDAP connection had it's password changed a while back and that was causing directory lookups to fail. I entered the new credentials and will monitor for improvment.
 
UPDATE: That was it Art_P  after giving it a few minutes for the system to catch up the delivery queue is clean!
 
Thank you much!