Messaging Gateway

Good afternoon,

I have been dealing with a problem for a few weeks that has started to become a major distraction to our employees...I'm wondering if someone can help shed a little light on this for me. A few weeks ago I upgraded to 8.0.3-11 without any issues that I noticed, just another upgrade to check off the list. Within 24-hours of the upgrade myself and a few other users were getting what looked like NDR-delay-spam and I just figured a few had snuck through the filter for one reason or another without looking closely at the messages.

Then every few hours the same messages would pop back up as delayed again. I looked closely and noticed that we were getting the generic delayed message (5 hours, keep trying for 4 days), so I started digging on the forums and found a few posts that sounded like our same problem (granted they were using v7.7). I followed a recommendation in this post to change the delay, which worked to stop the delay messages...but we still get the final NDRs. I had hoped after a day the message would disappear and they are still coming in.

Every user gets between 5-20 NDRs per day and I cannot figure out what is going on (source of one of the emails is below). It looks like there is a NON-NDR, typical, nothing fancy spam email that is sent to a good email address from that same email address...so far so good, typical simple spam. However, it looks like the Brightmail server is the one sending the "legit" NDR report to the good internal email address. We are seeing no other email issues, we can send and receive with no problem. According to the status screen, for a few years now we have received around 500,000 emails into Brightmail and less than 2% are legit...so these NDRs are not coming through for all spam messages, just a small portion.

Am I reading this wrong, or are these NDRs coming from Brightmail?
Any thoughts/ideas as to why?
Any thoughts/ideas as to what I can do to stop it?

I appreciate any and all help!
- Justin


Configuration:
Less than 25 internal users
Only filter outbound mail
Internet -> Firewall -> Brightmail -> Exchange 2003

 

Email source, scrubbed with **info**
-----------------------
Received: from **brightmail server** ([x.x.x.x]) by **internal email server** with Microsoft SMTPSVC(6.0.3790.3959);

Thu, 4 Feb 2010 12:39:46 -0500

From: <Mailer-Daemon@**brightmail server**> (Mail Delivery System)
To: **my work email**
Subject: Undelivered Mail Returned to Sender
Date: Thu, 04 Feb 2010 12:39:44 -0500
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
 boundary="kJoKGlomkv+5ezQKeX9JgGmyQuSueZKVxJY4lA=="
Return-Path: <>
Message-ID: <**internal email server**>

X-OriginalArrivalTime: 04 Feb 2010 17:39:46.0937 (UTC) FILETIME=[0B3E1690:01CAA5C1]

This is a MIME-encapsulated message.

--kJoKGlomkv+5ezQKeX9JgGmyQuSueZKVxJY4lA==

Content-Description: Notification

Content-Type: text/plain


I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.
For further assistance, please send mail to <postmaster>

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

<**my work email**>: 554 5.4.7 [internal] exceeded max time without delivery

--kJoKGlomkv+5ezQKeX9JgGmyQuSueZKVxJY4lA==

Content-Description: Delivery report
Content-Type: message/delivery-status

X-Symantec-Brightmail-Gateway-Queue-ID: 9E/FC-02599-2F89A6B4
X-Symantec-Brightmail-Gateway-Sender: rfc822; **my work email**


Reporting-MTA: dns; delivery
Arrival-Date: Thu, 04 Feb 2010 12:39:44 -0500

Final-Recipient: rfc822; **my work email**


Status: 5.0.0
Action: failed
Last-Attempt-Date: Thu, 04 Feb 2010 12:39:44 -0500
Diagnostic-Code: smtp; 554 5.4.7 [internal] exceeded max time without delivery

--kJoKGlomkv+5ezQKeX9JgGmyQuSueZKVxJY4lA==

Content-Description: Undelivered Message
Content-Type: message/rfc822

X-AuditID: c0a8011b-b7c6dae000000a27-00-4b6a98f0125b

Received: from cpe-67-252-144-193.buffalo.res.rr.com (Unknown_Domain [192.168.1.1])

 by **brightmail server** (Symantec Mail Security) with SMTP id 3B.FC.02599.1F89A6B4; Thu,  4 Feb 2010 04:52:50 -0500 (EST)

To: <**my work email**>

Subject: [Spam] RE: Order 64666 for 

From: Approved VIAGRA® Store <**my work email**>

MIME-Version: 1.0
Content-Type: text/html
X-Bmi-Source: external
X-Brightmail-Tracker: AAAAAhICaLsAAAFh

 Sorry, I just realized that looking up the message in the audit logs might help.



Recipient Data Intended recipient: **my work email**
Verdict: Verdict Filter Policy Group Details Spam
spam: modify subject line with"[spam]"(default) default None
Actions taken:Modify the subject line, Hold message in Spam Quarantine
Delivery: Delivery Failure
Failure Time Attempted Delivery to: **brightmail server**
Error: 554 5.4.7 [internal] exceeded max time without delivery
Thursday, Feb 04, 2010 12:39:44 PM EST

Untested verdicts: Suspected spam, Message was sent from a suspect spammer, Locally identified suspected virus, Suspected virus, Content Compliance violation: Delete Executable Files Violations, Content Compliance violation: Delete Email Policy Violations, Content Compliance violation: Legal Disclaimer, User allow, User reject, Unknown recipient, Connection Class, Default Connection Class, Connection Class 1, Connection Class 2, Connection Class 3, Connection Class 4, Connection Class 5, Connection Class 6, Connection Class 7, Connection Class 8, Connection Class 9, Bounce attack signature present, Blocked language, Known language

Based on the data provided here is my understanding:

It seems like the original message that came in from outside to your work address was spam and it triggered the spam policy that marks the subject with [spam] and sends the message to quarantine.  But for some reason, this message is not being delivered to the quarantine and ultimately triggering the NDR which goes to the sender and since in this case the sender was spoofed as yourself, the NDR is going to you.

So the question is why is the original mail that is being tagged as spam and supposed to go to quarantine not going to quarantine? 

One possible reason could be that the quarantine database has crashed or somehow not accepting messages sent to it.  An indication of this issue would be that you are NOT getting any new messages in the quarantine.  Also you will see some error messages in BrightmailLog.log indicating problem with the tables/database.  If that's the case, then perform a mysqlcheck and repair on the database.  For this procedure, you will need to contact support as this procedure can only be performed using the support account.

Another possibility could be that the control center threads that accept messages are hung.  May be a restart of the control center will help (using the CLI):

service controlcenter restart

As for preventing the spoofing of your internal addresses in emails coming from outside, please see the following KB article:

http://service1.symantec.com/support/ent-gate.nsf/docid/2008111714541154

Regards,

Adnan

 Thank you for the response!

We are still getting new messages in the quarantine as far as I can tell. I just looked in the spam quarantine  and we do have messages that have been added since the problem started. I assume that's a reasonable way for me to check, is there another way for me to check?

I just went into the CLI and restarted the process. I'll be sure to post back if that solved the problem.

Thanks for the KB article, I almost tried blocking our domain as a sender...but for some reason I just thought that was too easy.

- Justin

When you try to release or delete a message from the spam quarantine, you will see an error saying "An unexpected database error has occurred. Please contact your system administrator." in the Control Center, if there is a problem with the database.  You may also see one or more of the following, or similar, error messages in BrightmailLog.log file:

Feb 03 2010 20:03:46 [SmtpConsumer_0000_000] ERROR - Cannot create a spam message in the database.
com.brightmail.common.BrightmailException: Cannot create a spam message in the database.
...
Feb 03 2010 20:07:47 [SmtpConsumer_0000_000] ERROR - Cannot create a spam message in the database.
com.brightmail.dl.DataAccessException: An unexpected database error has occurred. Please contact your system administrator. ; nested exception is:
         java.sql.SQLException: Duplicate entry '15' for key 1
        at com.brightmail.dl.impl.DAOHelper.executeUpdate(DAOHelper.java:511)
        at com.brightmail.dl.impl.QuarantineDAOImpl.createSpamMessage(QuarantineDAOImpl.java:146)
...
Feb 03 2010 20:07:47 [SmtpConsumer_0000_000] ERROR - Cannot create a spam message in the database.
com.brightmail.common.BrightmailException: Cannot create a spam message in the database.
...
Feb 03 2010 20:49:15 [http-41443-Processor4] ERROR - Cannot read the spam message details from the database.
com.brightmail.dl.DataAccessException: An unexpected database error has occurred. Please contact your system administrator. ; nested exception is:
         java.sql.SQLException: Table 'spam_message' is marked as crashed and should be repaired
...
Feb 04 2010 01:00:09 [BrightmailScheduler_Worker-2] ERROR - Cannot delete messages older than a specified date.
com.brightmail.dl.DataAccessException: An unexpected database error has occurred. Please contact your system administrator. ; nested exception is:
         java.sql.SQLException: Table 'spam_message' is marked as crashed and should be repaired
...
Feb 04 2010 18:19:59 [http-41443-Processor3] ERROR - com.brightmail.common.BrightmailException: Cannot read the spam message details from the database.
...
Feb 04 2010 18:20:21 [http-41443-Processor3] ERROR - Cannot delete the selected spam messages from the database.
com.brightmail.dl.DataAccessException: An unexpected database error has occurred. Please contact your system administrator. ; nested exception is:
         java.sql.SQLException: Table 'spam_message' is marked as crashed and should be repaired

All these error messages indicate a problem with the database.

You can view the BrightmailLog.log file from the Control Center as well (click Status > Logs; then select Control Center as the component and choose BrightmailLog.log from the displayed list of log files).

Hi Justin,

Has the issue been resolved for you?

Thanks

Adnan

Justin,  can you please provide and update?

Thanks

Adnan