Symantec Messaging Security Effectiveness User’s Guide
|Article:TECH90453|||||Created: 2008-01-07|||||Updated: 2013-02-13|||||Article URL http://www.symantec.com/docs/TECH90453|
I need more information regarding the effectiveness and what to do when the spam percentage is increasing.
Spam represents as much as 75% of all email sent across the Internet. The variance of this number is representative of different regions that are impacted more or less by spam senders, as well as the ever increasing deployment of IP-based solutions to deal with spam before it is allowed to reach an MTA. Symantec has been benchmarked at greater than 99% anti-spam effectiveness for all spam. Anti-spam effectiveness is defined by the percentage of spam that is identified as spam by an anti-spam solution. This is separate from the ‘catch rate’ which is the measure of the percentage of all mail messages that have been identified as spam.
To illustrate this, consider a typical mail stream of 100 messages.
- 64 messages are spam (based on latest Symantec trend analysis of Internet mail.)
- Symantec Messaging Gateway successfully identifies 60 messages as spam
- The spam effectiveness is 93.75% (60/64 spam messages)
- The catch rate is 60% (60/100 messages.)
Symantec uses multiple methods to measure its anti-spam effectiveness:
- Control accounts at global service provider customers, including our Probe Network partners. These accounts provide Symantec with a direct measure of effectiveness, against a statistically significant number of accounts monitored in customer environments.
- The catch rate of the Global Intelligence Network. The GIN is made up of millions of email accounts that receive exclusively spam messages. Symantec Security Response measures the number of spam messages in the GIN that are correctly identified as spam.
- Missed spam submissions. The Email Security Group within Symantec Security Response analyzes the number of missed spam submissions from our customers. This provides direct customer feedback on the number of messages missed relative to the aggregate message flow through all mailboxes protected by Symantec mail security products.
End User Expectations
End-user experience is typically what customers refer to when discussing spam filter effectiveness. No single inbox or small group of inboxes can by themselves be an accurate gauge for measuring overall spam filtering effectiveness. One end-user may find their experience to be poor, while another finds spam filtering to be very effective. Symantec, and other antispam vendors, cannot guarantee the same effectiveness for every end-user's experience, since different users receive different types and volumes of spam.
End-users also have different opinions as to what constitutes spam. The definition of spam is very subjective to most end-users. Many end-users define spam as simply unwanted email (including legitimate advertisements that they no longer wish to receive). Symantec defines spam as Unsolicited bulk email (includes Unsolicited Commercial Email).
Many end-users, customers and even analysts are actually referring to spam in a broader sense as all unwanted communication.
Symantec does not include the following in its definition of spam:
- Unwanted direct marketing emails that have been solicited by the recipient
- Unwanted newsletters that have been solicited by the recipient
- Unwanted transaction emails, for example, receipts, confirmations, account statements, and similar items
- Hoaxes, urban legends, jokes, chain-letters sent by users known to the recipient
- Challenge/response emails
- Messages sent to the recipient in error
- Email bounce notifications and errant worm notifications
30-45% of all of missed spam reported by Symantec customer end-users is not spam according to Symantec’s definition.
Symantec’s antispam technology is focused on stopping true spam messages. Symantec also provides administrator and end-user tools to enable them to block unwanted messages. These tools include web based personal Allowed and Blocked Senders Lists as well as New Disposition verdicts available in the Symantec Messaging Gateway. See the following tech note for additional info on New Disposition verdicts:
If Symantec maintains the same effectiveness ratio (of spam caught vs. spam missed) but the total volume of spam increases, the end-user will experience a perceived drop in effectiveness. For example, one missed spam message out of ten total spam messages equates to 90% effectiveness. If the total volume of spam received increases from 10 spam messages to 100 spam messages, the effectiveness remains 90%. However the end-user perceives that the product is less effective, as there are now ten missed spam messages, compared to the one missed spam message previously. Therefore the volume of mail received by end-users is critical in understanding their perceived spam filtering effectiveness rate.
Steps to Follow if Seeing Increased Missed Spam
If spam effectiveness seems to have dropped, there are troubleshooting steps you can perform and information you can gather that can help determine where the issue may be. Please review your specific product documentation for details on how to investigate the following troubleshooting steps.
Use the following basic troubleshooting steps :
- Verify that you are running as many different antispam rule types as possible. Ensure that all rule types you have specified to use are currently running.
- Assure that the spam messages are not bypassing your Symantec servers. Check Received-from IPs.
- Confirm that the rulesets are current at the time the missed spam messages came through. Check to see that your rulesets are updating across the board.
- Verify that none of the Symantec Messaging Gateway services (Server, Client, or Conduit) were down when these messages came through. Verify that the various components and modules are functioning with no errors reported in the logs.
- If the Allowed Senders List or Safe Senders List IP services are enabled, ensure none of the senders of the missed spam messages are on those lists. Some troubleshooting steps may require you to temporarily change the log levels to INFO or DEBUG in order to see sufficient data in the logs. Be sure to reset the log levels to lower levels once you have completed troubleshooting to avoid incurring unnecessary overhead from verbose logging. Gather the following information before contacting Support:
- Note the time period that the suspected spike in missed spam occurred and ensure that you are securing the most recent spam messages for submission.
- How you are submitting samples of missed spam to the Symantec Security Response Center? See the information below.
- How you are tracking the increase in spam?
- Are these end user inbox complaints, management complaints or statistical in nature?
- Have all available software updates or patches been installed?
- Have you made any other changes to your environment that might have contributed to effectiveness issues? This includes server, OS, or datacenter changes. It also includes changes made to Symantec or other products in the mail stream that might negatively impact effectiveness.
Installing Software Updates and Patches
Symantec mail security products have the ability to react to most new attacks via new filters that use existing technologies. However, over time, Symantec introduces new anti-spam technologies into its products to deliver new capability. It is critical that customers evaluate new versions of Symantec mail security products, since some new spam attacks can only be caught with them. If you are experiencing lower spam effectiveness, you should consider upgrading to the latest version of your Symantec technology. Customers should plan to deploy the latest release to ensure the highest levels of antispam effectiveness.
You can obtain custom spam rules specifically for your organization based on the missed spam messages and false positive messages that administrators and end users submit.
See the following documents for additional info on Customer Specific Spam Rules:
Setting up customer-specific spam submissions: www.symantec.com/docs/HOWTO77719
About submitting messages for customer-specific spam rules: www.symantec.com/docs/HOWTO77718
Missed Spam Submissions
If you have followed the troubleshooting and information gathering steps outlined above and determined that the increase in missed spam is not related to configuration or version issues, then you should consider making a missed spam submission. Missed spam submissions are used by Symantec for the following:
- Antispam technology and effectiveness research
- Emerging threat research
- Internal reporting and data mining
- Antispam filter development
Submissions must be received within one day from the time they were initially sent. Since spammers rarely reuse old spam, Symantec does not accept submissions older than 24 hours. Submissions are processed using sophisticated algorithms. This process groups the message with other messages received from customers or through the extensive Global Intelligence Network. When a group reaches a threshold, it becomes an attack. At this point, the automation systems or an Email Security Analyst creates a rule to respond to the attack. Adding the rule to the rule set completes the process. Your computer becomes protected when your rule set is updated.
However, due to the volume of submissions received (approximately several million messages per day), Symantec cannot guarantee that filters will be written for particular submissions. Because many submissions contain a forged sender address, they cannot provide feedback for submissions.
How End Users Submit Missed Spam
The customer creates an alias for the appropriate Symantec-Brightmail missed spam address:
- North America: Gsubmit@submit-1.brightmail.com.
- EMEA: firstname.lastname@example.org
- APAC: email@example.com
- Japan: firstname.lastname@example.org
Note: Only missed spam messages are sent to this address. If your deployment is over 50,000 users, then unique submission addresses for missed spam and FP’s can be created.
The missed spam must be sent as RFC-822 MIME encoded attachments in order for Symantec Security Response to process the mail. Information on submitting messages is available in the following tech note:
Many spam messages look the same from the initial appearance, but contain many hidden characteristics to make the messages unique.
A few sample characteristics include:
- Hidden HTML comments or undefined HTML tags
- Using text that is the same color as the background (or nearly the same color – camouflage)
- Use of extremely tiny fonts placed strategically throughout the message
- Images that introduce randomized changes (text moved, color changed, image rotated slightly, different borders, etc.)
If end users encounter multiple missed messages that seem to be related, they should report them to Symantec, following the procedures outlined above.
There is no Service Level Agreement for missed spam and/or effectives issues. Escalations are not handled during weekends or non-business hours outside of U.S Pacific Time.
Note: Customers of Symantec OEM's, 3rd party vendors, and/or appliance partners that are not direct Symantec Brightmail Gateway customers need to go through their vendor who can contact the appropriate support agent to assist in this process. Those customers should not contact Symantec Support directly.
Probe Accounts and the Global Intelligence Network
Symantec’s Global Intelligence Network (GIN) ™ is a vast collection of email accounts. The patented GIN is built on a base of over 2 million accounts donated by service provider and enterprise customers, as well as accounts owned by Symantec. It is one of the key reasons why Symantec Messaging Gateway is the leading solution for accurately stopping spam.
Why the GIN is Important
The GIN is crucial to Symantec and its antispam customers for a variety of reasons:
- Drives early detection of spam attacks. Probe accounts are the first step in the real-time detection and analysis of spam. The structure of the GIN essentially provides Symantec Security Response a stream of real-time spam that is being disseminated over the Internet. This virtual “net” of numerous accounts spread all over the Internet makes it easy for Symantec to verify that a given message was sent using bulk methods. When the same questionable message is caught by different probes, alarms go off and Symantec can take action.
- Speeds the development of accurate filters. The key marketplace differentiator for Symantec messaging products is the near-perfect accuracy rate. Symantec messaging products boast a high accuracy rate because many of its core filters are based on actual spam. The Global Intelligence Network also provides key data that is used to develop Symantec's more predictive filters, such as heuristics. What makes all this possible is the volume, quality, and timeliness of data that flows in real time from the GIN to Symantec Security Response.
- Aids ongoing trend research. Spammers are constantly changing their tactics and dissemination methods to evade filtering software. Symantec’s Email Security Group mine data from the GIN to advance Symantec's antispam technology. Examples include staying abreast of the latest spam trends, evaluating the spam-catching differences between product versions, and monitoring detection rates in different languages.
Symantec Messaging Gateway strives to maintain a false positive (FP) rate of less than one FP in one million messages scanned. Symantec utilizes several methodologies to determine our FP ratio with a conservative estimate to account for data that is not reported.
The caveat with field data is that not 100% of end-users report FP’s. The other issue is that some customers elect to delete detected spam and therefore do not have the ability to report FP’s. Understanding this we can compare an aggregate FP ratio (used to establish a baseline) to a FP ratio of only those domains that reported FP’s. We compare the total number of reported legitimate FP’s to the total number of messages scanned. These numbers usually average to approximately 1 FP for every 20-35 million messages scanned.
How End Users Forward False Positives
The mail administrator creates an alias for the address:
- North America: Gfeedback@feedback-1.brightmail.com
- EMEA: email@example.com
- APAC: firstname.lastname@example.org
- Japan: email@example.com
Note: Only false positive messages are sent to these addresses. If the customer has more than 50,000 users, a unique submission address for missed spam and FP’s can be created End users send FULL HEADERS and BODY in the message as a RFC-822 MIME encoded attachment in order for Symantec to investigate and process the message. A copy of the message may also be forwarded to the customer’s Support Desk. Symantec investigates and adjusts filters, as necessary.
A: Symantec does not consider this as a true false positive. All three rulesets are optional features that customers opt in at their own discretion. If you encounter such “false positive”, you can do any of the following:
- Opt out the recipient from a group of users opted into the ruleset
- Disable the feature
Symantec targets the largest phishing email threats with gateway email detection. The primary differences between phishing and spam are the ability of phishing attacks to be a) very small and b) difficult to distinguish from legitimate direct e-mail communications. Phishing attacks deployed using spamming techniques are readily detected and stopped, but attacks that are targeted and presumed legitimate are difficult to discern from actual communications from banks or credit card issuers. Symantec endeavors to be as effective against these threats as we are with spam by utilizing our premium antispam technology to capture them.
Article URL http://www.symantec.com/docs/TECH90453