Catch Rate vs. Effectiveness

Spam represents as much as 75% of all email sent across the Internet. The variance of this number is representative of different regions that are impacted more or less by spam senders, as well as the ever-increasing deployment of IP-based solutions to deal with spam before it is allowed to reach an MTA. Symantec has been benchmarked at greater than 99% anti-spam effectiveness for all spam. Anti-spam effectiveness is defined by the percentage of spam that is identified as spam by an anti-spam solution. This is separate from the catch rate, which is the measure of the percentage of all mail messages that have been identified as spam.

To illustrate this, consider a typical mail stream of 100 messages:

1.  70 messages are spam (based on latest Symantec trend analysis of Internet mail).

2.  Symantec Messaging Gateway successfully identifies 66 messages as spam.

3.  The spam effectiveness is 94.29% (66/70 spam messages).

4.  The catch rate is 66% (66/100 messages).

It is critical that customers do not confuse effectiveness and catch rate when considering the performance of Symantec Messaging Gateway solutions.

Events that Change Spam Threat Landscape

Over the last few years, there have been several events that changed the spam threat landscape.  McColo shutdown (November 2008), Bredolab botnet and Spamit.com shutdown (October 2010), and Rustock botnet shutdown (March 2011) are few examples of major events that resulted in a large drop in overall spam volume.  The drop in overall spam volume in turn causes spam effectiveness and catch rate to drop lower as well.

To illustrate this, consider a bot takedown that eliminates 20% of spam.  Using the above example of a mail stream as a basis, the new calculation is as follows:

1. 14 spam messages (20% of 70 spam messages) have been eliminated as a result of the bot takedown.  Mail stream now consists of 86 messages (100-14 messages) and 56 messages (70-14 messages) are spam.

2. Symantec Messaging Gateway successfully identifies 52 messages as spam.

3. The spam effectiveness is 92.86% (52/56 spam messages).

4. The catch rate is 60.47% (52/86 messages).

Even though Symantec Messaging Gateway missed 4 spam messages in both examples, spam effectiveness and catch rate are lower in the latter example with a bot takedown.  When there are major events that cause a large drop in overall spam volume, both spam effectiveness and catch rate percentages will drop as a result even when there is no actual difference in effectiveness.

Alternative Ways to Measure Effectiveness

In addition to using spam effectiveness and catch rate percentages, there are other metrics that can be helpful in determining true effectiveness of the product:

1. Spam in the Inbox:  Monitoring the number of spam messages in a set of control group accounts.

2. Number of messages with no verdict: In both examples above, 34 messages had no verdict, indicating steady effectiveness.

3. Number of missed spam submissions: Tracking end-user complaints over a period of time.