Video Screencast Help
Security Community Blog

Accuracy and DLP Revisited

Created: 30 Oct 2008 • Updated: 02 Mar 2009
Kevin Rowney's picture
0 0 Votes
Login to vote

There’s been a recent run of blog postings  here and here reinforcing the fundamental importance of detection accuracy in DLP.  We fundamentally agree.  Starting in 2001, and over the course of the past seven years, many of the defining characteristics of what is now considered Data Loss Prevention were brought to the market by the DLP division of Symantec (once known as Vontu).   New breakthroughs in detection accuracy were among these category-defining innovations that in fact made big swaths of the DLP space viable.


Delivering DLP Solutions Looks a Lot Easier Than it is

As Heather Schneider (of RSA) indicates in her post on accuracy: the two major components of accuracy are precision and recall.  The term “precision” relates to how well detection algorithms perform on reducing false-positives.  The term ”recall” relates to how well detection algorithms perform on reducing false-negatives.  Basic algorithms like keywords and pattern-matching have pretty fundamental limits on these two fronts and force difficult tradeoffs between the two.    With this first-generation of detection technologies, customers would have to choose EITHER low-false positives OR low-false-negatives. Getting both good precision and good recall simultaneously using such blunt tools is, for most practical use-cases (like protection of Intellectual Property or customer data) simply not possible.  Solutions based on simplistic assumptions and basic detection algorithms flailed in the market or saw limited appeal to a narrow range of use-cases. 


Amazingly, there are still vendors out there trying to sell solutions based only on these very limited approaches.   Not only do these vendors ship limited detection capabilities, but they are additionally limited on the scope of what events they can detect (i.e.” email only”, or “endpoint only” solutions.) Our systems are frequently deployed with customers who tried to use these limited capabilities from other vendors.  We are seeing a clear trend now with these partial solutions from our competition:  the wave of false-positives is too painful to endure and the false-negatives are too risky to ignore. 


What it Takes to Deliver Accuracy in DLP

A summary list of the key features that produce sufficient accuracy that results in an effective DLP solution follows:


Scale – Slow DLP solutions that can’t keep up with volumes of data and flows of data will frequently miss confidential data exposures.  Detection recall problems are pretty common in many DLP vendor products because of their inability to handle load.


Coverage – DLP solutions that don’t cover all of the major vectors of threat (data-in-motion, data-at-rest, and data-at-the-endpoint) will have no opportunity to observe crucial events that may cause significant risk.  Despite the many vendors in DLP, there’s a very short list of players that can cover this full swath of threats.  There’s no other DLP vendor that has invested as much as Symantec has in breadth of coverage, and poor coverage can really hurt detection recall.


Algorithms – Core use cases like customer data theft, intellectual property exposure, or PCI compliance each  require specific detection algorithms that are built specifically around these threats.   There’s no DLP vendor that can cover all three of these threats at huge data volumes with near-perfect precision and recall.  No vendor, that is, except for Symantec.



Myths From the “The Tolly Report” on DLP Accuracy

I talk to customers a lot and have spoken to a few recently who seem to be getting copies of “The Tolly Report on Accuracy” as evidence of RSA/EMC/Tablus’ superior accuracy to other vendors.  Unfortunately, the report contains huge misrepresentations and purports to be about our product when the authors of the report never even ran our software. 


The problem with the report  is that the Tolly group ended up studying the wrong product!  Some years ago, before Vontu was acquired by Symantec, we licensed a tiny sub-set of our detection algorithms to Symantec.  The resulting product (Symantec’s email gateway plus a tiny fragment of Vontu’s detection technologies) was the system the Tolly Group ended up evaluating  against the RSA/EMC/Tablus platform.  The “results” indicated RSA/EMC/Tablus had the edge on accuracy compared to this very limited system.  Of course, this study has no relevance, at all, to the Symantec Data Loss Prevention product offered now nor was it relevant to the full Vontu product available at that time this report was published.


If You Are Evaluating a DLP Solution Right Now

If you are evaluating DLP solutions, accuracy is one of the primary evaluation criteria.  I’m sure it’s very confusing getting so many different claims from each of the vendors, so instead of listening to any more “spin” from any vendors (Symantec included) I offer this simple suggestion: just line us all up and try us out.  We’re confident that the significant differences between DLP vendors can be made clear in your own environment using your own data.  There’s no reason to listen to vendor hype out there, simply let the facts tell the story.   In a head-to-head bake-off, the leading vendor will clearly stand out.