On Monday, February 4th, Symantec and morethan 40 security software technologists and anti-malware testersannounced the first steps in creating the Anti-Malware TestingStandards Organization, or AMTSO. It’s been an interesting road to get here so I thought it would be interesting to chronicle a bit of the back story here.
Last May in Reykjavik, Iceland I gave a presentation on a new, morerelevant, form of anti-malware testing called "dynamic testing"(http://www.slideshare.net/frisksoftware/active-testing/download). Thiswas borne out of the fact that even though our security suitescontained numerous protection technologies, only one – static filescanning – was being evaluated in tests. While static file scanning iscertainly an important part of a security suite, taking the results ofthat single technology and extrapolating the total effectiveness of thesuite was equivalent to taking the results of seat belt tests andextrapolating the total safety of a motor vehicle. Others agreed.
The following evening a number of people representing five differentcompanies (Symantec, Panda, F-Secure, Kaspersky and AV-Test.org)gathered in the lobby of the hotel and discussed the problem and moreimportantly, a solution. We agreed that an industry coalition comprisedof competitors and focused on an industry wide problem was the way togo. It was decided that each of the five companies would produce awhitepaper describing how they felt the problem should be solved. Wewould then collectively combine those papers, keeping what was inagreement and using the differences as the basis to begin tackling theproblem. By keeping the group somewhat limited in the beginning, wecould move faster. Once we had reached basic agreement amongstourselves, we would reach out to the wider community.
Three papers were produced by the time August rolled around and theywere remarkably similar. In September at the VB 2007 conference inVienna, we met again to decide what our next steps were. Andreas Marxat AV-Test.org had a paper on testing methodologies accepted for theAVAR conference in Seoul, Korea, at the end of November. We would usethis forum to publicly describe the problems and solutions we hadderived to date.
Andreas’ presentation was generally well received, with a number ofnew people and companies asking some direct questions. That evening,over a very Korean dinner (none of us knows quite what we ate),representatives from Symantec, F-Secure, Virus Bulletin, Sunbelt,Panda, and AV-Test decided that the time was right to formally create agroup comprised of security vendors, testers, reviewers, academia, andconsumer advocates to address the problems with testing. Panda Securitygraciously offered to host the event, which would be scheduled for lateJanuary 2008. We all felt that it was important to maintain ourmomentum and get this group formed.
Six weeks later a group of 43 individuals from over 25 organizationswere seated around a table. All had been drawn by the universalrecognition that current tests – even when performed correctly – simplywere not relevant. We each knew that, on our own, complaints aboutindividual tests were seen as whining. Viewed through the lens of anindividual company, problems with testing methodology went largelyignored. What was needed was an industry voice, comprised of fiercecompetitors, sounding a united theme.
Over the next day and a half we discussed, argued, joked, and most importantly, agreed (unanimously) on a mission statement and charter. The result is what we collectively and officially announced this week, the Anti-malware Testing Standards Organization, or AMTSO.
Now, with the hard part of forming the group behind us, we move onto the really hard work of forming the standards. It is my belief thatwe will, over the course of this year, develop standards that willdeliver on our mission statement of “improving the objectivity,quality, and relevance of Anti-Malware testing.” That will then onlyleave the really, really hard part of convincing magazine reviewers andeditors that old ways are fatally flawed and that the new standards arewhat is best for the people who buy and use security software.
Our next meeting is scheduled for late April 2008 and I will keep you posted as we make progress.