Who Will Test The Testers? is a paper by myself and Andrew Lee on making anti-malware testers more accountable to their audiences, presented at the Virus Bulletin Conference in 2008 and published in the conference proceedings.
David Harley BA CISSP FBCS CITP & Andrew Lee CISSP, “Who Will Test The Testers?”, October 2008, Virus Bulletin. Copyright is held by Virus Bulletin Ltd, but the paper is made available on this site for personal use free of charge by permission of Virus Bulletin.
The anti-malware industry has been plagued since its earliest days by one poorly designed comparative test after another. In 2007, some of the best anti-malware researchers, comparative testers and product certification specialists took the first steps towards raising product testing standards with the formation of a group specifically focused on establishing standards and methodologies, educating both consumers and testers in discrimination between good and bad practice, and providing objective analyses of current testing practices. This paper summarizes current initiatives by the Anti-Malware Testing
Standards Organization and other groups, but also considers next steps, going beyond objectifying methodology, educational issues and blowing away the fog of misinformation and fallacy, to the next level. Underlying these vital issues is a question: is it possible to make testers and certifying authorities more accountable for the quality of their testing methods and the accuracy of the conclusions they draw based on that testing?
This paper attempts to answer that question.