This is a paper presented at the 2012 EICAR conference in Lisbon. (I actually presented two: the other one will be posted here in a day or two, or maybe a little longer as I’m travelling right now.) It’s posted here rather than on the ESET resources page for conference papers in accordance with EICAR’s copyright stipulation that EICAR conference papers be posted on personal web sites.
Here’s the abstract:
Imagine a world where security product testing is really, really useful.
- Testers have to prove that they know what they’re doing before anyone is allowed to draw conclusions on their results in a published review.
- Vendors are not able to game the system by submitting samples that their competitors are unlikely to have seen, or to buy their way to the top of the rankings by heavy investment in advertising with the reviewing publication, or by engaging the testing organization for consultancy.
- Publishers acknowledge that their responsibility to their readers means that the claims they make for tests they sponsor should be realistic, relative to the resources they are able to put into them.
- Vendors don’t try to pressure testers into improving their results by threatening to report them to AMTSO.
- Testers have found a balance between avoiding being unduly influenced by vendors on one hand and ignoring informed and informative input from vendors on the other.
- Vendors don’t waste time they could be spending on enhancing their functionality, on tweaking their engines to perform optimally in unrealistic tests.
- Reviewers don’t magnify insignificant differences in test performance between products by camouflaging a tiny sample set by using percentages, suggesting that a product that detects ten out of ten samples is 10% better than a product that only detects nine.
- Vendors don’t use tests they know to be unsound to market their products because they happened to score highly.
- Testers don’t encourage their audiences to think that they know more about validating and classifying malware than vendors.
- Vendors and testers actually respect each others work.
When I snap your fingers, you will wake out of your trance, and we will consider how we could actually bring about this happy state of affairs.
For a while, it looked as if AMTSO, the Anti-Malware Testing Standards Organization, might be the key (or at any rate one of the keys), and we will summarize the not inconsiderable difference that AMTSO has made to the testing landscape. However, it’s clear that the organization has no magic wand and a serious credibility problem, so it isn’t going to save the world (or the internet) all on its own. So where do we (the testing and anti-malware communities) go from here? Can we identify the other players in this arena and engage with them usefully and appropriately?
David Harley CITP FBCS CISSP
Small Blue-Green World
ESET Senior Research Fellow