Andrew Lee and I are presenting a paper on phishing quizzes at the Virus Bulletin conference in September 2007. While I already have pretty strong opinions on these, I’d appreciate some input from others with an interest in phishing education. The paper has to be submitted by the beginning of June, but this is an ongoing hobby horse of mine, so comments too late to be incorporated into the paper will not be wasted. This isn’t a formal research study (at the moment) so this isn’t a particularly structured survey: I’m looking for qualitative rather than quantitive data.
Do as much or as little as you like, by cutting and pasting the bits you want to comment on into the comments field, and feel free to expand. If you find the length of it a bit much, try going straight to the last couple of questions. If you don’t want to comment here, feel free to email me at david (dot) a (another dot) harley (at) gmail (yet another full stop) com.
1) Which quizzes have you looked at, if any?
2) Which did you feel were useful/useless/interesting/misleading/accurate/whatever?
3) Which format do you consider most useful?
- multiple choice text questions
- multiple choice identification of phish messages
- multiple choice identification of non-phish messages
- multiple choice identification of phish sites
- multiple choice identication of non-phish sites
- other (please describe)
4) If you tried multiple choice phish message/site identification quizzes, how well did you do overall? Which of the following did you do better on?
- phish messages
- legitimate messages
- phish sites
- legitimate sites
5) How useful do you think comparisons of static images are?
6) Do you expect to be able to ID a suspicious site or message from a static image? What supporting information is it useful to supply with static images (site or message image)?
7) What information do you expect to get back from a quiz site? Is a simple right or wrong enough?
8) What is (or should be) the purpose of a phish quiz?
9) What sort of question should a multiple choice text quiz ask?
10) How important is entertainment value in a quiz?
11) What heuristics do you use to identify a suspect site or message? How well does that map to phishing quiz answers, when they include an heuristic explanation? How convincing do you find quiz explanations?
12) What supporting material (eg FAQs) do you find or expect to find on quiz sites?
13) In principle, do you consider that phish quizzes are educationally useful?
14) Should a phishing quiz use real examples, either as modified samples or using controlled live access?
15) How should a quiz be constructed? Are the following useful?
- Static images
- Static images with supplementary info, eg annotations
- Animated content
- Role playing info
- Shortcuts to the end of the quiz
- Continuous feedback on the accuracy of your responses as opposed to a final score with no feedback during the test
- Detailed explanation of correct answer
- Detailed explanation with heuristic guidance
16) How do your rate your own expertise in this area?
- Security professional but not specialist
- Knowledgeable non-pro
- Not particularly knowledgeable.
Thanks for your help.