Yesterday, I wrote about the mystery of why spam was so bad at being spam and I claimed that it was a mystery that seemingly defied explanation. None of what I proposed as possible answers was really satisfying. In order to answer this question, I think you have to look further afield and ask some other interesting questions: “Why has there not been a non-pathetic foreign terrorist attempt on US soil since 9/11?” and “Why has there only been a handful of truly crippling computer viruses in the last 10 years”
Our first instinct is that such occurrences are rare because they are difficult. However, neither of these tasks actually are difficult. Two guys in a van managed to terrorize Washington DC for a month and no amount of security precautions could have prevented them from doing so. The Sasser worm was written “by someone that could barely get the code working” and attacked a security flaw that had been noted and patched months ago and other worms haven’t been much more sophisticated. Such things are not trivial but they aren’t of such herculean difficulty that would be sufficient to explain their rarity. Just why exactly isn’t there a legion of evil geniuses who are routinely executing the downfall of society?
An evil genius is anyone who is both a genius and evil where “Evil” encompasses everything from trolling to keying someone’s car to pedophilia, “Genius” is anything which evokes any degree of “huh, why didn’t I think of that?” or “That’s clever”. As a rough approximation, we assume that the number of evil geniuses can be calculated by multiplying the proportion of people who are geniuses with the proportion of people who are evil. But what I’ve noticed through looking at a huge range of diverse social systems is that evil geniuses exist at a stunningly lower frequency than this naive calculation would have us believe. The number of evil geniuses is so off base from the naive calculation that it indicates a our model of the world with regards to evil geniuses is unsalvagable and needs to be replaced, not just tweaked.
Such a claim has radical implications for the design of social systems as so much of our thinking about security, about design and about society is obsessed with preventing evil geniuses from wreaking havoc that we don’t even stop to notice that they aren’t.
Part of the reason we’re so obsessed with evil geniuses is because we think we know what they’re like: they’re just like us except they actually do the evil things we think about. Bruce Schneier, one of the most widely read security experts in the world writes about how
Uncle Milton Industries has been selling ant farms to children since 1956. Some years ago, I remember opening one up with a friend. There were no actual ants included in the box. Instead, there was a card that you filled in with your address, and the company would mail you some ants. My friend expressed surprise that you could get ants sent to you in the mail.
I replied: “What’s really interesting is that these people will send a tube of live ants to anyone you tell them to.”
“Why golly”, the man with the Security Mindset says, “I’ve found a great way to exploit this system. It’s lucky I’m a good person because all that is stopping me from executing this exploit for my personal gain is my innate goodness.”
It’s easy to imagine a person who is just like me except without my innate goodness. As a result, it’s easy to design a system with defenses against such a mythical attacker. What we completely fail to notice is that, most of the time, such an attacker simply does not materialize. But even though evil geniuses might not be a major problem, evil behavior most definitely is and it’s in our best interests to design a system which is resilient to pathological actions such as trolling, flaming and abuse.
Our naive view of the world is that we mentally segment people out into “good people” and “bad people”. Good people are people like us and bad people are people like us, except without any morality. The work of Milgram and Zimbardo shows though that goodness is largely a property of circumstance and the more correct way of thinking about the world is that most people are ordinary people and there are good situations and bad situations. If evil people are inherently evil, then it’s easy to imagine an evil genius. However, if evil is a product of the situation, then maybe the reason there are no evil geniuses was because noone gave them permission to be evil geniuses. The reason why Milgram and and Zimbardo managed to cause people to become evil was by relying on authority to signal that such actions were permissible. Genius, by definition, cannot provide be provided such social proof because you’re doing something new and unexpected. Without such social proof, it’s very hard to create an evil situation and, as a result, evil genius is hard to come by.
Such a statement has radical implications for design: you can cause pathological behavior simply by putting in visible mechanisms to prevent pathological behavior. We look to social cues within the system to understand acceptable bounds of behavior and in certain cases, one could reason that if the designer spent so much time building safeguards against certain behaviors into the system, such behavior must be prevalent and thus, acceptable to experiment with. In some cases, the correct approach to obsessing about the security of a system is to leave the system deliberately unsecured so that it does not even occur to people to test the security.
The “No Evil Geniuses” hypothesis is a radically different way to think about the world and one I don’t even think I can completely justify. At the same time, after having looked at all of these disparate cases in which there simply isn’t any other good explaination, it’s one I’ve been increasingly forced to take. Whenever I’ve gone out on a hunt to spot a rich treasure trove of evil geniuses, I’ve never been able to find them. Maybe there’s a simpler, more coherent explaination for all of this but until I find it, I’m going to bill this the No Evil Geniuses Paradox.