Bumblebee Labs is called Bumblebee Labs because of the following quote:

Aerodynamically, the bumblebee shouldn’t be able to fly. But the bee does not know this, so it goes on flying anyway – Antoine Magnan

A bumblebee is an occurrence which cannot be explained by our current theory and thus, demands special attention. Bumblebees are the keys to uncovering areas where our understanding of the world drastically fail and how we can construct a better theory to explain what is happening. But to even notice bumblebees, you have to be on the lookout for them. You have to make a commitment to noticing when you theory goes awry and be willing to dig for an answer.

I was reminded of bumblebees while reading about how researchers infiltrated the storm botnet and discovered that the response rate to spam is 1 in 125 million (Slashdot, Original Paper). How is it that spam is still so awful in this day and age? Spam should be just like any other business, those who are incompetent at it should go out of business and those who do the best, thrive. But this does not seemingly explain why spam seems to have such abysmal conversion rates and why spammers aren’t innovating and experimenting with better ways of spamming.

Spam seems like the perfect vehicle for a data driven, analytic approach. Each email is constructed programatically, websites are created in a largely automated fashion and the path from action to profit is easy to chart out. All the necessary ingredients for Spam 2.0 seem to have been around for the last 10 years and yet spam is still universally awful.

How do we explain the quality of spam then? I can think of a couple of possible explanations, none of them satisfying:

• The spam we are getting now has been optimized and is the spam which maximizes conversion rates. If so, I would be very surprised as this seems to violate almost everything we know about marketing.
• Spam suffers from a supply problem, not a demand problem. Spammers only profit when there’s something to sell and there’s simply not enough people wanting to sell via spam to bother increasing response rate. Andrew Chen writes about how your ad-supported Web 2.0 site is actually a B2B enterprise in disguise and the same issues could be facing spammers. However, for spammers hawking V1agra, it seems like the potential supply should be limitless so I’m going to discount this theory for now.
• Quality is totally irrelevant to a spam campaign, high quality spam and low quality spam get close enough to the same response rate that it doesn’t matter. This might be true if you view spam not as an inducement but as a provider. The purpose of spam is not to convince you that you need a 12 inch h4rd C0ck, it’s to be there for those who have already decided a 12 inch h4rd C0ck is what would rock their world. If this is the case, it doesn’t matter what you put in the messages. However, this does not seem to account for Nigerian Scam emails which very much are set up like an inducement.
• Spam is an oligopoly and hard to break into. It might be the case that there really only are 3 or 4 actual spammers in the world and it’s a hard market to break into. If that’s the case, then it could be none of them have the necessary awareness or expertise to conduct a data driven campaign. There does not seem any obvious structural element to spam though that would make this the case. Given how many Silicon Valley titans have been overthrown by entrepreneurs, spam doesn’t seem to be any different.
• Spammers are all universally stupid. No one in spam is smart enough to conduct a data driven approach. This may be true but if so, it points to a gaping niche in the market which has been open for an extraordinarily long time. By all rights, an entrepreneur should have filled this space by now.

None of these explanations are wholly satisfying and none of them just plain sound right. There is one other explanation I have though which holds some tantalizing clues as to what the true answer might be. However, this explanation is so paradoxical, so shocking and so counter to our intuitive experience that all I can do today is lay the necessary groundwork to show how the problem of spam is a bumblebee that defies resolving with any of our conventional theories. If you have a better explanation for why spam is the way it is, post it in the comments. Otherwise, tune in tomorrow to understand the problem of spam can be explained by the fact that there are no evil geniuses.

In cryptography, a zero knowledge proof is a way to prove that something is true without knowledge of what that thing is. For example, I could prove you know the password to a profile by having you insert the text “Hang, I own this profile qX45s” in the about me section but this would not give me knowledge about what the password is.

I use the term zero knowledge proof as ways of proving whether an assertion is correct without knowing anything about the domain itself. This is as opposed to “first order proofs” in which the proof relies on direct application of domain knowledge to determine truth. For example, if you were presented with a claim that the WTC towers were brought down with timed explosives, a zero knowledge proof would involve looking at who was making the claims, how coherently they are able to make their point, who are the major parties who disagree etc. A first order proof would computing the structural integrity of the buildings, verifying the melting points of steel and how it deforms under high temperature and assessing similar building collapses.

Zero knowledge proofs are powerful because they allow us to leverage the insight that we gain one domain to practically all aspects of life. Given powerful enough zero knowledge proofs, we can attempt to answer questions as diverse as “Is postmodernism bullshit”, “Is global warming real”, “Which president has better economic policies” and “Is the God described by any major religions real”. Unfortunately, most people’s toolbox of zero knowledge proofs suck.

Common zero knowledge proofs include things like whether the person sounds like they know what they’re talking about, whether they have an advanced degree from a prestigious institution, or simply whether they agree with you. Most of these zero knowledge proofs develop as instinctive heuristics and we never really give them much consideration.

Here are some basic zero knowledge proofs which I’ve found to be useful:

• How willing are they to admit the weaknesses and flaws in their own position?
• How well can someone argue against their own position? How aware are they of the best arguments from the other side?
• How willing are they to show you their raw data, their raw speculations and the tools neccesary for you to reach the position they are at?
• Do they have the support and endorsement of others who you know and trust according to similar zero knowledge proof or first order proof criteria?
• Have similar claims been made in the past and been systematically proven wrong?

A powerful toolbox of zero knowledge proofs is the most efficient way of applying analytical insight to a variety of fields. At the same time, even the best zero knowledge proofs cannot match a proper first order proof in determining power and the evidence gleaned from zero knowledge proofs must be placed in it’s context.

The world seems to be split into roughly three different types of people: Those who have a passion for nothing, those who have a passion for one thing and those who have a passion for everything. This way of categorizing is not to cast a value judgement onto any particular group. My informal observation is that aspects such as intelligence, courage, moral fibre and wisdom seem roughly evenly distributed across all three of these groups although it may initially not seem that way. It’s always difficult trying to describe a group with an insider’s perspective if you’re not an insider but I’m going to give it a try:

People with a passion with nothing are the ones who are content to lead an ordinary life. They are the ones who can grow up, go to school, get married, get a good job, buy a house in the suburbs, raise children and grandchildren and die utterly content with their lives.

People with a passion for one thing are those who have found some calling in life and live and breathe that calling. These people may have multiple “one things” for which they are passionate about but they are interested primarily in the thing itself. These are the people who have dreams about thier passion, who spend idle moments of their day thinking about it and who possess a sense of manifest destiny and purpose once they discover their calling.

People with a passion for everything are not interested in things themselves, they’re interested in interest. To them, the actual objects of study are actually incidental, what’s fascinating to them is the more abstract layers in which everything is interconnected. This is not to say that these people are equally interested in everything or even that there are large areas of human experience are completely alien and boring to them(sport gets cited as a common example). But these people are voracious and indiscriminate readers. They’ll be able to converse knowledgably about a huge range of topics and often know surprisingly huge amounts of trivia. If you’ve ever met someone who is a massive fan of TED talks, this is someone who is fascinated by everything. At the same time, for these people, their lives are constantly wracked by a guilt and longing that there is simply never enough time in the world to truly accomplish what they hope to accomplish or master what there needs to be mastered.

It’s no surprise to people who are reading my blog that I place myself firmly into the 3rd category. As a result, it’s been interesting but difficult for me to really peer into the minds of the other two groups of people. But what I’ve noticed in the process of doing so is how radical communication differences arise between members of different groups. If you’re not aware of these very different styles of thought, then you implicitly assume that other people think roughly like you with slightly tweaked parameters.

When a person who is passionate about one thing meets a person who is passionate about nothing, they feel extreme sadness that this person has not yet found their calling. To them, their life is so infused with purpose from their calling that they assume everyone else without a calling feels the same hollow emptiness inside them that they do. They are horrified with the prospect of living an utterly normal, undistinguished life.

When a person who is passionate about one thing meets a person who is passionate about everything, they just assume that this person is passionate about many “one things”. They understand how you could be passionate about two things or five things so they naturally assume the person they’re meeting must be on the far right end of the bell curve and interested in like… a dozen things or maybe twenty things. Widespread passion is mistaken for intelligence because they assume people who are passionate about everything manage their passions in the same way that people who are passionate about one thing do. What they fail to realise is that the passion is not thing-centric.

When a person who is passionate about nothing meets a person who is passionate about one thing or everything, there is a sense of otherworldliness to it, that those people possess some kind of mutant gene which compels them to action. To these people, passion is an utterly mysterious process which they can only reverse engineer from the outside. To them, it’s like thinking of love as really, really, really liking someone.

When a person who is passionate about everything meets a person who is passionate about one thing, they just assume that this is a person who has settled. Every person who is passionate about everything ultimately faces the dilemma about how to focus their attentions. In order to be successful, they need to settle on something to be “their thing”; They need to become a software engineer or a journalist or a academic. Settling one one thing can, on the surface, looking like being passionate about one thing.

But what people who are passionate about everything fail to grasp is that others could be passionate about something without being passionate about your things. It’s a grave affront to people passionate about everything that you cannot convince someone else that something is worth being passionate about. You can’t convert someone into being passionate about your things but you can at least give them a sense of why your thing is worth being passionate about. It’s an utterly alien mindset that someone could be passionate about A, B & C *only* and care not one whit about the things you’re passionate about.

When a person who is passionate about everything meets a person who is passionate about nothing, the lack of curiosity is mistaken for unintelligence or a lack of opportunity. If only they were smarter or if only they had been exposed to a brilliant teacher in school like I had, they would be infused with the same sense of wonder with the world that I have. I think this is one of the more insidious miscommunications that exists because it imposes a subtle form of prejudice and judgement.

 So much of the rancourous debates and misunderstandings I see in the world can be boiled down to a conflict between these basic personality types. Debates about education, about hope, about destiny and about ideals ultimately don’t boil down to the issues at all, they boil down to these three very radically different ways of thinking about the world. Each one is legitimate and each one is valuable and can act as a complement to each other.

The realisation that others have a system of values so shocking different that it seemed almost alien at first was one that enabled me to really connent with many people in a way which I had not previously been able to.

When I was in my very first undergraduate programming class, they hammered into me on very important truth:

The compiler is never wrong

The compiler has no bugs in it, the libraries have no bugs. If you’re not getting the output you expect, then the bug is in your code. Nearly every week, someone would be there furiously muttering to the tutor that he just needs to LOOK at this example because the code is so OBVIOUSLY correct that it MUST be a compiler error of some kind. And every time it happened, the tutors would simply smile complacently back and remind the student that “The compiler is never wrong”. Eventually, with enough repetition, we understood this fact down deep into our bones and I think it’s made us better programmers as a result of it.

On the face of it, this is absurd. Compilers are programs just like anything else and they contain bugs like every other program. If we were talking about established, battle scarred compilers like gcc, you might be able to make a credible argument but we were working with the Glasgow Haskell Compiler which most certainly did have bugs in it.

The statement “The compiler is never wrong” has such power because it’s so patently easy to prove false. And as I grow older and think I understand more and more about the world, some of the most powerful beliefs that you can hold are the obviously wrong truths. You can never tell an obviously wrong truth to someone who is not ready to hear it because it’s so obviously wrong. You need to take a leap of faith and accept that something can be obviously wrong and still true for such things to make sense.

If this sounds supiciously like what you’ve heard religious people say, it’s because maybe this is what religion is…

This is a concept which I’m currently struggling to come up with a better name for but it’s about responses to an argument. A crucial part of argumentation is actually understanding the claims and assertions that the other person is making. In order to do this, one can either solve the easy problem or the hard problem.

The difference between the easy problem and the hard problem is one of recognition vs recall. When confronted by an argument, the easy problem is to scan through your list of pre-canned responses to arguments. It’s a matching between arguments and replies. Is reply 1 close enough to fit? No. Is reply 2 close enough to fit? Yes. Stop, you’re done, spit out response 2.

The easy problem is seductive because it’s well, easy. But it’s more than that as well, it’s gratifying to the ego. You come up with a substantive response and it’s clever so you feel like you’re doing real work. Moreover, you spend your time compiling a larger database of pre-compiled responses and the larger your database is, the closer and more encompassing your matches become so you feel like you’re making progress. But when you solve the easy problem, you stop at the FIRST match which is sufficiently close. If the difference between the actual question and what you perceive the question is suffciently close, then you completely ignore the difference.

Solving the hard problem is taking the opposite approach. Instead of figuring out what response matches the question, you instead look at the structure of the question and reason out a response free from any pre-concieved biases. You conciously don’t try and recognize the question and place it into a particular category. Solving the hard problem can be valuable because it occasionally leads to genuine surprise. Solving the easy problem will never tell you something you could not be convinced of but solving the hard problem occasionally leads you down a difficult path.

The hard problem is more intellectually pure, with less chances of making a mistake. But it’s well, hard, and quite often doesn’t seem neccesary. So many of the questions we are asked every day seem like the sort that can be answered with a pre-cached answer and so we feel comfortable solving them with the easy problem. The problem is, it’s impossible to tell whether a problem is indeed something solvable or not with the easy problem because once you’ve determined that, you’ve already solved the hard problem.

This is especially true of internet comments and conversations. Once you mention certain key words in a posting, people will come in and post based on what happened to match that filter. This makes it very hard to present an argument which is very similar to a common argument because most people will match for the common argument.

So how do you get around this? I think the only way is to gain the respect of a core group of readers and have them get to the point where they assume that you’re not stupid and that it’s worth trying to solve the hard problem when you present them with an idea.

My post yesterday on how to think about the existance of god seemed to generate a fair bit of commentary, both on the blog and on reddit. Many people popped up with alternative rebuttals to the claim “you can’t prove that god exists”, all of which I was aware of. But here’s the fundamental problem with all the conventional claims: They don’t work. Yes, they might be strictly, logically sound. Yes, they might require less of a leap of logic. But the problem is that concepts like burden of proof and occam’s razor sound totally convincing to people who have already accepted that it’s true but it’s hard to overestimate just how bizarrely counterintuitive, highly abstract and just plain wrong-sounding these concepts are. Atheists don’t have an argumentation problem, they have a communication problem.

Here’s what atheists seem to be missing when they encounter a Christian who disagrees with them: There are actually two legitimate reasons why a Christian would hold the position that Atheism is not merely wrong, but absurd on the face.

the philosophical argument

One is that they disagree from a fundamentally philosophical standpoint. It’s a perfectly legitimate model to posit that God only reveals himself to those who have made the leap of faith and, indeed, the power of religion lies in the difficulty in finding God. The evidence for God does not lie in naturalistic experiment, it lies in the human quest for meaning or the structure and order of life. It’s not a position I agree with but it’s definately one I respect as a internally coherent explaination of the world. In this case, of course atheists can’t find any evidence of God’s existance, they’re simply too stubborn and persist in looking in the wrong place despite huge and obvious signs of their ineptitude.

The urn example in this case lays down explicitly the areas of agreement and disagreement. You can move from there to the much more philosophically demanding areas of how occam’s razor and burden of proof affect each side’s claims.

the factual argument

The second reason is that they simply disagree with you on a factual level. To them, faith healing is real and demons have a manifest effect on the world. Miracles happen all the time and you would have to be stupid and blind to be an atheist. It’s so obvious that supernatural events are happening that it becomes impossible to consider that another person could view the world differently.

What the urn example demonstrates is that yes, atheists too would be convinced by supernatural events. A ball with the number 417 would convince an atheist the urn is red just as easily as a bona fide miracle would convince them of God’s existence. The point of contention is on an interpretation of evidence and this can be used as a starting point to segue into skepticism, levels of evidence, basic human psychology and burdens of proof. If you differ on a factual level, then Occam’s Razor is completely a non convincing argument. If miracles are happening on a daily basis, then the simplest explaination really is that god exists.

The fundamental problem that I’ve seen is that the average atheist argues with a Christian like they’re an atheist but stupider and believing in silly things. This is a wholly ineffective way to argue with anyone and it’s not going to change anyone’s mind.

I’m going to break my sequence of concepts to present an interesting analogy I just came up with to explain why this argument is subtly wrong:

“There’s no way to prove that god does not exist”

Say I have two urns:

• One is filled with numbered green balls, all of which lie in the range of 1 to 100.
• The other is filled with numbered red balls all of which lie in the range of 1 to 500.

I draw a sequence of balls from a single urn, announce the numbers and I then ask you what color you think the urn I picked was.

Obviously, if there is a single ball >= 101, then you can assert with 100% probability that the urn is red. However, there’s no possible sequence of balls that could definitively prove a green urn. But if I keep on drawing balls under 100, consistently and without a single ball over 100. The more balls are being drawn, the more sure you are that I picked the green urn.

I view this as analogous to the problem of the existence of god. The space of possible universes in which god does not exist is a strict subset of the space of possible universes in which god does exist. It’s therefore strictly impossible to prove that god does not exist.

Each observation is like drawing a ball out of the urn and each observation can be consistent with an atheistic or supernatural interpretation of the world. Say you observed stones independently arranging to form the words of the koran, the ten commandments written in fire across the sky and routine, repeatable, spontaneous limb regeneration after praying. If any one of these happened (and they were verified to be bona fide miracles and not just what seemed like miracles), it would be the equivilant of drawing ball 328: absolute proof that god exists. But we keep on picking balls and observing the world and they keep on being strictly naturalistic phenomena.

Sure, it’s still possible that god exists and we’re going to find evidence of him if we keep on looking harder. But to me, we’ve picked enough balls that it’s not where the smart money is anymore.

Quite a while ago, I was reading an article on secret tips of the trade that different professions have which had the following quote:

Actor

Every actor eventually is called upon to act drunk. Most do this by slurring their speech, stumbling around, and perhaps drooling a bit. This is what a freshman drama teacher calls “indicating.” A better way to appear drunk is to act very, very sober. Walk very carefully, and try not to let anyone see that you’re inebriated. This is much more subtle and will register on a level the audience won’t immediately recognize.

How do we spot people who are drunk? They’re the ones who are acting sober. Drunk people behave like how they think sober people behave. We understand this instinctually and subconciously and we’re incredibly attenuated to it although we often can’t articulate why we feel something is off.

This applies to other aspects of human behaviour. We can spot when rich people are pretending to be middle class because they act like what they think middle class people act. We can spot unintelligent people trying to sound intellectual. Because in all these cases, these people are acting sober.

The converse side of this is that you can only spot acting sober if you’ve actually been sober before. If you’re a rich person, you’ll totally be convinced by another rich person pretending to be middle class because that’s totally how you think middle class people act.

Understanding acting sober allows you to see the various masks that people put on and, more importantly, it’s an inner lens on how they internally represent the world. It can be a window onto someone else’s soul.

Of course, there’s acting acting sober (which is what professional actors do) and acting acting acting sober and so on ad infinitum as well.

Personal growth is never a smooth process. There are many potential pitfalls along the way which can trap people for years at a time until they see the neccesary advanced wisdom to unstick themselves. One of these that I’ve been thinking a lot about recently is a concept I call “you can’t get here from here”.

Let me give you an example: Of all the most deeply spiritual people that I know, all of them have had some sort of deep crisis of faith where that have had to grapple with serious doubt and all of them have confessed to me that such struggle has been a key step in deepening their faith. I would go so far as to argue that such a crisis of faith is a neccesary step to forming a truely genuine bond of faith as it forces you to seriously grapple with your existential doubt rather than simply ignoring it. But if this is true, if a crisis of faith is a neccesary component, then you simply can’t get to having strong faith simply by having strong faith. You can’t get here from here.

You can’t get here from here is a pernicious process because the path to the goal inherently requires a leap of faith. None of my friends who went through a crisis of faith were thinking “oh good, I’m having a crisis of faith now which is Step 8 in my Good Christian handbook”, they were too busy freaking out that what they had taken for granted all their lives was being pulled out from under them. And indeed, none of them knew what they would look like coming out the other side. For some it grounded their faith, for others, it was the first steps towards atheism and for a few, it simply left them deeply confused and hurting. But for all of them, it required bravery to fully commit to the path they were on rather than staying in a state of denial. How many others were there who came on the verge of a crisis of faith but resisted because they deeply believed that the road to more faith was more faith?

I’m talking about others now because “you can’t get here from here” is a deeply personal process and it’s possible for me to see it more objectively from the outside. Examples from my own life are always messy and confusing and full of doubt but I think there are some I’m willing to tentatively put on this list:

• You can’t become truly arrogant until you’ve become humble
• I used to think being nice was about saying nice things. Then I realized that being truly nice is to think nice thoughts and to do this, you must say your nasty thoughts
• In order to master a sport to the level of being able to analyze constantly, you have to at some point, stop thinking
• You can’t be truly generous until you’ve achieve selfishness

If you don’t understand what I mean by these, they might be advanced wisdom. On the other hand, they could just be bullshit.

I was talking to a friend the other day about the nature of argument and how to argue well and it occured to me that I needed to take a step back and convince him that it was possible to argue well. Argument just seems like one of those things you do. You sit and you talk and you generally say the first thing that comes into your head. That argument would require training and skill is something which doesn’t appear immediately obvious at first grasp to people.

We seem to split up the list of personal qualities into skilled and intuitive. In the skilled category would be things building a house, playing a game of chess, arguing a case in court or solving math problems. On the intuitive side, we have things like having a good sense of humor, being co-ordinated, having the ability to draw well or coming up with good ideas.

The difference between the two is that we believe that to be good at skilled things is a process of mastery of certain skills which is relatively unmysterious. On the other hand, being good at intuitive things is something wholly mysterious that seems largely innate.

What’s interesting about argument is that it seems to have shifted from a skilled task to an intuitive one. In medieval times, rhetoric was part of the trivium, along with grammar and logic. People would spend 1/3rd of their university education solely on learning the art of arguing and people understood instinctively back then that you had to work to become a good arguer. Nowadays, rhetoric is a minor part of a minor department and argument seems like a purely intuitive process.

It seems to me a plausible explanation is that we rely on social signals to cue us in on what is skilled or not. Because we see people spend years learning to be a professional doctor but not learning to be professional arguers, we tend to believe one requires skill and the other does not.

So why am I going on about this? Because such a heuristic is imperfect at best and being aware of how it is flawed can help you gain an incredibly easy win over other people. By correctly identifying something as requiring skill, you can start to gain immediate improvements and, what’s more, others will believe that this improvement is innate and both be impressed with how good you are and not try and compete with you.