If you ever learn about optimization, the second technique they teach you after “hill climbing” is something called “simulated annealing“. Forget trying to work through Wikipedia’s definition of it, here’s how I mentally visualize it:

Imagine you your job was to find the highest point on a random Frank Gehry designed building, say the Hotel Marques de Riscal. The problem is, you’re blind so you can only determine the height of one point at a time.

Hotel Marque de Riscal

One way to do it would be to take the entire outside shell of the structure and turn it upside down and then drop a ball bearing in it. When the ball bearing stops moving, you declare that point to be probably the lowest point.

Upside Down Hotel Marque de Riscal

In the field of optimization, this is called hill climbing and it sure is fast but the problem with it is obvious: the ball bearing will likely drop into some rut on the side of the building before reaching the bottom (aka: a local minima).

Simulated annealing is like adding a giant paint shaker and imagining the ball bearing getting heavier over time. At first, the ball bearing is light as a feather and every random vibration is going to make it bounce around, sometimes to a lower point but also sometimes to a higher point. Over time, as the ball bearing gets heavier, it becomes progressively harder to jostle out of it’s rut but when it does move, it moves to a lower rut some distance away. Finally, the ball bearing becomes so heavy it’s virtually impossible to budge, at which point, you declare that position to be probably the lowest point. Simulated annealing can still get trapped in local minima but it usually does a hell of a lot better than hill climbing.

Now, imagine that the ball bearing is society and it’s searching for the solution to a problem like, say, attaching sheets of paper together. If you’re living in any time since the 1930’s, it’s overwhelmingly likely that you’ll be reaching for a “Gem” style paperclip.

A Gem style paperclip

What many people don’t realize is that the Gem paperclip wasn’t the only type of paperclip invented. In fact, the 1909 Websters Dictionary entry on paperclips referred to the “Konaclip” version which was considered the prototypical paperclip of the time.

A Konaclip style paperclip

From 1864 when the first paperclip patent was granted, till 1930 when the gem became dominant, there were literally dozens of designs for paperclips (which, thankfully, has been documented on the Paperclip section of the Early Office Museum). But since the 1930’s, technological progress in the paperclip arena has ground to a halt with the entirety of society standardizing on the Gem and every other style dying in obscurity (For a much more detailed look into the paperclip’s fascinating history, I refer you to the book The Evolution of Useful Things).

To explain this history, I refer back to the simulated annealing analogy. In the early days of paperclipdom, society wasn’t particularly attached to any one design so there was very little momentum in the system. Even a tiny vibration was enough to make another design viable. However, as time went on, certain styles of paperclips started to have legacy effects and it became a race between a few, select alternatives, most notably the Konaclip & the Gem. Finally, society had settled so firmly upon the Gem that it it would take the most extraordinary effort to have any other design supplant it which is why it remains the paperclip of choice to this day. In Simulated Annealing terms, the entropy was now so low that it became practically impossible to escape from a local minima.

Indeed, one of the characteristics of simulated annealing is that it goes through 3 distinct phases. A period of fluid, diverse shifting, a long period of stagnation punctuated by occasional radical shifts and then finally stability from which only incremental improvement is possible.

Looking at the pattern of historical developments of other technologies, it’s possible to spot these same shifts in between phases.

Before 1973, there was a diverse ecosystem of physical interaction paradigms for desktop computing. The most notable examples from that era were Ivar Sutherland’s Sketchpad using light pen interaction and Alan Kay’s tablet style Dynabook.

Sketchpad circa 1963

Dynabook circa 1972

Then, in 1973, the Xerox Alto was released and virtually halted the progress of computer interfaces from that point forward. Nearly every desktop computer today is a recognizable descendant of the Xerox Alto.

Xerox Alto circa 1973

The list of genuine innovations in physical UI that have been adopted since the Alto are as laughably few in number as they are trivial in scope. In rough order of importance: Speakers, microphones, webcams, the mouse wheel, wireless networking, higher resolution screens, a numpad, flatter screens & the Windows/Command button (this is not a sampling, it’s the complete list).

The list of potential innovations are as staggering as they are futile: Pen computing, Tabletop computing, Tablet computing, Augmented & Virtual Reality, Touch based interaction, Multitouch, MultiMouse, Bimanual Interaction, Single Display Groupware, 3D displays and the list goes on and on. In the last 37 years, billions of dollars have been poured into these alternative technologies in a vain attempt to supplant the Xerox Alto as the dominant paradigm with pretty much nothing to show from all that work except a bunch of pretty pilot projects.

And with each passing year, it’s become more and more difficult to foster a viable alternative paradigm because more and more gets invested into keeping Alto style computing firmly entrenched. Software is designed for a keyboard+mouse+screen, people have invested time learning the intricacies of a WIMP OS and thousands of companies have a vested interest in keeping the system entrenched.

The same pattern can be found in keyboard layouts with QWERTY, programming languages with C, web standards with HTML+CSS+JS, Office Productivity with Microsoft Office, Email with Eudora and so forth. In every instance, a period of rapid innovation was brought abruptly to a close with a dominant technology and, from that point forward, genuine changes in the status quo happen at best every decade and only after extraordinary effort *.

Almost every discussion of a new innovation focuses on the details of the innovation almost exclusively without considering the broader social context. Such discussion manages to miss the vital point that adoption of an innovation depends only a tiny bit on the actual innovation and almost completely on the current progress of society. Once a technology has reached a certain maturity point, whatever local minima that society has currently reached will establish itself be the dominant technology until the end of time regardless of what technological progress has occurred or has yet to occur. Like a game of musical chairs, it’s all about when the music stops. If you’ve managed to grab a seat at the table, then you’ll stay there forever, if you missed your spot, then it’s going to take extraordinary effort to regain it, no matter how much of an improvement you are over the status quo.

* This same model also explains why European & Japanese mobile & broadband technology is so far in advance of the US even thought the US were the pioneers of both technologies. Because the US started early, it’s industry matured around an earlier technological & social standard which caused it to fall behind the less mature, more technologically limber peers.

Evil Geniuses are so rare that it’s still surprising when one actually surfaces. In this case, the plot is so deviously brilliant I can’t do anything but heartily applaud it. I’ll let The Guardian take over from here:

The women were led to believe they were being filmed for a Big Brother-type television programme, according to the Dogan news agency and other news reports. Instead, their naked images were sold on the internet by their captors.

They were made to sign a contract which stipulated that they could have no contact with their families or the outside world, and would have to pay a fine of 50,000 Turkish lira (£20,000) if they left the show in the first two months, the agency reported.

She said the women were not abused or harassed sexually, but that they were told to fight each other, to wear bikinis and to dance by the villa’s pool.

Does any other discipline have problems establishing the basics of competency in the same way that software engineering does? Starting from Imran, to Reginald Braitwaite, to Jeff Atwood and now to Richard Banks, the story of FizzBuzz has become mildly famous among the circle of software programmers. What is it about the profession of programming in particular that allows for such stunning incompetence to be masked for so long?

Are there professional athletes who are unable to walk? Are there professional accountants who cannot do math? Are there professional postmodern english lit professors who occasionally let slip with a comprehensible sentence? I’d like to have some insightful insight at this point but the sheer lucridity of it has me baffled.

I’ve mentioned previously that the reason this company is named Bumblebee Labs is from a term I coined:

A bumblebee is an occurrence which cannot be explained by our current theory and thus, demands special attention. Bumblebees are the keys to uncovering areas where our understanding of the world drastically fail and how we can construct a better theory to explain what is happening.

and the FizzBuzz paradox is a Bumblebee of the highest order.

Just once, I’d like to hear the FizzBuzz story from the other side. I’d like to hear a veteran software engineer, confirmed to have been working on multiple projects and with largely positive references explain how he managed to fool so many people for so long about his basic competency in programming. Does this person know that they’re a bad programmer or are they suffering from a severe case of Dunning-Kreuger? How does knowing that you’re a fraud every single day weigh upon your psyche? How do your team mates treat you?

Sure, there are many other fields in which blatant frauds continue to go unexposed for many years: art, academia, holistic medicine, cult leaders etc. But everyone knows that fraud is endemic in these fields. Sure, there are fields in which people are completely self delusional about their ability: music, writing, acting. But everyone else knows those people are crap and never let them do anything of significance. Software Engineering shouldn’t be like those field. It should be like every other respectable, solid career. And yet it’s not.. why?

One of the hardest things about writing this blog is the feeling of talking into a void. I know there are people out there reading it because I can see the analytics data but it still feels lonely writing. So I’m going to experiment with a semi-regular feature where you, the readers respond.

Post a comment with how you found out about this blog, why you read it and also if there’s any particular area you’d like me to cover more.

As an added incentive , I’ll be donating 25 cents to charity for every person that responds. If you feel particularly strongly about any one charity, put that also in the comments.

In my recent post, defining what I refer to as the ego dilemma, I said my favorite ego dilemma was couples who didn’t believe in signing a pre-nup before getting married. I was wrong. My absolute favorite ego dilemma is when someone comes up to me with a revolutionary, amazing, WORLD CHANGING idea. Once they get over the inevitable 15 minute prelude about how I shan’t steal it and I shan’t be paid for the privilege of hearing it, they ramp up to the pitch of what will inevitably be about what I call one of the “holy grail” problem. With the latest conversation, it was about how to build a successful micropayments platform, the one before that, it was building a better craigslist.

When dealing with a situation like this, it is both impossible and futile to point out what’s wrong with their actual ideas. To counter any specific point would be to focus on the trivial dumbness of the project and completely miss the much move overwhelming general dumbness that it is hiding. Although I’ve long ago given up the possibility of changing any of these people’s minds on their pet project, it’s still a fun way to hurl some inventive abuse.

A billion dollars worth of resources have been poured so far into failed startups who were all incredibly confident that this time they have solved the micropayment problem. A billion dollars of PhDs and business models and academic journals and optimistic infrastructure rollouts so that this time, they’re going to do it differently from everyone else and be the SPECIAL ones.
The nub of the ego dilemma is that to believe you have solved a holy grail problem is to believe that you’re smarter than those billion dollars.

Do you really believe that your smarter, more creative, more insightful that every one of the people who have gone before you? Do you really, really believe this, even thought you acknowledge that every one of the people who have gone before you also believed the same thing and all of them were clearly wrong?

People do believe it. They really, really do.

This is quite possibly the most extreme ego dilemma that it’s possible to have, the one by which a rational application of an ego dilemma argument must surely convince any person that they’re quite clearly wrong. The argument against you being a billion dollar genius is so overwhelming that there cannot be any conceivable rebuttal to it and yet, it is also one that is clung to with fierce determination by everyone who experiences it.

The stereotypical image of the billion dollar genius is that of a mentally unstable crank who natters on constantly about their perpetual motion machine and, to be fair, they do make up the bulk of this category. But the really scary thing is that the billion dollar genius syndrome also affects smart, reasonable and otherwise clear thinking people with an equal ferocity. You could be a rationality ninja but still be unavoidably captivated by this particular ego dilemma.

How do I know all of this? Because not only have I seen my smart, reasonable, clear thinking friends suffer through this, I’m forced to confront the unfortunate reality that I’m afflicted by it myself. Even given the impeccable logic presented above, I believe that I have solved a holy grail problem. I’m not dumb enough to present the solution for public ridicule yet but I’m just dumb enough to publicly announce which holy grail problem I’ve solved: I believe I have an entirely novel and practical approach to monetizing a large swathe of Web 2.0 applications without relying on advertising.

Ridiculous, right?

Trust me, I am well aware of how ridiculous it all sounds. And yet… I’m SPECIAL goddammit!

The Dream Job is so simple I’m wondering why I’ve never heard anyone propose it before.

I’m going to start off describing The Dream Job by the most inconsequential and most easily changed details because, well… you’ll see.

The Dream Job pays one million dollars per year and only one is offered every year. Every application for The Dream Job must be made public. Once you’re hired, the only power the employer has over you is firing you. That’s it, those are the only overt constraint for The Dream Job, everything else flows from there.

You don’t apply for The Dream Job by sending in a resume. Well… you could but you’re not likely to get it. There’s only 1 Dream Job a year, you need to dazzle. The Dream Job is only for people who first of all love that company very very much but that love is tinged with a deep channel of ambivalence. It’s for people who are driven with the desire that this company, while great, could be so much greater if they could only fix this one thing and you are the right person to fix it.

I would be deeply, deeply tempted if Newegg offered The Dream Job (There are a half dozen other companies with which I would apply for a Dream Job without hesitation but I’m holding those closer to the chest as they’re related to stuff I’m actually working on). Classic usability isn’t even my field anymore but The UI behind online computer buying has essentially remained static since the mid 90’s and every time I want to purchase a computer online, it makes me deeply angry that the user experience is still so poor from a pure usability standpoint. I have so many ideas bursting in my about how you could revolutionize the user interface to make it orders of magnitude more productive and fitting with the tasks that people have. Give me a week to prepare and an hour to present to Newegg and I’m utterly confident I could convince them hiring me at a million dollars a year would be a bargain. Here’s a simple idea Newegg: Build a braindead reliable text parser so that I can paste the shopping cart from any other web store and you can tell me how much identical or similar items would cost at Newegg. Why should price comparing systems be a laborious half hour of hunting for equivilant components on multiple sites when some simple engineering could reduce it down to seconds?I managed to come up with 8 other ideas inside of an hour before I got bored at how easy it was.

The Dream Job is not for everybody. The freedom offered by it sounds alluring but when you consider the full implications of it, can also be slightly terrifying. Once you get in, you can do literally whatever the hell you want but that also means not a single person will actually know what you do until you sell them on it. By definition, you’re hired to do something at that company that’s never been considered before so you’re starting off with nobody obligated to give you the time of day. It’s up to you to build up the support within the company and selling people on your vision or your mission is dead before it’s even born. There’s a reason why a company full of otherwise smart people hasn’t been able to see a problem that’s so obvious to you and it most probably stems from a complete difference in cultures. You need to be not only a visionary but also an anthropologist and a translator. On top of that, you need to constantly justify your $1 million dollar expense or you will be swiftly canned. The Dream Job requires not only brilliance and passion but also deft people skills and the ability to work around showstopper obstacles.

So, given all this, why one million dollars? Simply because, even in this day and age, one million dollars still means something. It still has that allure when those crisp syllables roll off your tounge. The actual number is meaningless, mostly symbolic, and the bargain of the century to boot. Anyone that brilliant willing to work for one million dollars a year is clearly not in it for the money. Instead, the requirements for The Dream Job are simply the natural result of the observation that hiring only for the skills you know you need is a rather stupid way of doing things.

The conventional way of hiring is you first figure out what resources you need, how much you want to pay them, where they slot in the org chart and then you search for a candidate. This was great when your grandparents were busy climbing the corporate ladder but why don’t we shake it up a bit. How the hell is a company supposed to know what it needs anymore? If you’re a process nerd, then you’re going to hire a bunch of other process nerds and build a great process nerd company culture but how can you ever know what you really need is a deep design aesthetic as well? Similarly, if you’re a design person, how are you going to find out how an obsessive A/B tester can transform how you build? The simple answer is that you never will with a conventional hiring model. Instead, you need them to tell you how they want to do their job.

Everything about The Dream Job stems from transferring the onus of responsibility of defining your job from the employer to the employee. The limit of one a year is what gives it the specialness, the prestige and the cache neccesary to attract the rare people who could handle such responsibility. The million dollars and the enforced hands off approach is what gives them the confidence that The Dream Job is something the company is taking seriously and is committed to integrating as a core part of how they do business. The requirement that applications be public is a filter that screens out the chuckleheads and leaves only those who have a credible chance of deserving it. The requirements are not carved into stone, they’re simply my interpretation of what would be the minimum required for The Dream Job to even work.

 The Dream Job is so stunningly obvious that it must be wrong. I can’t possibly have been the first person to have come up with this. But if it’s wrong, it’s probably at least going to be wrong in an interesting way. If you’re in a position to, do you have the balls to offer a Dream Job? If so, you better hurry because I know the first chance I get to scrape a million spare dollars together, this is what I’m doing.

• Professional corporate ladder climber: If I had an alternate 30 years, I’d love to start from the mailroom at some gigantic company and try to work my way up to CEO. The key to playing the corporate ladder game is that you not only have to be confident but also willing to play office politics and so many people are not only bad at office politics, they actively disdain it which I don’t understand at all. There’s something immensely fun to me about figuring out how to effectively neutralize your enemies from within  your organization and move up the ladder.
• Operations research person: The goal of operations research is to streamline your manufacturing process to make it just a little bit smoother and quicker and more efficient every single day. There’s something zen about that devout focus to the small details which I find relaxing.
• Clinical psychologist: Clinical psychology contains some of the most fascinating issues of epistemology that you have to grapple with on a day to day basis. How do you effectively deal with someone’s self-sabotage when their self-sabotage is sabotaging their ability to deal with their self-sabotage for example?
• Professional chef: I’m a pretty decent cook and people ask me all the time if I’d ever considered doing it professionally. I’d always answered no because professional cheffing is nothing like cooking at home but if I had an alternate 30 years, it’d be fun to see if I could really perform at the highest levels.

I love meeting engaged people when I’m drunk because it allows me to ask my most drunkly assholish question ever:

“So, are you guys going to sign a pre-nup?”

Roughly two thirds of the time, they give some version of an acceptable answer:

• Yes
• No because we have no assets
• No because, while it minimizes the fallout from a divorce, we feel it increases the chance of one by starting the marriage off on a wrong footing so we’d rather not risk it.

But about one third of the time, I get my absolutely most favorite answer of all which is

• No because we don’t believe it’s likely we’ll get divorced.

It’s my most favorite answer of all because, after many years of experience, I’ve found that it’s the best way to force people to actually grapple with the ego dilemma.

The ego dilemma goes something like this:

“So, why don’t you think you’re going to get a divorce? Nobody enters a marriage expecting a divorce yet many of them do”

“Well, sure, other people get divorces but we have X & Y and that makes our marriage special”

“Well, yeah, but there were plenty of people who thought they were also X & Y at the start of their marriage but they eventually found out that didn’t help them much in the end”

“OK, but did those people have Z-which-is-so-uniquely-rare-only-we-have-it?”

“You’re right, they didn’t have Z, but when asked a similar line of questioning, they had the same reaction except they put in Z* which was unique only to their marriage, it didn’t help them much”

“Look… we’re just SPECIAL, OK?”

It’s the “Look, we’re just SPECIAL” which is the hallmark of the ego dilemma, it might not ever be as blatantly obvious as that but it’s always hidden in there somewhere.

The ego dilemma is the belief, against reasonable evidence, that there is something unique contained in your ego that challenges previous historical experience. In short, the ego dilemma would be a perfectly reasonable assumption if you lived in a movie where you were the main character but a deeply tricky one in the real world.

Other example ego dilemmas include believing you’re of significantly above average intelligence, setting aside your life so that you can “make it” as a famous actor/musician/sports star/writer, thinking you WILL get the girl with that desperately creepy romantic gesture or, if you’re coming here from Hacker News, assuming that your startup has a reasonable chance of success commensurate with the effort you’re putting into it.

The truly frustrating thing about the ego dilemma is that it tells you nothing of any value. Recognizing that you’re caught in an ego dilemma doesn’t mean that you’re wrong. You could, after all, be the next Mark Zuckerberg. Someone has to be after all. But also likely is that you’re a clueless idiot who’s utterly convinced at your own fallacious arguments. We know this intellectually because we’ve all experienced the ego dilemma from the outside, you’re trying to convince someone that they’re just plain wrong but they keep on returning back to what makes them SPECIAL. And if you’re experienced it from the outside, it’s meant that someone’s experienced it from the outside at you.

When confronted with the ego dilemma, there are two wrong reactions and one right reaction.

The first wrong reaction is to aggressively try and deflect yourself away from an ego dilemma: “Oh, yeah, I probably SUCK at programming but I just don’t know it yet”. STFU: That you can even concieve that you suck at programming is proof positive that you’re above average and your sanctimonious faux-modest attitude isn’t fooling anyone, including yourself. Deep inside, you still think you’re an awesome programmer and so you still have an ego dilemma.

The second wrong reaction is to instantly assume the question is futile and throw your hands up in the air. “Who can ever KNOW if I’m smart or not?”. Obviously, you don’t live in a world where you believe that to be true. You still think and act like a person who believes they are smart.

Unfortunately, the right way to deal with the ego dilemma is tricky and complex and deserves an entire post of it’s own. It really involves revamping your entire belief structure into something deeply probabilistic with a much finer and more nuanced representation of ignorance which I promise to write at a later date when I’ve fully processed what I’m actually doing.

But the absolutly most fascinating thing about the ego dilemma, and the reason why I so love torturing the almost married is that, even if you fully agree with and accept the argument and logic behind the ego dilemma, even if you’re an otherwise intelligent and reasonable person who doesn’t commit the obvious errors against rationality, when confronted with an actual ego dilemma from the inside, knowledge of the ego dilemma helps you barely at all.

The ego dilemma is what I call an unthinkable thought, you can almost see it slip around people’s head, evading capture. It’s so fascinating to me watching otherwise intelligent people utterly unable and unwilling to grapple with the ego dilemma set in front of them.

Back to our married couple:

“So you understand what an ego dilemma is now?”

“Yes, it all seems very logical and well thought out”

“So you see how it applies to you signing a pre-nup?”

“Oh? No, that doesn’t count, our pre-nup is special”

“What? But saying it’s special is how you RECOGNIZE it’s an ego dilemma”

“It is… but this is a special exception to the ego dilemma because of…”

“ARGH”

As a consumer in the age of the iPhone, I encounter frustrating user interfaces on a daily basis, whether it is with said iPhone, a poorly designed doorhandle, or otherwise. Often, with little tweaks, these could be remedied. Expect this to be a regular feature, as there is always more assclownery about.

Google Chrome

The Ctrl+K shortcut

Background: Chrome has brilliantly mixed the search bar with the URL bar. Using small, custom keywords, I can quickly point my browser precisely where I need to. For example, if I type “wiki cow,” I’ll soon be reading about domesticated ungulates.

Problem: Let’s say I’m done looking at cows, and want to look up sheep instead. I could press Ctrl+T, open a new tab, then type “wiki sheep.” However, if I don’t want to open a new tab, I need to get a cursor in the URL bar. I could take my hands off the keyboard and click. Chrome, however, has a shortcut: Ctrl+K. Ctrl+K automatically clears the URL bar, and sticks the cursor there so the user can navigate. However, Chrome also sticks a question mark in there.

…WHY?

Ctrl+K is brilliant! I love being able to quickly bounce around the nets without a mouse. Ctrl+K lets me do so… but not without deleting the mysterious question mark that jumps in my way for no apparent reason. Who the hell put that in there in the first place?

The Fix: It’s beyond my reach to modify Chrome code, but I can’t imagine it would be a difficult fix on their end.

Irritation Level: 3 of 5

Extra point for the problem being so obvious and basic.

Google’s ChromeOS is heavy on vaporware and light on details at the moment which leaves fertile room for random speculation. Most of the guesses I’ve been reading are really kind of boring so I’m going to sketch out what I think a truly exciting ChromeOS cold be.

I’m extrapolating my guess from three pieces of data:

1) The Chrome moniker is deliberate
2) It’s targeting netbooks for a reason
3) Google has in it’s DNA, the instinct to play David to Microsoft’s Goliath (cf. Google Docs)

Netbooks are great in theory but as soon as you buy one, you run into all of the classical frustrations of owning more than one PC, namely: trying to keep all of your various files, bookmarks & settings synchronized. Sure, you can get your files synchronized and there’s probably a firefox plugin to synchronize bookmarks and probably another one to keep your open tabs in sync and… blargh, who could keep up with all that? My hypothetical ChromeOS solves this by simply saying all your netbook is is a portable browser window. You won’t be able to run photoshop, notepad or even a command prompt. Instead, the only thing running will be Chrome.

But, at a stroke, synchronization is no longer something you have to think about. ChromeOS won’t be an OS in the traditional sense. It’ll just *be* the Chrome browser window you have running on your desktop. Open a new tab on your desktop Chrome, a new tab will appear on your netbook Chrome, half compose an email, go sit in a park and you’ll magically have that half email for you to resume work on, get halfway through a game of Bejewelled and go finish the rest while you’re on the throne. For the first time, you’ll be able to stop in the middle of something, move to a completely different machine and be confident that you can resume exactly where you left off.

What would be so brilliant about this move is that it enters into a space that Microsoft can’t replicate. ChromeOS works, not by doing more than Windows, but by doing less. ChromeOS correctly recognizes the tradeoffs inherent in netbooks. Would you like to run photoshop on a netbook? Maybe once in a while. But what you would really like much more is never having that pit of the stomach feeling when you realize that presentation file is on your home desktop and you’re in Iowa with the work laptop.

Will the real ChromeOS be anything like what I’ve sketched out? Well, here’s hoping…