Most amateur double blind tastings are horrible from a statistical perspective. They barely shed any insight into the truth at all but, what’s worse, they give a false sense of knowledge. Last night, I made the assertion that top shelf vodkas are indistinguishable from each other and that any perceived taste differences were purely psychological. This lead me to be responsible for a quick, impromptu blind vodka tasting of 3 top shelf vodkas (Ketel 1, Grey Goose & Ciroc) between myself & 4 other skeptical participants (in retrospect, we should have added a well vodka as a control but we did try a well vodka after the blind tests and the difference was pretty apparent).
Our very helpful bartender marked the bottom of each glass with the vodka brand such that we could not see them, then we proceeded to taste & rate. Now, most amateur double blind studies I’ve seen rely on a single tasting then ranking. This is somewhat fine in a large lab setting with a sufficient number of participants and samples but, in our circumstances would lead to 0 statistical insight. The reason why is pretty simple, among a sample of 3 vodkas, there are only 6 different permutations. Thus, with 5 participants, it’s more likely or not, someone will get a “hit” purely by chance.
Instead, what we relied on was a double tasting procedure. Each person would sip & rank the vodkas, an independant 3rd party would then proceed to shuffle the order while we closed our eyes and we then proceeded to sip & rank the vodkas again. What we were looking for was not whether you could correctly assign the brand to a vodka (which is relatively hard) but whether you could rerecognize a vodka you had just drank (which is relatively easy). As it turns out, of the 5 participants, I was the only one who correctly determined how the vodka had been shuffled.
Now, despite the fact that I was crooning all night about how I “won” the challenge, this is not the correct conclusion to be drawn from the data. What it demonstrated was that at least 4 of the 5 participants were unable to distinguish top shelf vodkas with reliability, despite their certainty before revealing the results that there were clear and distinct differences. What this proves was that the perceived differences were purely physiologically and psychologically based and not as a result of the chemical qualities of the vodka. Additionally, it is unknown whether I could truly distinguish the difference. Remember, there’s still only 6 possible answers so it’s pretty probably that I got them right purely on luck. A further shuffle & taste would be able to shed more insight into this hypothesis but we were out of vodka at that point.
Most amateur double blind studies aren’t worth the blog post they’re written on because the authors have such a poor grasp of experimental setup that the data is worthless. Amateur studies don’t have the resources of a professional study to collect large enough amounts of data to make confident predictions, thus you need to scale back the expectations of the experiment to match the resources you have on hand. If you want to perform a double blind study with either a small sample set or experimental group, you need to use a repeated tasting procedure rather than a single tasting procedure or you run the risk of making assertions which are not statistically supported.