Jobs Podcast iTunes Podcast RSS Tumblr @TimOfLegend @DoubleFine Google+ YouTube Facebook RSS

Archives

Jan
May
Sep
Feb
Jun
Oct
Mar
Jul
Nov
Apr
Aug
Dec

Coke vs. Pepsi vs. Humans at Double Fine

Last week, Double Fine Productions was host to a critical scientific research event. It was documented by programmer Brandon Dillon. Here are his findings.


Abstract
Many humans claim to have a preference between the soft drinks Coca-Cola® and Pepsi-Cola® (hereafter referred to as Coke and Pepsi). We conducted an experiment on humans in the employ of Double Fine Productions to look for evidence that humans are even capable of distinguishing between the two sodas.


Hypothesis
Participant convictions towards their capacity for soda differentiation will be consistently strong, but will not be borne out by empirical trials. In layman’s terms, everyone is full of crap.


Experimental Apparatus
Analysis proceeded via a double-blind taste test.

Two eight-packs of 8-oz cans of Coke and Pepsi were purchased from a local grocery store and refrigerated side-by-side in Double Fine’s drink refrigerator for a period of 24 hours.


image


The trials were run by two experimenters.

The first experimenter labeled five otherwise unmarked cups A-E. The experimenter then flipped a fair coin and recorded the results of the flip. For each corresponding “head” flip, a finger of Coke was poured into one cup; Pepsi was poured for each tail. Precise measuring tools were not necessary because the experimenter has a master bartender’s pour.


image


The trial setup was done in a room separate to the other experimenter and all trial participants. To heighten the quality of the experiment, the room featured dimly lit mood lighting and a large collection of rad black velvet paintings.

Once a trial was prepared, the other experimenter was called in, transferred the five glasses to the testing chamber (known colloquially as “the lunch room”) and administered the test to a participant.


image


Participants were free to approach the tasting via means of their choosing for any length of time.

Techniques chosen by participants included judging exclusively by smell, drinking carbonated water as a palette cleanser between sips, and a number of different wine tasting-derived swirling motions.


image


Participants would then record which product they believed were contained in each cup. Some participants chose to place question marks (?) next to guesses to express a lower relative confidence level of a given claim.


Results and Analysis


Trial configurations:

Trial Cup A Cup B Cup C Cup D Cup E
1 Coke Coke Pepsi Pepsi Coke
2 Pepsi Pepsi Pepsi Pepsi Coke
3 Pepsi Pepsi Coke Coke Pepsi
4 Pepsi Pepsi Pepsi Coke Coke
5 Pepsi Coke Pepsi Pepsi Pepsi
6 Pepsi Coke Pepsi Coke Pepsi
7 Pepsi Pepsi Coke Pepsi Coke
8 Coke Pepsi Coke Coke Pepsi
9 Pepsi Pepsi Coke Coke Pepsi


Participant survey results:

Trial Participant Cup A Cup B Cup C Cup D Cup E
1 Gabe Miller Coke Coke Pepsi Coke? Coke
2 Jeremy Natividad Pepsi Coke Coke Pepsi Coke
3 Kjeld Pedersen Pepsi Coke Coke Pepsi Coke
4 Ray Crook Pepsi Coke Coke Pepsi Pepsi
5 Patrick Hackett Pepsi Pepsi Coke Coke Pepsi
6 Paul Du Bois Pepsi Coke Coke Pepsi Pepsi
7 Dan McGarry Coke Coke Coke Pepsi Pepsi
8 Drew Skillman Coke? Coke? Coke? Pepsi? Pepsi?
9 Matt Hansen Coke Coke Pepsi Coke Coke


All participants expressed confidence in their ability to distinguish brands prior to trials.

No single participant correctly guess every cup’s contents in their trial.

Furthermore, no participant incorrectly guessed every cup’s contents. This result would correlate with the ability to effectively distinguish between the products, despite an ability to correctly assign differentiated tastes, which would have been evidence in opposition to the hypothesis. That is to say, a participant who incorrectly guessed every single cup’s contents would still have consistently identified a difference between the two products.


image


Participants offered excuses including the following (not an exhaustive list):
• “I could definitely distinguish the first two, but then the flavors mixed too much in my mouth.”
• “I’m sure I could’ve done it if I had a reference can of Coke or Pepsi with me while doing the trial.”
• “If I keep practicing, I can get good enough to tell the difference.”
• “I went to the dentist this morning and the fluoride is messing with my taste buds.”

The most successful participant was Gabe Miller, who correctly analyzed all cups but one, and expressed less confidence in his single incorrect answer.


Conclusion
Gabe Miller is not human.

6
38
07/18/2012 - 10:24 AM
07/18/12 - 01:01 PM
empika:

"Ha! Great post :D Any chance you could switch up the colours on that results table though? Pretty tricky to read being colour blind."
07/18/12 - 01:06 PM
Lozzen:

"Scientific research.. Is there anything this company doesn't do???!"
07/18/12 - 01:09 PM
Android Surgeon-General Kraken:

"I took the "Pepsi Challenge" once and was informed that i had failed. How does one fail a preference test? If anyone failed, it was Pepsi. I demand a refund."
07/18/12 - 01:13 PM
DF Chris Remo:

"Kraken, We were only testing for the participants' ability to distinguish between the two sodas. Personal preference was irrelevant. Even if people had misidentified the two sodas--but misidentified every single one of them, thus correctly determining the difference--we would have accepted that result, but nobody achieved that either."
07/18/12 - 01:14 PM
KelNishi:

"Seems like you made the same mistake that the Pepsi Challenge testers made. The quantity of the serving affects the perception of the flavor. In my own experience, drinking Pepsi more quickly fatigues my ability to taste sweet things. Seems like the same thing happened to your test subjects here."
07/18/12 - 01:15 PM
DF Chris Remo:

"KelNishi: Could you be more specific on the nature of the mistake? Test subjects were free to take as long as they liked. Some of them took a very long time indeed."
07/18/12 - 02:34 PM
friendofrobots:

"I think KelNishi is referring to the critique that Malcolm Gladwell raises in Blink (discussed here: http://en.wikipedia.org/wiki/Pepsi_Challenge#Criticism). I'm not sure that's totally relevant here since you weren't testing preference."
07/18/12 - 02:46 PM
Unno:

"My friends and I did a similar (though not identical) experiment a few years ago. We found that participants had a much easier time differentiating between the two colas when they were poured from cans. When they were poured from two-liter, plastic bottles, however, only one person did scored better than chance. In summation, I recommend re-testing Gabe with two-liter poured cola."
07/18/12 - 03:50 PM
AnAnemoneInAnonymity:

"I am going to be unreasonable and continue to believe that I could probably do it. Even if it turns out I'm full of crap, I have always wanted someone to actually test me on this. In addition to being allowed a certain interval time between each tasting, I would also demand that each serving of cola be from a freshly opened source. I would specifically request that it not be from a 2-liter, since everything poured from a 2-liter does taste the same: like crap."
07/18/12 - 03:54 PM
AnAnemoneInAnonymity:

"Actually... now that I think about it... I wonder if there is a difference in the level of carbonation between Pepsi and Coke? I recall listening to an NPR story where a study reported that the reason humans enjoy carbonation is because our bodies misinterpret carbonation as a pleasant sour flavor. This would mean that even if Pepsi and Coke were perfectly identical recipes but had different levels of carbonation, there would still be a *perceived* difference in flavor. It would also means that the two beverages would taste more alike as they became flat."
07/18/12 - 09:38 PM
matthansen:

"That fluoride I got from the dentist 30 minutes before the experiment did mess with my taste buds! I want a rematch."
07/19/12 - 12:19 AM
Tim Schafer:

"Get to work you bums!"
07/19/12 - 01:49 AM
KestrelPi:

"It seems like you need to remove the uncertainty of whether tastebuds were getting confused by the multiple drinks. My recommendation is that one week you do one of these a day. You give them a cup of coke or pepsi, and they record privately what they thought it was. I think that would eliminate the possibility that they're getting confused by having the drinks so close together. The other thing I'd be worried about is that people might be making irrational choices just because the brain is difficult at dealing with randomness. For example, even though the coke/pepsi order was chosen randomly, people are inclined to think 'it was pepsi the last two times, the next one surely isn't pepsi again!' when in fact what came before should make no difference if it's random. That's difficult to account for, but having the time gap in between might help with that, and also emphasising that it's completely random every time."
07/19/12 - 06:51 AM
SinbadEV:

"A couple problems with your methodology: 1) People who regularly drink Pepsi or have no preference have "idiot mouths" incapable of distinguishing quality, if you don't prefer coke you clearly are less able to distinguish quality. As such, drink preference (or lack thereof) should be correlated with the results (for example: "Only Drink Coke", "Prefer Coke", "Prefer Some Cola Other Than Coke Or Pepsi", "Don't Care", "Don't Like Cola", "Prefer Pepsi", "Only Drink Pepsi"). 2) You need at least half a can of a soda to really appreciate the nuanced differences between colas. I have often gotten halfway through a meal before realizing that the soda I have been served is the wrong kind (aka Pepsi). Provide testers with at least 200ml in each sample. 3) Note that almost all of your testers got the first drink CORRECT and almost all of your testers got the second drink WRONG. I understand that you made an effort to ensure that people's pallets were cleansed between drinks but clearly something has gone wrong here. Perhaps spacing out the tests more or providing a more effective pallet cleanser like bread or crackers and give more time between tastes. 4) If you intended to test people's ability to distinguish you may have skewed the results by asking them to identify the products by name. If you paired two random samples for each test and asked "Are these the same cola?" you may have had more consistent results."
07/19/12 - 09:20 AM
AnAnemoneInAnonymity:

"@Surplus: Remo has said that they were allowed to take as much time as they wanted between each tasting. Are you suggesting that it should have been even longer? Or maybe that long intervals should have been required instead of optional? Also, I'm not sure if the gambler's fallacy and people's inability to deal with randomness necessarily applies here. It would be applicable if we were testing whether subjects could correctly guess which of any two outcomes would result, but what we're really testing is whether they can tell the difference between those two outcomes. In other words, it's not whether they can accurately guess whether a heads or a tails has just been flipped, it's whether they can look at a coin and identify the difference between the heads side and the tails side. You could make the argument that, when they're not sure, they resort to calculating odds, but that just proves the point that they can't tell the difference. @SinbadEV: I'm the opposite. I have grown up driking Coke and Pepsi pretty equally, and usually once I've been drinking one for a while, it's all the same to me. I can most tell the difference immediately after popping the tab and taking the first, fresh, ultra-carbonated swig."
07/19/12 - 10:48 AM
KestrelPi:

"@Anemone - Yep, I'm saying even longer. A standardised long length. The fact that some were reporting that their tastebuds were confused shows that they were taking a short amount of time between sips even if they had the option of taking longer. So, standardise it to one a day, over the course of a week. Also, the gambler's fallacy DOES apply because expectations affect perception. This is a well documented phenomenon (e.g. the placebo effect) Someone expecting Pepsi is more likely to fool themselves into thinking it's Pepsi, even if they don't realise it. That might mean that the difference is slight enough to fool them, but if we're just testing if they can get better than random chance, that doesn't matter. Another way you could help reduce the effect of a placebo-style taste perception is just by increasing the sample size."
07/19/12 - 11:14 AM
AnAnemoneInAnonymity:

"@Surplus. Oh I see what you mean. That makes sense then. That's similar to other experiments where they swap fast food and five-star food and find that most people can't taste the quality difference. The way the food is presented affects their perception of it more than its actual taste. That sort of effect would especially apply to people with brand loyalty. So yeah, that makes sense. I think your idea of increasing the sample size is a good idea. Maybe add in a third dummy coke that is neither coke nor pepsi, and don't reveal the brand of the dummy."
07/19/12 - 11:16 AM
AnAnemoneInAnonymity:

"*dummy cola Freudian slip. My apologies to science."
07/20/12 - 08:36 AM
fuzzybOotz:

"Fanta-stic!"
07/20/12 - 09:52 AM
gsm:

"Matt: Hey, what are the results? Gabe: You know it's all random Matt: No, that's a trick, they want us to think that, but in reality they have to make every taste the same otherwise the order can affect the results. Gabe: I don't think it works that way. Matt: Just tell me! Gabe: Performing query... Retrieving... Output... "CCPCC" Matt: Got it, CCPCC. (He-he, with GabeBot I'm going to win this thing)."
07/20/12 - 11:53 AM
Acefox:

"Myth Busted! (and although I don't really have a strong preference either way, I think I can tell which is which if given only 2 cups. Beyond that, I think my taste buds will get dulled)"
07/25/12 - 04:37 PM
Belcaw:

"I used to take the "Taste Test Challenge" at JC Penny's back in the early 80's. My dad worked security at for Penny's at the time, so I was at the store quite a bit. Back then, Penny's was more like a Sears department store and sold all sorts of things for the home. Anyway, let me get back on track. They used to have a table with a person performing and recording cola taste tests. You would always be given two clear cups each filled with 2 or 3 ounces of Pepsi or Coke (always one of each). You would then taste each one and tell them your preference. I was so good at picking out the Pepsi from the Coke that eventually the tester (paid by Coke) told me I could have a whole can of Pepsi if I would just leave her alone because I was skewing the results away from Coke. The recipes for the two have changed over the years and now I don't think I could tell the difference if I had to drink them back to back. I agree that a blind sample once per day for a week or so would probably allow for the purest results."
10/04/12 - 12:53 AM
larrzz0jh:

"bestessays.ws Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I'll be subscribing to your feed and I hope you post again soon."
12/27/12 - 10:13 PM
larrzz0jh:

"bras I just started posting comments for blog and facing problem of lots of rejections. I think your suggestion would be helpful for me. I will let you know if its work for me too. Thanks for sharing."
10/12/14 - 05:34 AM
wind:

"This is simply the sort of data that i had been searching for, i am as of now your now and i would consistently watch out for the new posts.best cordless drills"
11/03/14 - 03:45 PM
wind:

"The online journal is completely fabulous. Heaps of extraordinary data and impulse, both of which we all need.clash of clans hack android"
11/05/14 - 02:47 PM
wind:

"I did a little research on the point and found that most individuals concur with your web journal. digital frame"
11/09/14 - 04:25 AM
wind:

"Thanks for your marvelous posting! I indeed liked lesson it, you mind be a fantastic maker. jouluruno"
11/11/14 - 02:36 AM
jackdon:

"I agree with you. This post is truly inspiring. I like your post and everything you share with us is ac repair miami current and very informative, I want to bookmark the page so I can return here from you that you have done a fantastic job ."
11/20/14 - 07:08 AM
wind:

"This is precisely what I was searching for. A debt of gratitude is in order regarding imparting this extraordinary Information! Pikavippi"
11/23/14 - 11:34 AM
jackdon:

"Very nice blog and articles. I am realy very happy to visit your blog. Now I am found which I actually want. I check your blog everyday and try to learn something from your blog. Thank you and waiting for your new post buy facebook likes cheap"
11/26/14 - 02:30 PM
wind:

"Much obliged concerning imparting the thought there would be a few fears from portion yet i am up for it. Lainaa"
12/01/14 - 08:55 AM
jackdon:

"When I first visit this site ac repair miami and try to look what this site talks about I was really amazed with."
12/06/14 - 04:57 AM
jackdon:

"carding I would also motivate just about every person to save this web page for any favorite assistance to assist posted the appearance."
12/07/14 - 03:42 AM
wind:

"Many of us can’t obtain these kinds of information.garcinia cambogia side effects"
12/14/14 - 01:44 AM
wind:

"I merely started in this and also I am just learning the needs of the idea far better! agen klik4d"
12/18/14 - 06:13 AM
wind:

"Simply needed to give a snappy yell out and say that I really appreciate perusing your articles. liquid electronic cigarette"
12/20/14 - 08:40 AM
wind:

"Wonderful weblog! My spouse and i undoubtedly adore the way it is clear-cut on our eye balls as well as the particulars are usually appropriately published. cougar cub dating"
You must be logged in to leave a comment.