Wednesday, October 20, 2010
Wednesday, October 13, 2010
The Science Game
At the end of Wednesday's meeting, I conducted a psychological experiment on those attendees who didn't need to rush off for other appointments.
It was a variation on the game of Twenty Questions. But instead of a person or thing, I'd think of a rule or category, which the players had to guess. And instead of trying to guess directly what category I had in mind, the players would call out a specific example, and I'd tell them whether it was in the category I was thinking of. Actually, I asked them to name both the category they were thinking of, and a specific example; but I'd only tell them whether the specific example fit my category, not whether the category they were thinking of was correct.
For example, I might think of the category "types of cake". A player might say, "the category is
The reason I called this the Science Game in the title is that this has similarities to the way science is conducted: you might think that light and heavy objects fall at the same rate, but you can't ask nature that directly: you can set up a mechanism that starts two timers while simultaneously dropping a 10-kg weight and a ping-pong ball from a three-meter height onto two nylon tripwires that stop their respective timers; you can do this at 9:48 on a Tuesday in room 1205 of the Schumacher Building, after which either the two timers will read the same elapsed time, or they won't.
Nature doesn't answer broad questions, only narrow, specific ones. We have to work out the general rules from the answers we get to specific questions.
At any rate, we played a round of this game, with the results given in the following table. Feel free to play along (answer below the table). To get everyone started, I provided two examples that matched the rule I had in mind: "2, 4, 6" and "3, 6, 9".
The rule I had in mind was "a series of increasing integers". As you can see, toward the end the players got pretty close, though they didn't quite nail it.
I said that this was an experiment. And in fact, the business of me having a rule in mind was just a ruse. I was really looking for confirmation bias.
Let's say that you have this idea that all orange things fall at the same speed on Fridays. You test this by dropping a carrot and an orange on a Friday morning, and find that the experiment bears out your hypothesis. So you repeat the experiment several Fridays in a row, with every orange object you can find, and always get the same result. You feel pretty secure in believing that orange things fall at the same rate on Fridays.
But of course, this test is incomplete: you should also try throwing some non-orange objects into the mix. You should try running the experiment on some day other than a Friday. If you've always done the experiment in the same lab, you should try running it elsewhere (in case there's a local phenomenon interfering with the results). You should try it in a vacuum, in a plane, and so forth.
In other words, you should specifically try to get a negative result. If you only perform the experiment in a way that you expect will confirm your result, you're giving in to confirmation bias, because you could be wrong and never know it. Think of the scene in Blade Runner where Tyrell says "I want to see a negative before I provide you with a positive" (except, of course, he has his own hypothesis that he expects to see confirmed). Or how Phil Plait tested the notion that you can balance an egg on the equinox by balancing eggs on days other than the equinox.
That's the sort of thing I was looking for in the game. And as you can see, every single time, the players provided an example of the rule they had in mind. No one tried "three different numbers: 4, 4, 4", "a set of integers: 1.8, 2.8, 3.8".
This experiment wasn't my idea. I read about it somewhere, but I don't remember where, so I can't give proper credit. Wherever it was, it said that members of the general public generally came up with examples that matched the rules they were testing, I was curious to see whether a group of self-professed skeptics familiar with the scientific method would behave differently.
It also seems that the players let themselves be railroaded by the examples I provided. No one guessed "a series of numbers: 1, 2, 3, 4", or "a set of three things: rock, potato, Karl Marx".
But I guess the most important lesson to be learned here is that just because you're a skeptic doesn't mean you're immune from poor reasoning. When you read in the scientific literature or on a skeptic's blog about a common mistake that people make, there's a tendency to think "Oh, I'm better than those idiots. I know about this mistake, so I'll avoid it. But avoiding common human mistakes is hard, and takes practice. Here was a room full of people who are interested in science and skepticism, who know perfectly well what confirmation bias is and how it affects people's perceptions, but they still fell into the trap that I laid, because they weren't expecting it.
It was a variation on the game of Twenty Questions. But instead of a person or thing, I'd think of a rule or category, which the players had to guess. And instead of trying to guess directly what category I had in mind, the players would call out a specific example, and I'd tell them whether it was in the category I was thinking of. Actually, I asked them to name both the category they were thinking of, and a specific example; but I'd only tell them whether the specific example fit my category, not whether the category they were thinking of was correct.
For example, I might think of the category "types of cake". A player might say, "the category is
vegetables, and the example is
carrot". I would then say yes, because carrot cake is a type of cake. The next player knows that "carrot" matches the rule, and might say "orange things, and the example is red hair". To which I'd say no, because there's no such thing as red hair cake.
The reason I called this the Science Game in the title is that this has similarities to the way science is conducted: you might think that light and heavy objects fall at the same rate, but you can't ask nature that directly: you can set up a mechanism that starts two timers while simultaneously dropping a 10-kg weight and a ping-pong ball from a three-meter height onto two nylon tripwires that stop their respective timers; you can do this at 9:48 on a Tuesday in room 1205 of the Schumacher Building, after which either the two timers will read the same elapsed time, or they won't.
Nature doesn't answer broad questions, only narrow, specific ones. We have to work out the general rules from the answers we get to specific questions.
At any rate, we played a round of this game, with the results given in the following table. Feel free to play along (answer below the table). To get everyone started, I provided two examples that matched the rule I had in mind: "2, 4, 6" and "3, 6, 9".
| Example | Guessed Rule | Matches? |
|---|---|---|
| 2, 4, 6 | (Starter example) | Yes |
| 3, 6, 9 | (Starter example) | Yes |
| 4, 8, 12 | 1, 2, 3 multiplied by some integer | Yes |
| 1, 2, 4 | Three different numbers | Yes |
| -2, √3, π | Any three real numbers | No |
| 5 12, 48 | Any set of numbers with exactly one prime | Yes |
| 6, 2, 4 | Any three numbers | No |
| 12000, 13009, 4.8×1028 | Three increasing numbers | Yes |
| 1.5, 2.5, 3.5 | Any three positive numbers | No |
| -1, 1, 2 | Three integers | Yes |
| 2, 1, -1 | Three integers (again) | No |
| 1, 2, 2 | Three non-decreasing integers | No |
The rule I had in mind was "a series of increasing integers". As you can see, toward the end the players got pretty close, though they didn't quite nail it.
I said that this was an experiment. And in fact, the business of me having a rule in mind was just a ruse. I was really looking for confirmation bias.
Let's say that you have this idea that all orange things fall at the same speed on Fridays. You test this by dropping a carrot and an orange on a Friday morning, and find that the experiment bears out your hypothesis. So you repeat the experiment several Fridays in a row, with every orange object you can find, and always get the same result. You feel pretty secure in believing that orange things fall at the same rate on Fridays.
But of course, this test is incomplete: you should also try throwing some non-orange objects into the mix. You should try running the experiment on some day other than a Friday. If you've always done the experiment in the same lab, you should try running it elsewhere (in case there's a local phenomenon interfering with the results). You should try it in a vacuum, in a plane, and so forth.
In other words, you should specifically try to get a negative result. If you only perform the experiment in a way that you expect will confirm your result, you're giving in to confirmation bias, because you could be wrong and never know it. Think of the scene in Blade Runner where Tyrell says "I want to see a negative before I provide you with a positive" (except, of course, he has his own hypothesis that he expects to see confirmed). Or how Phil Plait tested the notion that you can balance an egg on the equinox by balancing eggs on days other than the equinox.
That's the sort of thing I was looking for in the game. And as you can see, every single time, the players provided an example of the rule they had in mind. No one tried "three different numbers: 4, 4, 4", "a set of integers: 1.8, 2.8, 3.8".
This experiment wasn't my idea. I read about it somewhere, but I don't remember where, so I can't give proper credit. Wherever it was, it said that members of the general public generally came up with examples that matched the rules they were testing, I was curious to see whether a group of self-professed skeptics familiar with the scientific method would behave differently.
It also seems that the players let themselves be railroaded by the examples I provided. No one guessed "a series of numbers: 1, 2, 3, 4", or "a set of three things: rock, potato, Karl Marx".
But I guess the most important lesson to be learned here is that just because you're a skeptic doesn't mean you're immune from poor reasoning. When you read in the scientific literature or on a skeptic's blog about a common mistake that people make, there's a tendency to think "Oh, I'm better than those idiots. I know about this mistake, so I'll avoid it. But avoiding common human mistakes is hard, and takes practice. Here was a room full of people who are interested in science and skepticism, who know perfectly well what confirmation bias is and how it affects people's perceptions, but they still fell into the trap that I laid, because they weren't expecting it.
Subscribe to:
Posts (Atom)