Wednesday, October 20, 2010
Wednesday, October 13, 2010
The Science Game
At the end of Wednesday's meeting, I conducted a psychological experiment on those attendees who didn't need to rush off for other appointments.
It was a variation on the game of Twenty Questions. But instead of a person or thing, I'd think of a rule or category, which the players had to guess. And instead of trying to guess directly what category I had in mind, the players would call out a specific example, and I'd tell them whether it was in the category I was thinking of. Actually, I asked them to name both the category they were thinking of, and a specific example; but I'd only tell them whether the specific example fit my category, not whether the category they were thinking of was correct.
For example, I might think of the category "types of cake". A player might say, "the category is
The reason I called this the Science Game in the title is that this has similarities to the way science is conducted: you might think that light and heavy objects fall at the same rate, but you can't ask nature that directly: you can set up a mechanism that starts two timers while simultaneously dropping a 10-kg weight and a ping-pong ball from a three-meter height onto two nylon tripwires that stop their respective timers; you can do this at 9:48 on a Tuesday in room 1205 of the Schumacher Building, after which either the two timers will read the same elapsed time, or they won't.
Nature doesn't answer broad questions, only narrow, specific ones. We have to work out the general rules from the answers we get to specific questions.
At any rate, we played a round of this game, with the results given in the following table. Feel free to play along (answer below the table). To get everyone started, I provided two examples that matched the rule I had in mind: "2, 4, 6" and "3, 6, 9".
The rule I had in mind was "a series of increasing integers". As you can see, toward the end the players got pretty close, though they didn't quite nail it.
I said that this was an experiment. And in fact, the business of me having a rule in mind was just a ruse. I was really looking for confirmation bias.
Let's say that you have this idea that all orange things fall at the same speed on Fridays. You test this by dropping a carrot and an orange on a Friday morning, and find that the experiment bears out your hypothesis. So you repeat the experiment several Fridays in a row, with every orange object you can find, and always get the same result. You feel pretty secure in believing that orange things fall at the same rate on Fridays.
But of course, this test is incomplete: you should also try throwing some non-orange objects into the mix. You should try running the experiment on some day other than a Friday. If you've always done the experiment in the same lab, you should try running it elsewhere (in case there's a local phenomenon interfering with the results). You should try it in a vacuum, in a plane, and so forth.
In other words, you should specifically try to get a negative result. If you only perform the experiment in a way that you expect will confirm your result, you're giving in to confirmation bias, because you could be wrong and never know it. Think of the scene in Blade Runner where Tyrell says "I want to see a negative before I provide you with a positive" (except, of course, he has his own hypothesis that he expects to see confirmed). Or how Phil Plait tested the notion that you can balance an egg on the equinox by balancing eggs on days other than the equinox.
That's the sort of thing I was looking for in the game. And as you can see, every single time, the players provided an example of the rule they had in mind. No one tried "three different numbers: 4, 4, 4", "a set of integers: 1.8, 2.8, 3.8".
This experiment wasn't my idea. I read about it somewhere, but I don't remember where, so I can't give proper credit. Wherever it was, it said that members of the general public generally came up with examples that matched the rules they were testing, I was curious to see whether a group of self-professed skeptics familiar with the scientific method would behave differently.
It also seems that the players let themselves be railroaded by the examples I provided. No one guessed "a series of numbers: 1, 2, 3, 4", or "a set of three things: rock, potato, Karl Marx".
But I guess the most important lesson to be learned here is that just because you're a skeptic doesn't mean you're immune from poor reasoning. When you read in the scientific literature or on a skeptic's blog about a common mistake that people make, there's a tendency to think "Oh, I'm better than those idiots. I know about this mistake, so I'll avoid it. But avoiding common human mistakes is hard, and takes practice. Here was a room full of people who are interested in science and skepticism, who know perfectly well what confirmation bias is and how it affects people's perceptions, but they still fell into the trap that I laid, because they weren't expecting it.
It was a variation on the game of Twenty Questions. But instead of a person or thing, I'd think of a rule or category, which the players had to guess. And instead of trying to guess directly what category I had in mind, the players would call out a specific example, and I'd tell them whether it was in the category I was thinking of. Actually, I asked them to name both the category they were thinking of, and a specific example; but I'd only tell them whether the specific example fit my category, not whether the category they were thinking of was correct.
For example, I might think of the category "types of cake". A player might say, "the category is
vegetables, and the example is
carrot". I would then say yes, because carrot cake is a type of cake. The next player knows that "carrot" matches the rule, and might say "orange things, and the example is red hair". To which I'd say no, because there's no such thing as red hair cake.
The reason I called this the Science Game in the title is that this has similarities to the way science is conducted: you might think that light and heavy objects fall at the same rate, but you can't ask nature that directly: you can set up a mechanism that starts two timers while simultaneously dropping a 10-kg weight and a ping-pong ball from a three-meter height onto two nylon tripwires that stop their respective timers; you can do this at 9:48 on a Tuesday in room 1205 of the Schumacher Building, after which either the two timers will read the same elapsed time, or they won't.
Nature doesn't answer broad questions, only narrow, specific ones. We have to work out the general rules from the answers we get to specific questions.
At any rate, we played a round of this game, with the results given in the following table. Feel free to play along (answer below the table). To get everyone started, I provided two examples that matched the rule I had in mind: "2, 4, 6" and "3, 6, 9".
| Example | Guessed Rule | Matches? |
|---|---|---|
| 2, 4, 6 | (Starter example) | Yes |
| 3, 6, 9 | (Starter example) | Yes |
| 4, 8, 12 | 1, 2, 3 multiplied by some integer | Yes |
| 1, 2, 4 | Three different numbers | Yes |
| -2, √3, π | Any three real numbers | No |
| 5 12, 48 | Any set of numbers with exactly one prime | Yes |
| 6, 2, 4 | Any three numbers | No |
| 12000, 13009, 4.8×1028 | Three increasing numbers | Yes |
| 1.5, 2.5, 3.5 | Any three positive numbers | No |
| -1, 1, 2 | Three integers | Yes |
| 2, 1, -1 | Three integers (again) | No |
| 1, 2, 2 | Three non-decreasing integers | No |
The rule I had in mind was "a series of increasing integers". As you can see, toward the end the players got pretty close, though they didn't quite nail it.
I said that this was an experiment. And in fact, the business of me having a rule in mind was just a ruse. I was really looking for confirmation bias.
Let's say that you have this idea that all orange things fall at the same speed on Fridays. You test this by dropping a carrot and an orange on a Friday morning, and find that the experiment bears out your hypothesis. So you repeat the experiment several Fridays in a row, with every orange object you can find, and always get the same result. You feel pretty secure in believing that orange things fall at the same rate on Fridays.
But of course, this test is incomplete: you should also try throwing some non-orange objects into the mix. You should try running the experiment on some day other than a Friday. If you've always done the experiment in the same lab, you should try running it elsewhere (in case there's a local phenomenon interfering with the results). You should try it in a vacuum, in a plane, and so forth.
In other words, you should specifically try to get a negative result. If you only perform the experiment in a way that you expect will confirm your result, you're giving in to confirmation bias, because you could be wrong and never know it. Think of the scene in Blade Runner where Tyrell says "I want to see a negative before I provide you with a positive" (except, of course, he has his own hypothesis that he expects to see confirmed). Or how Phil Plait tested the notion that you can balance an egg on the equinox by balancing eggs on days other than the equinox.
That's the sort of thing I was looking for in the game. And as you can see, every single time, the players provided an example of the rule they had in mind. No one tried "three different numbers: 4, 4, 4", "a set of integers: 1.8, 2.8, 3.8".
This experiment wasn't my idea. I read about it somewhere, but I don't remember where, so I can't give proper credit. Wherever it was, it said that members of the general public generally came up with examples that matched the rules they were testing, I was curious to see whether a group of self-professed skeptics familiar with the scientific method would behave differently.
It also seems that the players let themselves be railroaded by the examples I provided. No one guessed "a series of numbers: 1, 2, 3, 4", or "a set of three things: rock, potato, Karl Marx".
But I guess the most important lesson to be learned here is that just because you're a skeptic doesn't mean you're immune from poor reasoning. When you read in the scientific literature or on a skeptic's blog about a common mistake that people make, there's a tendency to think "Oh, I'm better than those idiots. I know about this mistake, so I'll avoid it. But avoiding common human mistakes is hard, and takes practice. Here was a room full of people who are interested in science and skepticism, who know perfectly well what confirmation bias is and how it affects people's perceptions, but they still fell into the trap that I laid, because they weren't expecting it.
Tuesday, September 21, 2010
Placebo, God of Indifference
Today's Speak With Monsters takes a critical look at curing with leeches. (NB: Speak With Monsters makes most sense if you're familiar with the Monster Manual, or at least know about AD&D.)
Saturday, May 8, 2010
Cracked on Psychics
Cracked has an article entitled "5 Cheap Magic Tricks Behind Every Psychic". The introductory paragraph reads:
Now go read the whole thing.
I got into magic at the age of five. I stopped thinking psychics were real at the age of five-and-a-half. Mainly because most of them were doing tricks I had just read in the colorful magic book I had bought for three dollars the week before.
Now go read the whole thing.
Wednesday, February 24, 2010
Monday, February 22, 2010
Wednesday, February 17, 2010
Saturday, February 6, 2010
I Just Had a Supernatural Experience
Spoiler: not really.
I was just in the kitchen fixing dinner for the cats, when I heard a woman cough. Just a single throat-clearing cough. It definitely sounded like a woman.
And this was odd, since there aren't supposed to be any women in the house. Any company would be unexpected, seeing as how I'm snowed in by Snowmageddon. But it was convincing enough that I checked the living room and looked for fresh tracks in the snow outside. Perhaps someone got stranded and wandered in when she found an accidentally-unlocked door? Or could it be a ghost?
Obviously, there was no one there, as you know, since you've read the spoiler at the top. I tried replaying the sound on the tape recorder of my mind[1] and play it back, trying to figure out what I'd heard, rather than what I thought I'd heard. That is, I asked myself what sounds I might mistake for a woman's cough.
The most likely candidate I came up with is a chair leg creaking across the wooden floor. This is consistent with one of the cats jumping off of the chair to get his dinner, especially since there's a chair that he particularly likes to lie on. I don't know that that's what happened, of course, but it makes a heck of a lot more sense than ghosts. (Besides, $CAT sashayed into the kitchen with his tail held high, and not at all as if he'd seen a ghost.)
At any rate, this illustrates why one should be skeptical about reports of UFOs, the paranormal, etc. When someone says "I saw X", usually what they mean is "I saw something that I interpreted as X". Yes, they're sincere, but it's quite possible for people to be sincerely mistaken. And unfortunately, it's not possible to reach into people's heads and pull out what they actually saw or heard, as opposed to what they say they saw or heard. Which means that oral testimony, however sincere, isn't sufficient to prove the existence of UFOs or paranormal phenoma. As the kids say, "Pics or it didn't happen."
(HT to Anne for the meteorologist video.)
(Cross-posted at Epsilon Clue.)
[1] Note to young people: in ancient times, a tape recorder was a device that recorded sounds so you could play them back later. They were very good at playing back ambient sounds. So if you recorded an interview with someone and played it back, you'd hear the creaking of your chair with perfect such clarity that it would drown out the interviewee.
I was just in the kitchen fixing dinner for the cats, when I heard a woman cough. Just a single throat-clearing cough. It definitely sounded like a woman.
And this was odd, since there aren't supposed to be any women in the house. Any company would be unexpected, seeing as how I'm snowed in by Snowmageddon. But it was convincing enough that I checked the living room and looked for fresh tracks in the snow outside. Perhaps someone got stranded and wandered in when she found an accidentally-unlocked door? Or could it be a ghost?
Obviously, there was no one there, as you know, since you've read the spoiler at the top. I tried replaying the sound on the tape recorder of my mind[1] and play it back, trying to figure out what I'd heard, rather than what I thought I'd heard. That is, I asked myself what sounds I might mistake for a woman's cough.
The most likely candidate I came up with is a chair leg creaking across the wooden floor. This is consistent with one of the cats jumping off of the chair to get his dinner, especially since there's a chair that he particularly likes to lie on. I don't know that that's what happened, of course, but it makes a heck of a lot more sense than ghosts. (Besides, $CAT sashayed into the kitchen with his tail held high, and not at all as if he'd seen a ghost.)
At any rate, this illustrates why one should be skeptical about reports of UFOs, the paranormal, etc. When someone says "I saw X", usually what they mean is "I saw something that I interpreted as X". Yes, they're sincere, but it's quite possible for people to be sincerely mistaken. And unfortunately, it's not possible to reach into people's heads and pull out what they actually saw or heard, as opposed to what they say they saw or heard. Which means that oral testimony, however sincere, isn't sufficient to prove the existence of UFOs or paranormal phenoma. As the kids say, "Pics or it didn't happen."
(HT to Anne for the meteorologist video.)
(Cross-posted at Epsilon Clue.)
[1] Note to young people: in ancient times, a tape recorder was a device that recorded sounds so you could play them back later. They were very good at playing back ambient sounds. So if you recorded an interview with someone and played it back, you'd hear the creaking of your chair with perfect such clarity that it would drown out the interviewee.
Tuesday, January 12, 2010
Fishy Invitation
On a social network that I'm on, I recently received an invitation to attend a Landmark Education seminar. The invitation included this description:
This set off my BS-O-Meter: vague generalities with positive connotations, and a high buzzword density: "quantum leap", "outside the box", "breakthrough", "platform", "the next level".
The web site's What People Say page has the testimonials (read: anecodtal data) that one might expect, but also a section called "Independent Research", which I thought would be more objective. Unfortunately, none of the links there point to studies published in peer-reviewed publications. If this program were half as effective as its proponents claim, and they could prove it, surely they'd want the academic repute that would come with a published paper. Instead, the "full study" at the top of the list looks more like a sales brochure, or at best a poorly-done newspaper article, than a psychological or sociological study: no abstract, no references, no detailed discussion of methodology. That pushed my BS-O-Meter further into the red.
The people at the JREF forums aren't as kind as Wikipedia or me. My favorite line from that thread is
In the end, while I don't think the organization qualifies as a cult, it does seem to be a cross between life coaching (as seen on Penn & Teller's Bullshit!) and an Amway-style recruiting scheme. I don't think I'll be attending the seminar.
A breakthrough is often thought of as a one-time event—a quantum leap that moves us “outside the box.” This seminar addresses how to operate powerfully once we are in that new territory—how to manage and sustain a breakthrough and have that new place be a platform for generating the next level of living.
Breakdowns are an integral and critical part of any breakthrough. When breakdowns are seen as stepping stones to a breakthrough - something that was not predictable becomes possible. Breakdowns occur only against a background commitment - they are an occasion for extraordinary action, for making something happen that would not have happened otherwise.
This set off my BS-O-Meter: vague generalities with positive connotations, and a high buzzword density: "quantum leap", "outside the box", "breakthrough", "platform", "the next level".
The web site's What People Say page has the testimonials (read: anecodtal data) that one might expect, but also a section called "Independent Research", which I thought would be more objective. Unfortunately, none of the links there point to studies published in peer-reviewed publications. If this program were half as effective as its proponents claim, and they could prove it, surely they'd want the academic repute that would come with a published paper. Instead, the "full study" at the top of the list looks more like a sales brochure, or at best a poorly-done newspaper article, than a psychological or sociological study: no abstract, no references, no detailed discussion of methodology. That pushed my BS-O-Meter further into the red.
The people at the JREF forums aren't as kind as Wikipedia or me. My favorite line from that thread is
I went through one of the seminars years ago. A key stepping stone on my path to skepticism.
In the end, while I don't think the organization qualifies as a cult, it does seem to be a cross between life coaching (as seen on Penn & Teller's Bullshit!) and an Amway-style recruiting scheme. I don't think I'll be attending the seminar.
Subscribe to:
Posts (Atom)