The tendency is strong that we consider our own personal experience to be representative of what has happened to everyone else. When we hear an explanation for events, we immediately fit it against our own lives, or children, or jobs. If twenty people hear that teenage converts to Christianity tend to be Evangelical at first and then move to mainline churches, nineteen of them will immediately confirm or deny the truth of the statistic on the basis of what happened to them, or their children, or whoever the nearest teenage convert is. I used to think this was more common among women, but I am now convinced this is not so. I even begin to suspect the opposite is true.
"My friend's cousin got the vaccine, and two weeks later he got bitten by a moose. Do the research, people."
I don't quite know what to say to such folks. If you don't already know that the incident that happened near your granddaughter's soccer game is not necessarily representative of what happens in general, I don't think I have a ready explanation to convince you. I readily see why people start there. Sometimes one can actually develop a refutation by noticing that one incident, at least, sharply contradicts that. Whatever else is true about the hairdressers in America, you might be able to assure the assembled crowd "Not always. My hairdresser doesn't own any small dogs at all."
I think we apply this unscientific experiment with N=1 more often when we do not understand the underlying mechanism or have little grasp of the data. It is the opposite of the Bayesian approach, where we first ask how many highschool students there are, then what percentage of highschoool students are basketball players, and what percentage of students in general are taking chemistry this year to begin with before we try and figure out how many basketball players are taking chemistry this year. Looking at whether your kid is a small forward in chem lab is useless. Even estimating from a few years' teams and school enrollments would be more useful.
I suppose you have to start with what you know. I agree, it is frustrating. And since your friends and family are often apt to be like you, your observations start out somewhat skewed.
ReplyDeleteWhat comes to mind for me is a village a millennium ago. Strangers are bad people. Why? Well, traders are useful, but apt to cheat you. Tax collectors are always bad news. Clergy may be OK, but often they turn out to be a species of tax collector. And travelers who aren't either--there's usually a good reason their own families didn't want them around, and we painfully learn what it was.
QED.
Moose have big lips. Big feet, too,
ReplyDeleteBayes had nothing on your hairdresser. Many people do not realize the extent to which availability bias, unless we make an effort to counteract it, dominates much of what we think we know.
ReplyDeleteWhy do you assume data (in general) is accurate, or is accurately quoted?
ReplyDeleteThere is a replication crisis in research. (To decrease the number of links, I suggest the readers search for the term "replication crisis, although I expect readers of this blog will be familiar with the problem.) Canny investors have learned to be wary of statistics offered in other areas: https://bothsidesofthetable.com/73-6-of-all-statistics-are-made-up-3c30e8ff272#more-1930.
For example, if someone told me that anvils tended to float off on windy days in Canada, my first impulse would not be to believe the speaker. Certainly, I would observe that in my experience, anvils don't float. (Road Runner cartoons don't count.) Even if the speaker were an authority on anvils, or gravity, I would be skeptical.
In my experience, some people lie. Some people are more likely to lie when incentives are in play. Some people believe them. Caveat emptor applies, as does TANSTAAFL. People do make statements that are contrary to observed happenings, such as, oh, "printing money doesn't lead to inflation."
I may not be understanding your point in your original post. You stated that, "If twenty people hear that teenage converts to Christianity tend to be Evangelical at first and then move to mainline churches, nineteen of them will immediately confirm or deny the truth of the statistic on the basis of what happened to them, or their children, or whoever the nearest teenage convert is."
Well, they should ask for the source of the statistic, right? "Do you have a source for that? Was it an Evangelical church representative trying to gain support for cooperation between churches?" But that might sound too hostile for social chatter, so they say something like, "that reminds me of my wife's cousin, who moved to Georgia..."
You are already in the advanced class, cranberry. Those nineteen people would never think to verify the information in any way. That was my point.
ReplyDeleteThere is an exception to your point, which arises when one is arguing against a universally-quantified claim. Claims similar to "All X are Y" can be disproven by a single counterexample of an X that is not a Y. Personal experience is a perfectly good source of such counterexamples; after all, you only need one, and this is the subset of things you have good reason to be to be true because you were there and observed it. You could be wrong, but your evidence is as strong here as it is likely to get.
ReplyDeleteAside from that, though, it's generally right that personal experience is of limited value applied to whole societies or broad phenomena.
@James:
"Clergy may be OK, but often they turn out to be a species of tax collector"
This reminds me of Sir Walter Scott's wonderful poem/song, "The Barefooted Friar."
https://www.poetry.com/poem/35558/the-barefooted-friar