Sometimes Papers Contain Obvious Lies by Cremieux Recueil*
The authors of scientific papers often say one thing and find another; they concoct a story around a set of findings that they might not have even made, or which they might have actually even contradicted. This happens surprisingly often, and it’s a very serious issue for a few reasons.
Yeah, I'll bet. I am guilty of being one of those people who reads the abstract and trusts it, skimming the rest of the paper, especially graphs and charts. So I would be one of the easiest to be fooled in this way. Bsking has mentioned a few times followup up on a paper, usually a link in a text, to see exactly what it says, only to find that the linked paper is nearly irrelevant to the claimed result, misleading, or even pointing in the opposite direction.
I had never heard of an Everest Regression, but it is easy to get the concept from the explanation in the article. Cremieux discusses a paper he dissected that claimed that students got lower test scores in rooms with higher ceilings. He found, sure enough, that they actually got slightly higher scores.
This paper’s abstract, title, text, and the public remarks to journalists from the authors all implied that was what they found, but their actual result—correctly shown in a table in the paper and reproducible from their code and data—was the complete opposite: higher ceilings were associated with higher test scores! Making a viral hubbub about this managed to get the paper retracted—eventually—but the retraction notice barely mentioned any of the paper’s problems and, instead, said that whatever issues warranted retraction were examples of good ole “honest error.”But he doesn't confine himself to "gotcha" criticism over odd and unimportant experiments - he never does - but focuses on large issues where legislation and policy hinge on what authorities falsely believe is true. For example, he went over a paper from Germany which found that the presence of more foreigners in a region did not affect the crime rate. This is what many people of authority would like to believe, or at least like you to believe, because pretending to reduce crime with the tried-and-untrue methods is politically safer than trying to fix the real problem, which has something to do with (gulp) foreigners. The Everest Regression in action.
this is just controlling for altitude and declaring Everest is hot, or stating that stadiums make people run fast!
*A particularly good example of the letters in a French word not having anything to do with pronunciation.
6 comments:
Hmm. I've read papers where the body, despite the best efforts of the authors, didn't quite support the conclusion, but not one where it contradicted it like that. Yikes.
Chains of citations that change the scope of a paper's original conclusions are a dime a thousand.
Now that I think of it, I read something a story where it did--but that was in a newspaper. The headline bruited how socially significant it was that minority students had a non-minority friend, though not always the reverse--but simply thinking about the statistics of the situation put the lie to the story.
The value of an academic paper is measured by citations, so an ambitious academic writes a paper with citation maximization in mind. This is especially true of obscure ambitious academics, since a paper that supports a claim no one else wishes to make will never be cited and can easily be ignored. I think you might be surprised how seldom the word "truth" is used in meetings of a Tenure and Promotion Committee. It is all but crowded out by "funding," "pubs" and "citations." Whether these are proxies for "truth" depends on the honesty of the whole system.
I was under the impression that the anonymous Crémieux Recueil is a woman, so I'm slightly distracted by the uses of 'he' in the above.
Crémieux is usually a surname. Thus, it can apply to either men or women; but I think the most famous was Adolphe.
I don't know myself, but someone is using the "his" after a podcast interview. https://www.thepodcastbrowser.com/cremieux-recueil-on-using-data-to-simplify-complex-problems/
Post a Comment