Via Bird Dog over at Maggie's
A Simple Argument Proving You Must Not Trust Standard Scientific Evidence. He gets bogged down in describing what he means, but he rather has to. The possibility of misunderstanding requires it. He works in quickly the idea that the focus on p-values and the distortion that creates does not mean one should disbelieve claims along that model, only that we should distrust them. So he got on my good side early with that.
The best part of is is deep in the article, in the conclusion phase, where he goes on a bit of a rant, being quite emphatic, just in case you missed it.
In hypothesis tests, P = “No effect exists”, and Q = “Data more extreme than we actually got”. If Pr(Q|PE) is small (less than the magic number) we are asked to believe Pr(notP|notQE) is high, or act like it is equal to 1. That is another way of saying Pr(effect exists | data we saw & E) = 1 (or high).
Not only is all this forbidden in the theory that gives p-values, though everybody does it, the act itself is a fallacy. It is an invalid argument. It is wrong. It is bad reasoning. It might be true, because of other reasons, that “effect exists” is true or is of high probability (with respect to those other reasons), but that is no justification for accepting the hypothesis test. At all. Ever. Every use of hypothesis tests is fallacious.
Enjoy, stats and logicheads.
1 comment:
We do a poor job of teaching probability, and I think that most people don't fundamentally "get" the idea that, except perhaps in cases where you have to worry about quantum effects, probability statements are statements about our knowledge of the world, not about the world itself. If the weatherman says there's a 65% chance of rain tomorrow, he's just saying that in the context of a model; presumably it either will rain or it won't, and the fact that we don't know for sure which doesn't mean it's undetermined in some deep sense. I don't think this concept is too hard for most people to grasp, if it is carefully explained.
But there's a more practical problem with hypothesis testing at the margins: the fact that we have to resort to logical contortions to convince ourselves that an effect exists at all indicates that, if indeed it does exist, it must be pretty tiny. If my drug is any good, and my trial size is reasonably large, I shouldn't be bragging that there's only a 5% chance of getting of getting results as good or better with a placebo. There shouldn't be a chance in a billion of getting results as good or better with a placebo.
Post a Comment