Bayesian Analysis - Parapsychology and Vaccines

The most recent podcast on "The Studies Show" (reachable from the Stuart Ritchie link in my sidebar) is a special Hallowe'en episode about parapsychology, or more exactly, what the research shows about "psi." Spoiler Alert:  It still doesn't work, despite the occasional studies even in respectable journals showing that it does, a little, with certain people. 

The podcast was intriguing for a claim they made about prior beliefs, which are part of Bayesian Analysis. There are many studies that show that psi is bosh in all its forms: precognition being the most studied.  If you are one who already believed that, the studies only strengthen your belief.  The few studies that show otherwise do not necessarily undermine your belief that it is bosh, however. And especially, studies that show a strong effect have the reverse result of making you suspicious of the researchers, that they are frauds, or at least in error. If they keep coming up with what you think are unlikely results, it does not dent your priors, it reinforces your belief that something else must be up.

They related it to an earlier episode they had done on Growth Mindset, which the Stanford researcher showed dramatic results for at first.  Other studies showed weaker results, and it was a fascinating podcast listening to them describe how better and better studies - and they described what that means in terms of closing off the loopholes and escape routes into what people want to be true, including the motive that they are getting paid 5 and 6-figure fees to speak before corporate groups about it - showed less and less effect. Finally, the very best studies - large, avoiding publication  bias, double-blind, preregistered, and about six other correctives - showed no effect or almost no effect. Such things have the effect of undermining their trust in the researchers, not the results.

So in Psi research, they thought it was quite reasonable that continued good results, when interpreted in a Bayesian sense, did not undermine their own belief in the ordinary rules of physics, cause and effect, and linearity of time, they undermined their trust in those particular researchers, and even the methods they used.  For example, they are increasingly distrustful of meta-analyses in general because the abuse of them is so common now. Following on, Stuart described what he thought was the best psychological study he had ever seen, because it closed off every known escape route on parapsychology: it was antagonistic research, including people with different priors from the outset; it used "born open" data (which I had never heard of), meaning that followers online could see the responses and answers as quickly as the researchers could, in the moment that the subjects answered on their keyboards and the widespread laboratories around the world. The podcast is worth it for that alone.  His discussion of the Transparent Psi Project begins at about the 49-minute mark. 

But for those more interested in current events and the interaction between culture and science, the searchlight was turned on the evidence of vaccine efficacy. The claim was that for those who already had some trust in researchers, the repeated studies showing vaccine safety and effectiveness reinforced their belief that they were worth it. But for those who already had distrust of academic and government- or corporate-funded researchers, no matter what good effects the studies showed, they did not dent the mistrust of vaccines one bit.  They dented those folks' trust in the researchers instead. A positive vaccine study reinforced their belief that "the BBC is in on it and won't report the truth," or "Big Pharma is suppressing the true results."

This got worse as time passed, so that eventually no study, by anyone, convinced those who were conspiracy-minded. They only widened the circle of who they mistrusted. In the same way that the believers in parapsychology dug in deeper as results went against them, so too did the distrust in vaccines. 

Your priors matter, even with a powerful tool like Bayesian analysis. It's worth going all the way back to those frequently, rather than plodding down your current road regardless of the new data.