How Quackery "Works" Part 6: Confirmation Bias

Most quackery and pseudoscience depends on testimonial evidence rather than scientific evidence. In the absence of controlled trials, there are many other explanations for why a treatment seems to work, when it actually doesn't. This series of posts aims to address those explanations and to highlight how anecdotal evidence is no replacement for controlled scientific study.

So far, this series has tried to explain reasons for false positives - how people can think something is working when it isn't. The next couple of posts will focus on why those false positives seem to be all that's reported. It's a common refrain in the face of skepticism that "there are none who have anything negative to say about the product." In the comments of the last post on recall bias, AC pointed out that you're more likely to recall the one time a treatment 'works' than the hundreds of times that it doesn't. Confirmation bias is searching for or interpreting facts in a way that supports or confirms existing beliefs (or, conversely, ignoring evidence that contradicts existing belief). The Skeptic's Dictionary offers the following example:

" if you believe that during a full moon there is an increase in admissions to the emergency room where you work, you will take notice of admissions during a full moon, but be inattentive to the moon when admissions occur during other nights of the month."

(More on that particular fiction here) Confirmation bias is an underlying element of all the reasons for positive testimony previously discussed in this series. The placebo effect causes temporary relief - this is confirmatory. Fluctuating symptoms give the appearance of a working treatment - this is confirmatory. When AC (quoted above) says you're more likely to recall the one time when something works than the times when it doesn't it isn't just about remembering, it's about a natural cognitive bias to confirm our hypotheses.

Deliberate or no, this leads to an accumulation of testimony asserting a treatment works. And this effect is self-amplifying: someone who purchases the next miracle pill searches out confirming testimony. This confirming data convinces the purchaser that the product *does* work, leading to yet another piece of positive testimony. Scientists are not immune to this, and similarly publication bias is the tendency for positive results to be treated differently from negative ones. To combat confirmation bias, one must consider all evidence - whether it confirms or not - and, rely on solid experimental evidence. Furthermore, the peer-review process helps minimize false results due to confirmation bias, unless the reviewers hold the same bias themselves.