How Quackery "Works" Part 5: Recall Bias

Most quackery and pseudoscience depends on testimonial evidence rather than scientific evidence. In the absence of controlled trials, there are many other explanations for why a treatment seems to work, when it actually doesn't. This series of posts aims to address those explanations and to highlight how anecdotal evidence is no replacement for controlled scientific study.

A common psychological bias that plagues retrospective and self-reporting studies is recall bias. This is when answers aren't based strictly on observation but also on the responder's memory. Because memories can be influenced or otherwise unreliable, relying on recall to draw conclusions can be problematic. This bias is unintentional, and can often be based on expectation. For example, if you have a cavity you may 'remember' eating more sweets than usual, even if it's not the case. There are other examples in the literature: A study of mothers in Brazil found that richer or better educated women were more likely to over-recall breast-feeding durations. A more relevant example is recall bias among parents of autistic children. After publication of a paper suggesting a link between autism and the MMR vaccine, parents were more likely to "recall the onset [of autism] as shortly after MMR more often than parents of similar children who were diagnosed prior to the publicity." Preconceived ideas of what the answer is (or should be) can taint recall.

Testimonials given in support of a product are also subject to recall bias. Given the above example, one can easily imagine a now-healthy person recalling that they started to get better 'around the same time' as starting a supplement even if there are other reasons for an improvement. This inaccuracy of memory, or even differences in what two groups are likely to recall can lead to incorrect conclusions and false positive testimony. Recall bias can be a problem for evidence-based clinical studies as well. Anytime memory and self-reporting is involved, this opens an opportunity for this kind of unconscious bias. A good study needs to take this into account in its design. Ideally, direct empirical measurements are used to eliminate recall and self-reporting altogether, but that isn't always possible. In retrospective studies, good questionnaire design helps minimize recall bias. For example, detailed questions or blinding the subject to the hypothesis help with accurate reporting. While neither evidence-based studies nor anecdote aggregation is immune, a product pitch based on several individual self-reported testimonials is more likely to suffer from recall bias than a well-designed retrospective study.

Read more about recall bias here.