How Quackery "Works" Part 3: Regression to the Mean

Most quackery and pseudoscience depends on testimonial evidence rather than scientific evidence. In the absence of controlled trials, there are many other explanations for why a treatment seems to work, when it actually doesn't. This series of posts aims to address those explanations and to highlight how anecdotal evidence is no replacement for controlled scientific study.

How a person feels isn't static, it fluctuates. You can feel tired one day but not the next. If you suffer from migraines, one can be more severe than the next. How "well" you feel can vary from day to day, even if overall your condition isn't changing (neither improving nor declining). Regression to the mean is the tendency of extreme measurements to be less extreme (closer to the mean) on subsequent measurements. A example that should be familiar to most sports fans is the concept of the 'sophomore slump'. This is when an athlete has performs more poorly in their second season than they did in the rookie year. The reason for this isn't a slump per se, but rather due to the fact that the first measurement - an outstanding rookie season - lies at an extreme, while the second measurement is closer to the 'mean' performance for that player. That is, it's closer to the average standard of performance for that player. It looks like a slump, but in fact is due to random fluctuations in player performance.

The same principle holds for medical testimonial. An ineffective treatment may seem to work based on random fluctuation of symptoms, resulting in positive testimonial. In fact, people who are sick are more likely to look to experimental remedies and unconventional treatment when they feel at their worst. Since this first 'measurement' is at an extreme of their condition, subsequent 'measurements' of how they feel will be an improvement for purely statistical reasons as opposed to an effective treatment. Evidence-based medicines rely on multiple measurements and large sample sizes (a testimonial is a sample size of 1) to account for normal disease fluctuation and get statistically meaningful results.