How to make your data significant

Anyone who's been in science long enough to either get a grasp of statistics, or alternatively figure out empirically how to game it, knows that given enough parameters and data points you will inevitably reach statistical significance on something. So what's the difference between a p-value of 0.051 and 0.049? Well this guy sums it up with a good anecdote :

"About two years ago the Wall Street Journal (registration required) investigated the statistical practices of Boston Scientific, who had just introduced a new stent called the Taxsus Liberte.

Boston Scientific did the proper study to show the stent worked, but analyzed their data using an unfamiliar test, which gave them a p-value of 0.049, which is statistically significant.

The WSJ re-examined the data, but used different tests (they used the same model). Their tests gave p-values from 0.051 to about 0.054; which are, by custom, not statistically significant.

Real money is involved, because if “significance” isn’t reached, Boston Scientific can’t sell their stents. But what the WSJ is quibbling, because there is no real-life difference between 0.049 and 0.051. P-values do not answer the only question of interest: does the stent work?"

[...]

" Significance is vaguely meaningful only if both a model and the test used being are true and optimal. It gives no indication of the truth or falsity of any theory.

Statistical significance is easy to find in nearly any set of data. Remember that we can choose our model. If the first doesn’t give joy, pick another and it might. And we can keep going until one does."

Statistics for Dummies

For those of you who can't remember the difference between standard deviation and standard error of the mean, or who never bothered to learn this paper is a good primer on when to use which, what independent replicates really means, and how to interpret the error bars on the graph presented at yesterday's seminar. It's the kind of thing we should all know but many of us don't bother with.

[h/t: juniorprof]

Meta-analysis

One week caffeine is good for you, the next it isn't. One week smoking marijuana is worse than cigarettes, the next cigarettes are worse. Which is true? How is the general public supposed to sort out which medical claims are true, and which aren't when contradictory findings are blared across the front pages of the newspaper every week?

Part of the problem is meta-analysis, the analysis of combined results of previous studies. Meta-analysis can be a powerful tool as a researcher, particularly when sample sizes aren't large enough to reach statistically significant conclusions. Business Week has a nice article discussing meta-analyses (a meta-meta-analysis??) and some of the associated pitfalls with that approach, mainly the lack of raw data for analysis and the introduction of bias with study selection. From the article:

'"We know there is publication bias," says Frank E. Harrell Jr., chair of biostatistics at Vanderbilt University. It's much easier to get a study published that says, "something works!" than one saying, "Oops, the treatment had no effect." Using published data alone thus typically makes the final result more positive.'

The bottom line, really, is the need to look at the methodology (behind ANY study) before accepting a conclusion. "If people understand the process of science better, they'll be able to spot the gray reality behind the next black-and-white headline."