Can most studies be wrong? - Part 2

• Part 1 (previous)
• Part 2 (current)
• Part 3 (next)

In part 1 we discussed a 2005 paper by Ionnidis entitled "Why Most Published Research Findings Are False" [Full Text] [PMID: 16060722] that shows that there exist reasonable assumptions that could imply that most studies are false. That paper is the most downloaded paper from the PLoS Medicine journal. Here we discuss a 2007 paper by Goodman and Greenland who further illuminate the one by Ioannidis. They basically agree with Ionnidis pointing out that the argument is based on three basic components:

1. that more than half the hypotheses that scientists consider are false, i.e. the prior probability is less than half.
2. the division of studies into ones that are significant at the 5% level and ones that are not together with significance seeking on the part of researchers
3. use of a mathematical argument (Bayes theorem) to show that with a low prior probability and weak evidence that the likelihood of the studies being correct is less than half (i.e. the posterior probability is less than half).

The methodology is insufficient and the bias too large to overcome the handicap of having such a large percentage of false hypotheses to wade through to get to the true ones. The authors point out that there may be dependence between studies, e.g. a study on cardiac function and mortality might suppress the lack of improvement in mortality if it were nevertheless found that it improved cardiac function. Also they point out that the degree of bias may vary substantially from one study to another. These points and others affect the mathematics used to imply that most studies are false. Nevertheless, the authors of the response do basically agree with the original paper and the basic purpose of this paper seems mainly to clarify the original by framing it in the terms discussed above.

• Part 1 (previous)
• Part 2 (current)
• Part 3 (next)