Discrepancies between meta-analysis and large randomized clinical trials


(LeLorier et al., 1997)  Investigated the ability of meta-analyses to predict the results in large clinical trials. They described their findings as follows


We identified 12 large randomized, controlled trials and 19 meta-analyses addressing the same questions. For a total of 40 primary and secondary outcomes, agreement between the meta-analyses and the large clinical trials was only fair (kappa= 0.35; 95 percent confidence interval, 0.06 to 0.64). The positive predictive value of the meta-analyses was 68 percent, and the negative predictive value 67 percent. However, the difference in point estimates between the randomized trials and the meta-analyses was statistically significant for only 5 of the 40 comparisons (12 percent). Furthermore, in each case of disagreement a statistically significant effect of treatment was found by one method, whereas no statistically significant effect was found by the other. CONCLUSIONS: The outcomes of the 12 large randomized, controlled trials that we studied were not predicted accurately 35 percent of the time by the meta-analyses published previously on the same topics.


In their analysis they failed to take account of the inherent fallibility of any prediction not based on infinite amounts of data, nor did they critically examine the alternative to meta-analysis. For example, they could have checked to see whether the second largest trial predicted the largest trial better than did the meta-analysis.


This project will consist of a more careful investigation of this question, paying particular attention to any inherent limits on predictability imposed by sampling variation.


LeLorier, J., Gregoire, G., Benhaddad, A., Lapierre, J., & Derderian, F. (1997), Discrepancies between meta-analyses and subsequent large randomized, controlled trials New England Journal of Medicine, 337, 536-542.


Back to Stephen Senn's research page