Bias in meta-analysis detected by a simple, graphical test¶
Why this mattered¶
Egger and colleagues’ paper mattered because it changed meta-analysis from a mainly aggregative technique into a field with routine diagnostics for its own vulnerability to distortion. Before this, funnel plots were known as a visual warning sign, but the paper made asymmetry operational: a regression of standardized treatment effects on study precision could turn the suspicion of “small studies show larger effects” into a reproducible statistical test. Its empirical comparison with large trials gave the point clinical force: when meta-analyses disagreed with later large trials, the meta-analyses tended to show larger benefits, and funnel plot asymmetry was concentrated in those discordant cases.
The paradigm shift was not that Egger’s test solved publication bias, but that it made bias in evidence synthesis measurable enough to become part of standard practice. After this paper, systematic reviewers could no longer treat a pooled estimate as self-validating simply because it combined many studies. They had a compact method for asking whether the literature being pooled was itself selectively visible, methodologically uneven, or dominated by small-study effects. That made possible a more skeptical, second-order evidence culture: meta-analyses came to be judged not only by inclusion criteria and statistical heterogeneity, but also by whether their evidentiary base looked systematically distorted.
Its influence runs through later developments in evidence-based medicine: routine funnel-plot assessment, small-study-effect tests, trim-and-fill methods, selection models, prospective trial registration, reporting standards, and the broader meta-epidemiological study of how design and publication processes shape reported effects. The paper also established a caution that remained important: asymmetry is not a pure detector of publication bias, and the test has limited power when few small trials exist. Its lasting contribution was therefore both technical and epistemic: it gave reviewers a simple alarm for a pervasive failure mode of biomedical evidence, while reminding them that detecting bias is itself an uncertain inference rather than a mechanical correction.
Abstract¶
Abstract Objective: Funnel plots (plots of effect estimates against sample size) may be useful to detect bias in meta-analyses that were later contradicted by large trials. We examined whether a simple test of asymmetry of funnel plots predicts discordance of results when meta-analyses are compared to large trials, and we assessed the prevalence of bias in published meta-analyses. Design: Medline search to identify pairs consisting of a meta-analysis and a single large trial (concordance of results was assumed if effects were in the same direction and the meta-analytic estimate was within 30% of the trial); analysis of funnel plots from 37 meta-analyses identified from a hand search of four leading general medicine journals 1993-6 and 38 meta-analyses from the second 1996 issue of the Cochrane Database of Systematic Reviews . Main outcome measure: Degree of funnel plot asymmetry as measured by the intercept from regression of standard normal deviates against precision. Results: In the eight pairs of meta-analysis and large trial that were identified (five from cardiovascular medicine, one from diabetic medicine, one from geriatric medicine, one from perinatal medicine) there were four concordant and four discordant pairs. In all cases discordance was due to meta-analyses showing larger effects. Funnel plot asymmetry was present in three out of four discordant pairs but in none of concordant pairs. In 14 (38%) journal meta-analyses and 5 (13%) Cochrane reviews, funnel plot asymmetry indicated that there was bias. Conclusions: A simple analysis of funnel plots provides a useful test for the likely presence of bias in meta-analyses, but as the capacity to detect bias will be limited when meta-analyses are based on a limited number of small trials the results from such analyses should be treated with considerable caution. Key messages Systematic reviews of randomised trials are the best strategy for appraising evidence; however, the findings of some meta-analyses were later contradicted by large trials Funnel plots, plots of the trials' effect estimates against sample size, are skewed and asymmetrical in the presence of publication bias and other biases Funnel plot asymmetry, measured by regression analysis, predicts discordance of results when meta-analyses are compared with single large trials Funnel plot asymmetry was found in 38% of meta-analyses published in leading general medicine journals and in 13% of reviews from the Cochrane Database of Systematic Reviews Critical examination of systematic reviews for publication and related biases should be considered a routine procedure
Related¶
- cite → STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT — Egger’s meta-analysis bias test cites Bland-Altman because both use simple graphical methods to detect systematic statistical discrepancies.
- cite → The Effect of Intensive Treatment of Diabetes on the Development and Progression of Long-Term Complications in Insulin-Dependent Diabetes Mellitus — Egger et al. use the diabetes intensive-treatment trial as an example dataset for detecting publication bias with funnel-plot asymmetry.
- enables ← STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT — Bland-Altman plotting used differences against averages to reveal bias graphically, a template for Egger's funnel-plot test of meta-analysis bias.