Reproducibility Project: Cancer Biology found that landmark preclinical results mostly shrank or vanished on replication
An eight-year effort to repeat 50 experiments from 23 high-impact cancer biology papers found that replication effect sizes were on average 85% smaller than the originals, and fewer than half of the effects replicated by most criteria.
The Center for Open Science and Science Exchange set out in 2013 to replicate key experiments from 53 high-impact cancer biology papers published 2010-2012, with registered reports and original-author consultation. Only 50 experiments from 23 papers could be completed; the companion paper (Errington et al., eLife 2021, 'Challenges for assessing replicability') documents that no original paper contained enough methodological detail to design a replication without contacting the authors, and that about a third of authors were unhelpful or unresponsive.
Across 158 measured effects, the median replication effect size was 85% smaller than the original; 92% of replication effects were smaller than the originals. Using five criteria, 46% of effects replicated on more criteria than they failed; for original positive results, about 40% replicated, while null results replicated at 80%.
The project quantified the preclinical reproducibility problem that pharmaceutical groups (Begley and Ellis 2012; Prinz 2011) had reported anecdotally, and shaped funder requirements for rigour and data sharing.
- 50 experiments from 23 papers completed out of 193 planned from 53 papers
- Median replication effect size 85% smaller than original; 92% of replication effects smaller than originals
- 46% of 158 effects replicated on more criteria than they failed; original positive results replicated about 40% of the time, null results 80%
- No original paper described methods in enough detail to replicate without author contact; 32% of authors were minimally helpful or unresponsive
Many exciting laboratory findings that motivate drug programmes are weaker or less reliable than published, which helps explain the high failure rate of drugs entering clinical trials. It argues for pre-registration, detailed methods, data sharing and independent replication before major translational investment.
- Replications used the original protocols where possible, but reagents, animals and laboratories differ; some failures may reflect context sensitivity rather than error
- Selection of high-impact papers may not represent the field
- Under-powered replications could miss true effects; effect-size shrinkage is the more robust finding
- Only about a quarter of planned experiments were completed, which limits generalisation
Pages like this
not linked directly; found by shared links- IdeaAn independent replication institute that re-tests key preclinical cancer findings before trials
Shares Patient-derived xenografts, Patient-derived organoids, Preclinical results do not reproduce, The valley of death between lab and product.
- IdeaMulti-centre randomised animal trials before committing to a human trial
Shares Patient-derived xenografts, Preclinical results do not reproduce, The valley of death between lab and product, Preclinical models that do not predict people.
- IdeaScore every model system on how well it predicted real trial results
Shares Patient-derived xenografts, Patient-derived organoids, Preclinical results do not reproduce, Preclinical models that do not predict people.
- IdeaSelf-driving laboratories that run the cancer biology hypothesis loop autonomously
Shares Patient-derived organoids, Preclinical results do not reproduce, The valley of death between lab and product, Preclinical models that do not predict people.
- IdeaMake in vivo metastasis screens a required step in drug discovery
Shares Patient-derived xenografts, Patient-derived organoids, Failures are hidden, Preclinical models that do not predict people.
- IdeaShared reference organoid and PDX panels that every lab can test against
Shares Patient-derived xenografts, Patient-derived organoids, Preclinical results do not reproduce, Preclinical models that do not predict people.
- InstitutionThe Jackson Laboratory Cancer Center
Shares Patient-derived xenografts, Preclinical results do not reproduce, Preclinical models that do not predict people.
- IdeaPre-specified sample sizes for animal studies; no more 'representative' experiments
Shares Patient-derived xenografts, Preclinical results do not reproduce, Preclinical models that do not predict people.