ideasIdea
External validation at five or more sites in two countries before clearance
No cancer AI would be approved until it has been tested on patients from at least five different hospitals in at least two countries, none of which contributed training data.
Many cleared AI devices were validated on data from one or two sites, often overlapping with development sites. The proposal sets a minimum external validation requirement (at least five independent sites, at least two countries or health systems, no training-site overlap, pre-registered analysis, subgroup reporting) for regulatory clearance of cancer AI, with the sequestered benchmarks as one accepted route.
Hypothesis
Models meeting the requirement will show smaller performance drops on deployment than models cleared under current rules, and the requirement will not materially slow clearance for well-built models.
Rationale
Generalisation failure across sites is the best-documented failure mode of medical AI; multi-site external validation is the direct test and is inexpensive relative to the harm of deploying brittle models.
What would test it
Compare post-deployment performance drop for cleared models grouped by number of external validation sites; if the association holds, adopt the requirement and re-measure.
Maturity
speculative
Who has to act
regulator
Cost to try
Small (under $1M)
Years to first evidence
2
Bottlenecks it attacks
- AI that is built but not validated or deployed · Thousands of cancer AI models are published; a handful are in clinical use, and fewer have shown they help patients.
- Regulatory divergence between regions · Regulatory divergence means a drug approved in one country can take years to reach another, or never arrive.