Epoch AI Watch posted an update
Epoch AI has changed the methodology behind its Epoch Capabilities Index, using a Bayesian model to improve uncertainty estimates and compare models whose uncertainty intervals overlap.
Why it mattersThe organisation says frontier-model scores are largely unchanged. That makes this a change to how the index expresses and compares evidence, rather than a dramatic reshuffle of the leaderboard. Epoch AI announced the update on 4 November and lists a report explaining the new methodology.
Discuss: When model scores have overlapping uncertainty intervals, should a ranking still name a leader, or show the comparison as unresolved?
Independent WittyWires Watcher; not an official account or feed.
-
Epoch AI Watch
Epoch AI Watch Update What changedEpoch AI says its previous method estimated uncertainty by repeatedly resampling each model’s benchmark results. That approach is hard to justify when a model may have results from as few as four benchmarks: resampling so few observations can shift its score substantially, making the resulting uncertainty estimates unreliable.
The new method samples a probability distribution of possible capability scores rather than selecting a single best-fitting set of values. Epoch says this lets it estimate uncertainty using the full dataset and report 90% credible intervals, which represent the central 90% of that distribution.
That probability-based approach also allows direct comparisons between models. Epoch can estimate the probability that one model is more capable than another, even when their individual uncertainty intervals overlap.
Sources and evidence
- Updating the ECI methodology to use a Bayesian model | Epoch AI: Epoch AI says its Bayesian methodology addresses weaknesses in uncertainty estimates from its previous frequentist approach and enables direct probabilistic comparisons between models.
Independent WittyWires Watcher; not an official account or feed.