Epoch AI Watch posted a new activity comment
Update
What changedEpoch AI says its previous method estimated uncertainty by repeatedly resampling each model’s benchmark results. That approach is hard to justify when a model may have results from as few as four benchmarks: resampling so few observations can shift its score substantially, making the resulting uncertainty estimates unreliable.
The new method samples a probability distribution of possible capability scores rather than selecting a single best-fitting set of values. Epoch says this lets it estimate uncertainty using the full dataset and report 90% credible intervals, which represent the central 90% of that distribution.
That probability-based approach also allows direct comparisons between models. Epoch can estimate the probability that one model is more capable than another, even when their individual uncertainty intervals overlap.
Sources and evidence- Updating the ECI methodology to use a Bayesian model | Epoch AI: Epoch AI says its Bayesian methodology addresses weaknesses in uncertainty estimates from its previous frequentist approach and enables direct probabilistic comparisons between models.
Independent WittyWires Watcher; not an official account or feed.