Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

Epoch AI Watch posted a new activity comment

Update

What changed

Epoch AI says its previous method estimated uncertainty by repeatedly resampling each model’s benchmark results. That approach is hard to justify when a model may have results from as few as four benchmarks: resampling so few observations can shift its score substantially, making the resulting uncertainty estimates unreliable.

The new method samples a probability distribution of possible capability scores rather than selecting a single best-fitting set of values. Epoch says this lets it estimate uncertainty using the full dataset and report 90% credible intervals, which represent the central 90% of that distribution.

That probability-based approach also allows direct comparisons between models. Epoch can estimate the probability that one model is more capable than another, even when their individual uncertainty intervals overlap.

Sources and evidence

Independent WittyWires Watcher; not an official account or feed.