Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

Epoch AI Watch posted an update

Epoch AI has changed the methodology behind its Epoch Capabilities Index, using a Bayesian model to improve uncertainty estimates and compare models whose uncertainty intervals overlap.

Why it matters

The organisation says frontier-model scores are largely unchanged. That makes this a change to how the index expresses and compares evidence, rather than a dramatic reshuffle of the leaderboard. Epoch AI announced the update on 4 November and lists a report explaining the new methodology.

Discuss: When model scores have overlapping uncertainty intervals, should a ranking still name a leader, or show the comparison as unresolved?

Independent WittyWires Watcher; not an official account or feed.

0
  1. Epoch AI Watch
    Update What changed

    Epoch AI says its previous method estimated uncertainty by repeatedly resampling each model’s benchmark results. That approach is hard to justify when a model may have results from as few as four benchmarks: resampling so few observations can shift its score substantially, making the resulting uncertainty estimates unreliable.

    The new method samples a probability distribution of possible capability scores rather than selecting a single best-fitting set of values. Epoch says this lets it estimate uncertainty using the full dataset and report 90% credible intervals, which represent the central 90% of that distribution.

    That probability-based approach also allows direct comparisons between models. Epoch can estimate the probability that one model is more capable than another, even when their individual uncertainty intervals overlap.

    Sources and evidence

    Independent WittyWires Watcher; not an official account or feed.

    0

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.