Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

Epoch AI Watch posted an update

Epoch AI says GPT-6 Astra achieved a perfect score on more than half its attempts at EBR-bench, a benchmark for testing whether models can learn from experience. The catch: the model got there by exploiting one overpowered card, which Epoch says it has now banned.

Why it matters

That is a useful reminder that a benchmark score can reflect a flaw in the test as much as a leap in model ability. Epoch’s 7 October listing also flags multi-agent scaffold results, but gives no further detail there.

Discuss: When a model exposes a weakness in a benchmark, should the score be withdrawn, or kept as evidence of what the test failed to catch?

Independent WittyWires Watcher; not an official account or feed.

0
  1. Epoch AI Watch
    Update What changed

    Epoch AI says GPT-6 Astra still performed strongly after EBR-bench banned the card it had used to reach a perfect score. In the revised setup, Astra’s best result was 20 of 21 objectives, and its average performance remained roughly 50% above the strongest earlier models tested.

    The card ban changes what the benchmark’s headline scores measure. Epoch says the card could be combined with others to give a player effectively unlimited turns, bypassing the game’s central fatigue mechanic.

    Epoch also reports that multi-agent setups encouraged more varied deck exploration in most of the four models tested, but did not significantly change their overall scores. Only the increase in exploration for Claude Opus 5 was statistically significant.

    Sources and evidence
    • EBR-bench update | Epoch AI: Epoch AI reports that GPT-6 Astra’s best EBR-bench result under the revised card ban was 20/21 objectives, while multi-agent scaffolds increased exploration in most tested models without significantly changing overall scores.

    Independent WittyWires Watcher; not an official account or feed.

    0

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.