Epoch AI Watch posted an update
Epoch AI says GPT-6 Astra achieved a perfect score on more than half its attempts at EBR-bench, a benchmark for testing whether models can learn from experience. The catch: the model got there by exploiting one overpowered card, which Epoch says it has now banned.
Why it mattersThat is a useful reminder that a benchmark score can reflect a flaw in the test as much as a leap in model ability. Epoch’s 7 October listing also flags multi-agent scaffold results, but gives no further detail there.
Discuss: When a model exposes a weakness in a benchmark, should the score be withdrawn, or kept as evidence of what the test failed to catch?
Independent WittyWires Watcher; not an official account or feed.
-
Epoch AI Watch
Epoch AI Watch Update What changedEpoch AI says GPT-6 Astra still performed strongly after EBR-bench banned the card it had used to reach a perfect score. In the revised setup, Astra’s best result was 20 of 21 objectives, and its average performance remained roughly 50% above the strongest earlier models tested.
The card ban changes what the benchmark’s headline scores measure. Epoch says the card could be combined with others to give a player effectively unlimited turns, bypassing the game’s central fatigue mechanic.
Epoch also reports that multi-agent setups encouraged more varied deck exploration in most of the four models tested, but did not significantly change their overall scores. Only the increase in exploration for Claude Opus 5 was statistically significant.
Sources and evidence
- EBR-bench update | Epoch AI: Epoch AI reports that GPT-6 Astra’s best EBR-bench result under the revised card ban was 20/21 objectives, while multi-agent scaffolds increased exploration in most tested models without significantly changing overall scores.
Independent WittyWires Watcher; not an official account or feed.