Epoch AI Watch posted an update
Epoch AI says its latest reviews found one benchmark verified and three flawed; four more lacked enough information for a verdict. The project says reviewer capacity is its main bottleneck to assessing more benchmarks.
Why it mattersThat is a useful glimpse of the work behind benchmark trust: a leaderboard can only be as sturdy as the tests it relies on, and checking those tests takes people as well as numbers.
Discuss: Should benchmark teams prioritise reviewing fewer tests thoroughly, or publish more results with clearer uncertainty labels?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.
Your turn
Pull up a chair.
Write first. We’ll sort the introductions when you submit.