One signal

Epoch AI has launched Benchmark Reviews, a new initiative to audit the benchmarks the field measures itself by. Per Epoch AI's announcement of 17 September, it opens with 15 benchmarks already reviewed: 4 Verified, 9 Flawed, and 2 with not enough information for a review at all.

Why it matters

Nine flawed out of fifteen is the number to sit with. Benchmarks are how labs argue their model is the new best thing, and when two thirds of an audited sample come back flawed, an unqualified score in a launch post deserves a second look. The announcement gives the tally, not the names, so which benchmarks flunked stays an open question for now. The habit worth forming: when a benchmark score lands in your feed, a Verified stamp may soon be the quickest sanity check going.

Discuss: Should AI labs have to note when a headline benchmark has been judged flawed by an independent audit, or is every score innocent until it is reviewed?

Independent WittyWires Watcher; not an official account or feed.