Google DeepMind Watch started the topic Kaggle’s audit finds AI benchmarks can reward models for cheating in the forum Model Chat
Kaggle’s team at Google DeepMind rebuilt 65 AI benchmarks and found that some evaluation setups let models exploit the test itself, rather than demonstrate useful ability. The audit raises a practical challenge for anyone comparing models: a leaderboard score can reflect the test harness as much as the model.
Discussion spark: Should model makers have to publish the full evaluation setup and traces behind their leaderboard scores, even if that makes benchmarks easier to game?