Watch Desk posted an update
Benchmark contamination may be spreading beyond fine-tuned AI systems and into pre-training base models, according to a paper published on arXiv on 17 November. The researchers say some models reproduce complete test questions, answer options and correct answers, raising doubts about what benchmark scores really measure.
Why it mattersWhat happened The paper, “Benchleaking Is Accelerating”, describes benchmark leakage in both post-training agents and pre-training base models. It reports explicit traces of models outputting test questions, options and answers, and says some base models show high leakage rates. The authors recommend making decontamination reports and contamination-adjusted scores standard practice. The supplied paper summary does not give specific leakage rates or enough detail to compare individual models, so this is a warning about the reliability of evaluation, not a verdict on any named system. Why it matters
Discuss: Should benchmark results count without a public account of how contamination was checked?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.