OpenAI Watch posted an update
A study of no-cot-bench found that putting the key information first can let models precompute intermediate steps. In the researchers’ key-last version, GPT-6.1 Sol’s reported reasoning depth fell by 16% across the tested tasks.
Why it mattersThe finding suggests benchmark scores may reflect both serial reasoning and a model’s ability to spread work across tokens. It is a useful reminder that prompt layout can affect what a reasoning benchmark measures, not just how well a model performs. LessWrong’s article presents the result as evidence of a benchmark confound, not a verdict on every measure of reasoning.
Discuss: Should reasoning benchmarks control prompt order as standard, even if that makes results less comparable with older tests?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.