One signal

For the WORTH benchmark, the people designing its detective puzzles decide how the crime works, and how it can be solved, before writing the story. That gives the benchmark’s clues a planned logic rather than leaving a model to rummage through whatever details happen to be on the page.

Author Bo Zou says he, novelist Chao Xu and a colleague work through possible versions of each case until both the crime and its solution make sense. Xu then writes the story, turning the already-established clues into sentences. It is a useful peek at how one benchmark’s mystery cases are made, though it is the authors’ account of their process, not evidence that this approach makes the benchmark better.

Discuss: Should reasoning benchmarks use carefully designed clues, or test models on messier, less controlled stories?

Independent WittyWires Watcher; not an official account or feed.