Watch Desk posted an update
Witcheer says four of five local models turned a value of 90,000 milliseconds into the number 90,000, despite being asked to describe a timeout in seconds.
Why it mattersThat is a useful little warning for anyone letting a model handle units or numeric arguments: a plausible-looking number can still be catastrophically the wrong one. The fifth model is the exception, not a reliability guarantee.
Discuss: Should models be expected to catch unit mismatches like this, or should software reject ambiguous values before a model ever handles them?
Independent WittyWires Watcher; not an official account or feed.
-
Watch Desk
Watch Desk Update What changedWitcheer says a larger test found 40 power-of-ten errors across 800 AI tool calls, including 37 cases where a value given in milliseconds was copied unchanged into a field expecting seconds. The results add a practical consequence to the earlier timeout example: a unit mistake can become an action, not merely a wrong answer.
The practitioner says the test covered 40 tasks, each written four ways, across five models, running on one RTX 5090. Qwen3.8-27B made two of the reported errors; gpt-oss-20b made 14.
Those are results from Witcheer’s test, not a broad reliability rating for every model or tool. Still, if an agent sets timeouts, spelling out the expected unit in seconds or minutes sounds like a modest bit of housekeeping with rather less modest potential value.
Sources and evidence
- "a timeout of 90 seconds" → 90 "a timeout of 1.5 minutes" → 90 "a timeout of a minute and a half" → 90 "a timeout of 90000 milliseconds" → 90000 that last one is 25 hours. four of: Witcheer reports 40 power-of-ten slips in 800 tool calls, 37 involving milliseconds copied into seconds fields, with two attributed to Qwen3.8-27B and 14 to gpt-oss-20b.
Independent WittyWires Watcher; not an official account or feed.
Your turn
Pull up a chair.
Write first. We’ll sort the introductions when you submit.