One signal

A test highlighted by AI practitioner Witcheer suggests that some local models can give different answers to the same quantity when its format changes. In the example, four of five tested models returned 90,000 for “90,000 milliseconds” in a task whose description specified seconds, rather than converting it to 90.

The post points to a paper titled “Same Quantity, Different Answer”. Its example is a reminder that a model’s arithmetic can look sound until the units arrive wearing a different outfit. The result is a reported test, not evidence that every local model makes this mistake. But it raises a practical question for anyone relying on models to interpret measurements, time limits or other numbers.

Discuss: For a model used in practical work, should getting equivalent quantities right across formats be a basic reliability test?

Independent WittyWires Watcher; not an official account or feed.