A small transformer’s accuracy on five modular-arithmetic tasks levelled off at about 99% by step 60,000. But as training continued to step 120,000, measurements of its internal pathways kept changing, offering a striking reminder that a flat score does not necessarily mean a model has stopped learning or reorganising.
Watch Desk analysis
What happened
In a community article on Hugging Face, Bill Brothers of Saccadic-AI describes a 1.4-million-parameter mixture-of-experts model trained on addition, subtraction, multiplication, division and exponentiation modulo 113. The run continued for 120,000 steps, taking about three hours on a Mac mini.
Brothers used IRIS, an instrumentation tool developed by Saccadic LLC, to record the model’s internals 120 times. He reports that effective-pathway readings continued to fall after accuracy flattened, ending between 96 and 242 across the five operations. The instrumentation added about 0.7% to wall-clock time; its 120-frame history occupied 80MB. The article describes this run and the tool’s measurements, rather than establishing that the same pattern holds across models or tasks. Read the Hugging Face article.
Why it matters
Accuracy is an appealingly simple scoreboard, but it can hide changes in how a model reaches its answers. If the measurements prove useful beyond this experiment, researchers could have another way to study what happens during training after the familiar performance curves go quiet.
That is an intriguing result, not yet a new general-purpose test for model understanding. The evidence here comes from one small model, a narrow set of arithmetic tasks and an instrumentation method developed by the article’s authors.
Our read
The useful idea is that a model’s internal activity may keep moving even when its headline score does not. IRIS makes that change visible in this run, at a reported modest time cost. The next step is not to crown a new theory of learning, but to see whether other teams can reproduce the measurements and connect them to meaningful differences in model behaviour.
What to watch
- Whether the authors publish the underlying paper and data for independent inspection.
- Whether other researchers reproduce the pathway measurements on different models and tasks.
- Whether these internal changes predict anything that accuracy scores alone miss.
Discussion spark: If a model’s accuracy stays flat while its internal pathways keep changing, should researchers keep training it, or treat the score plateau as the point to stop?
Sources and evidence
- Accuracy plateaued. The model kept changing. (27 October 2026, 06:02 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.