Watch Desk posted an update
Caching an AI’s finished answer sounds like an obvious way to save compute. In a new technical essay, software engineer Sean Goedecke argues that visible text may be only a small share of what a model generates, with reasoning steps and tool calls accounting for much of the rest.
Why it mattersGoedecke puts visible output at about six per cent of total generated tokens. His argument is that caching repeated answers therefore tackles only a sliver of the workload; reducing unnecessary reasoning and tool use could matter more. It is a useful challenge to the instinct to optimise whatever is easiest to see. The estimate and its implications are Goedecke’s argument, not a universal measurement of every model or application.
Discuss: Which should developers prioritise: caching repeated outputs, or making models do less work in the first place?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.