AMD says its open-source TokenSpeed inference engine served Kimi K3 at 216 output tokens per second per user on eight Instinct MI355X GPUs, in a single-user test. The results point to the engineering behind fast, interactive AI agents, while the test conditions make clear this is not a universal speed claim.
AMD Watch analysis
What happened
In a technical article published on 9 October, AMD describes TokenSpeed’s multi-vendor inference software and its work serving Kimi K3 on MI355X accelerators. The benchmark replayed coding-agent sessions, starting with a roughly 50,000-token prompt and adding tool results over 10 to 15 turns. AMD reports a 90% cache-hit rate and 216 output tokens per second per user at concurrency one, or 4.62 milliseconds per output token.
AMD says performance was on par with or slightly faster than ATOM at concurrency levels two, four and eight. At 16 concurrent users, TokenSpeed delivered 39 output tokens per second per agent and ATOM led. AMD attributes some of the comparison to ATOM’s different scheduling and more aggressive quantisation. Read AMD’s TokenSpeed technical article.
Why it matters
Agent performance is not just a question of how quickly a model emits its next token. Long conversations, reused context and repeated tool calls all affect how responsive an agent feels. AMD’s test focuses on that kind of repeated interaction, rather than a short prompt followed by a single answer.
The article also shows how much performance work sits below the model: AMD describes specialised kernels, cache handling and scheduling as parts of the effort. Its reported numbers apply to the named hardware, model and workload, not every deployment.
Our read
This is a useful, unusually detailed look at inference engineering, and the concurrency results are more revealing than a lone headline multiplier. AMD’s figures make a case for MI355X and TokenSpeed in this specific workload; the comparison also shows there is no single winner at every scale. For anyone evaluating inference systems, the practical lesson is to test the traffic pattern and latency trade-offs that actually matter to their users.
What to watch
- Whether independent tests reproduce the results on the same model and hardware.
- How TokenSpeed performs on other models and longer or less cache-friendly conversations.
- Whether the scheduling trade-offs at higher concurrency change the user-facing experience.
Discussion spark: When choosing an inference stack for AI agents, would you prioritise snappy single-user responses or higher throughput as more agents run at once?
Sources and evidence
- Advancing Kimi K3 inference on AMD Instinct MI355X GPUs with TokenSpeed – AMD (9 October 2026, 17:43 UTC)
Independent WittyWires tracker for public updates about AMD. Not affiliated with or endorsed by AMD; this is not an official account.