A team behind vLLM has released open-source TPU kernels that it says served Moonshot AI’s Kimi K3 at 709 tokens per second on 16 Google TPU v7 chips, versus 452 on 16 NVIDIA GB200 GPUs. The comparison suggests software optimisation can shift the AI hardware race, though it is a low-batch, single-user test, not a verdict on every workload.
Watch Desk analysis
What happened
Inferact’s megakernel combines Kimi K3’s 92 mixture-of-experts layers into a single program, overlapping the movement of model weights with computation. Its code is available in the inferact/tpu-megakernels repository.
The figures were reported by OfficeChai, drawing on SemiAnalysis and Inferact’s data. The 709-token result uses speculative decoding, which lets a smaller draft model propose tokens for Kimi K3 to verify. At a shorter acceptance length, Inferact reports 350 tokens per second against 229 for the GB200 baseline. It also reports TPU wins on Qwen 3.8 27B, while a separate Cerebras comparison is approximate.
Why it matters
Fast responses matter when coding tools and AI agents make repeated model calls: shaving latency from each call can make the whole interaction feel less like waiting for a particularly thoughtful kettle. The result also gives Google’s TPU ecosystem a concrete software demonstration beyond raw chip specifications.
But the comparison has boundaries. The GB200 figure uses vLLM’s published recipe, not a bespoke tuned kernel or a full NVL72 rack, and the test measures single-user speed rather than serving many users at once. Those details matter before translating a striking chart into a purchasing decision.
Our read
This is a meaningful demonstration of what careful kernel engineering can do, and a welcome sign that AI serving is not settled by buying the biggest accelerator and hoping for the best. The useful takeaway is not “TPUs win”; it is that the right software can make a hardware comparison look very different. Operators should inspect the code and reproduce the test against workloads that resemble their own.
What to watch
- Whether independent operators reproduce the results on the released kernels.
- How the TPU advantage changes at higher concurrency and across different model workloads.
- Whether Inferact’s planned work on agentic use and KV-cache movement extends the gains beyond this setup.
Discussion spark: For interactive AI workloads, should buyers prioritise the fastest result on a carefully optimised single-user test, or performance at the concurrency their service actually needs?
Sources and evidence
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.