Baseten says a new weight-sync design cuts model update pauses to under a second

Baseten says it has cut the pause needed to update a roughly 770 GB GLM-5.3 model from six seconds to 0.8 seconds, while reducing total sync time from 36 seconds to 15. The new design uses the model weights already on the GPUs instead of keeping a second full checkpoint in host memory, a change with practical consequences for both update speed and memory available to serve AI workloads.

Baseten Watch analysis

What happened

In a technical post published on 30 October, Baseten describes a redesigned system for applying sparse weight updates to models already running on its inference infrastructure. The company says the old approach kept a full copy of the parent checkpoint in CPU memory; the new one uses GPU memory as the source of the existing weights and reads only positions changed by an update.

Baseten reports that a Kimi K3 rollout on two GB300 nodes can now apply an 8.4 GB delta and serve the new weights in 75 seconds, with a 4.1-second pause. It also says freeing host memory for KV-cache offloading raised cache hits from 2% to 92% and output throughput per GPU by 3.2 times at concurrency 48 in its AgentX benchmark. Those are Baseten’s own measurements, not an independent comparison.

Why it matters

Updating a live model is not just a matter of moving new weights into place. The old system’s CPU copy consumed substantial memory and took about 45 seconds to promote after an update, according to Baseten. Removing that copy can shorten the handover and leave more host memory available for other parts of inference, including cached context that can save work on repeated prompts.

That makes this more than a neat reduction in a timing chart. For operators serving large models, shorter pauses and better use of memory can affect how quickly they can roll out a policy and how much useful work their hardware handles. The gains will depend on workload and setup, but the published figures give teams concrete claims to test against their own systems.

Our read

This is the kind of infrastructure improvement that matters because it connects a low-level systems change to operator-facing results: less waiting during updates, less host memory tied up in duplicate weights, and a reported throughput gain when that freed memory supports KV caching. Baseten’s results are promising; the sensible next step is to see whether independent workloads reproduce them. The bottleneck, as ever, has a habit of moving once you fix it.

What to watch

• Whether other workloads show similar reductions in update time and serving pauses.
• How the results change with different model sizes, delta sizes and hardware.
• Whether the reported KV-cache and throughput gains hold up beyond Baseten’s AgentX benchmark.

Discussion spark: For live model updates, which should operators value most: shorter serving pauses, lower memory use or higher throughput under load?

not affiliated with or endorsed by Baseten