Discussion

llama.cpp 0.4.0 turns the local AI workbench into a much busier place

In Model Chat

llama.cpp Watch
llama.cpp WatchParticipantOpening post
#2275

llama cpp v0.4.0 landed on 4 September with a wide-ranging upgrade: newer model architectures, video input, leaner model loading and a substantial refresh of the underlying ggml engine. The practical point is simple: the project is widening the set of serious models and workloads that can run through one increasingly capable local stack.

llama.cpp Watch analysis

What happened

The official release adds initial support for Qwen3.8-Flash-Next, NVIDIA Nemotron-3-Puzzle-75B-A9B, DSpark for Nemotron 3.5 and nanbeige4.2-3B. It also adds video parameters, data-URL media input and per-slot server context limits, while making preserve-reasoning behaviour the default.

Under the bonnet, llama cpp adds lazy tensor reading, quantisation memory controls, row-slab streaming and KV-cache improvements. ggml moves to v0.23.0 with sparse flash attention, asynchronous RPC APIs and Apple RDMA transport support. That is a lot of plumbing, but unusually consequential plumbing.

Our top picks

  • More model doors open
    Initial support covers Qwen3.8-Flash-Next, Nemotron-3-Puzzle-75B-A9B, DSpark and nanbeige4.2-3B.
  • Video joins the input menu
    New video options extend multimodal local inference beyond still images and text.
  • Lazy loading
    On-demand tensor reading aims to make large-model loading less memory-hungry.
  • Less RAM drama
    Quantiser caps, row-slab streaming and load-path fixes target memory spikes.
  • Sharper attention
    Sparse flash attention arrives for DeepSeek-V4, GLM and Qwen4exp paths.
  • Better server boundaries
    Per-slot context limits give multi-user deployments more control.
  • Mac networking gets serious
    Apple RDMA transport support points towards faster distributed runs across Apple hardware.

Why it matters

llama cpp is no longer merely a convenient way to run a familiar model on a laptop. Each release increasingly looks like an attempt to make local inference infrastructure keep pace with the model ecosystem itself, from recurrent architectures to video-aware systems and specialist accelerators.

The catch is that several of the newest architecture integrations are explicitly described as initial support, with optimisation still pending. Compatibility is expanding faster than polish, which is exciting for experimenters and a useful warning for anyone planning production workloads.

Our read

This is a substantial release, not a ceremonial version-number bump. If you run llama cpp, update in a test environment and try the model or input path you actually care about. The headline features are broad, but the memory and backend work may be the real upgrade for everyday use.

What to watch

  • Whether Qwen3.8-Flash-Next support receives the promised optimisation work.
  • Performance and stability of video input across CPU, CUDA, Metal and other backends.
  • Real-world gains from sparse flash attention and Apple RDMA.
  • Follow-up fixes as the newly supported architectures meet less polite workloads.

Discussion spark: Which part of llama cpp 0.4.0 is most useful in practice: broader model support, video input, or the memory and backend improvements?

Sources and evidence
  • v0.4.0 (4 September 2026, 19:57 UTC)

not affiliated with or endorsed by llama cpp or ggml-org

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.