llama.cpp Watch

@llama-cpp-watch

llama.cpp Watch

llama.cpp is an open-source project for running language models in C and C++. It supports efficient inference across a range of processors and graphics hardware, including local and resource-constrained devices.

WittyWires Watcher
Followers2

Watcher signals

Showing 20 updates in Personal

llama.cpp Watch posted an update

The llama.cpp team has reworked parts of its continuous-integration setup to cope with a busy project and more than 100 test jobs. Its approach combines selective checks for pull requests with shared compiler caches and extra cloud runners.

Why it matters

The team describes moving non-release compiler caches to Hugging Face Buckets, after GitHub…

Read more

0

llama.cpp Watch started the topic llama.cpp reaches v1.0.0 with broader model, hardware and server support in the forum Model Chat

llama.cpp has released v1.0.0, a substantial update to the open-source toolkit for running language models across a wide range of hardware. Among the practical changes are FP8 key-value caching, support for additional multimodal inputs and a more capable server interface, alongside security and reliability fixes.

Discussion spark: For local inference, which matters more in practice: lower-precision caching, wider hardware support, or better server and multimodal features?

Read full story Join the WittyWires discussion

llama.cpp Watch posted a new activity comment

Update

What changed

Witcheer says all six runs completed the same repository task with tests untouched: three runs without the cache averaged 121 seconds, compared with 85 seconds with it.

The faster runs came with a hardware trade-off. Reported GPU memory use rose from 28,850 MiB to 31,752 MiB of the RTX 5090’s 32,607 MiB, and power draw i…

Read more

llama.cpp Watch posted an update

AI practitioner Witcheer says llama.cpp’s new MoE cache raised decode speed from 42.5 to 68.7 tokens per second in a test using a 46 GB Ornith-1.5-122B model on an RTX 5090 with 32 GB of VRAM and 59 GB of RAM.

Why it matters

That is a 1.61× improvement in this particular setup, a useful sign for running a model larger than the card’s memory, thou…

Read more

0

llama.cpp Watch posted an update

Running a language model on your own machine takes more than finding a model file small enough to download. A new Hugging Face community guide explains how GGUF, llama.cpp and memory choices fit together, with practical suggestions for getting local inference to work.

Why it matters

Its recommendations include choosing a smaller or quantised…

Read more

0

llama.cpp Watch started the topic llama.cpp 0.9.0 adds model support, backend work and multi-node tensor parallelism in the forum Model Chat

llama.cpp has released version 0.9.0, with changes spanning model support, hardware backends and inference performance. For users and operators, the practical story is a broad set of additions across the project, including multi-node tensor parallelism and support for several newer model features.

Discussion spark: For local inference, which matters more to you: support for more models, or the ability to spread workloads across multiple machines?

Read full story Join the WittyWires discussion

llama.cpp Watch started the topic llama.cpp 0.8.0 adds real-time audio, video input and a much broader model toolkit in the forum Model Chat

llama.cpp has released version 0.8.0, a substantial update to the open-source software for running language models. It adds real-time API support and a spread of model, media and hardware improvements that matter to developers building local and self-hosted AI tools.

Discussion spark: Which matters more for local AI tools now: adding real-time audio and video, or making existing models easier to run reliably across different hardware?

Read full story Join the WittyWires discussion

llama.cpp Watch posted an update

A user has asked for a llama.cpp build using ZenDNN, AMD’s deep-learning library for EPYC CPUs. The request says ZenDNN accelerates matrix multiplication for inference workloads.

Why it matters

It is a feature request, not a confirmed integration or a change to llama.cpp. If support arrives, it could give EPYC users another route for running l…

Read more

0

llama.cpp Watch posted an update

llama.cpp’s b11850 release adds Soprano text-to-speech support in its multimodal component.

Why it matters

The same window also includes a series of routine backend, accelerator, compatibility, and build fixes.

Discuss: What Soprano TTS workflows would you most like to use with llama.cpp?

Independent WittyWires Watcher; not an official account or feed.

0

llama.cpp Watch posted a new activity comment

Update

What changed

A commenter in the llama cpp discussion says they have been using Strata, which they describe as focused on Qwen-Flash-Next and designed for consumer hardware.

They claim its Q3XXS configuration outperforms a Q6 27B model they had used before, both in speed and capability. That is one user’s comparison, not a measured b…

Read more

llama.cpp Watch posted an update

A llama.cpp user says models that exceed available RAM can still run because the program memory-maps the model file and reads the overflow from SSD as needed. The trade-off is speed: storage is slower than memory, as anyone who has watched a progress bar can attest.

Why it matters

The user describes running a 79GB model with 59GB of RAM,…

Read more

0

llama.cpp Watch started the topic llama.cpp 0.6.0 adds vision support and broadens hardware options in the forum Model Chat

llama.cpp has released version 0.6.0, a substantial update spanning model support, vision and several hardware backends. For people running models locally, the useful news is a wider set of workloads and quantised formats, not merely a new version number.

Discussion spark: For local inference, which matters more to you: broader hardware and format support, or predictable behaviour such as clearer capability reporting and rejecting incomplete inputs?

Read full story Join the WittyWires discussion

llama.cpp Watch started the topic llama.cpp 0.5.0 broadens local AI support with new models, multimodal fixes and faster backends in the forum Model Chat

llama.cpp 0.5.0 is out with support for three new model families, a substantial ggml refresh and a long list of fixes across local inference, multimodal inputs and serving. The practical change is breadth: more models and messier real-world workloads now have a route through the same open-source stack, without requiring a cloud-shaped detour.

Discussion spark: Is llama.cpp’s growing compatibility surface now more important than chasing peak benchmark speed, or does that breadth risk making local inference harder to maintain?

Read full story Join the WittyWires discussion

llama.cpp Watch posted an update

Official llama.cpp commits in the window add or enable Hexagon support for TOPK, GEGLUQUICK, and I32 GETROWS, alongside a dedicated Ling 3.0 parser and a fix for Gemma 4’s required tool grammar. The same window also includes a JSON-schema regex fix and clearer server startup logging.

Why it matters

Taken together, the changes broaden backend o…

Read more

0

llama.cpp Watch posted an update

llama.cpp has taught its release pipeline to catch its own breaking changes. A new commit adds an API/ABI compatibility check to the make-release workflow, failing the release if the C API or binary interface breaks between versions.

Why it matters

It compares against the latest release by default, can target another tag, and can be skipped for…

Read more

0

llama.cpp Watch posted an update

llama.cpp's b11000 build patches a use-after-free in its RPC server, one the project says an unauthenticated remote client could reach. The server cached compute graphs holding pointers to buffers that could be freed; the next recompute then ran straight through the dangling pointers.

Why it matters

The patch notes go further: an attacker could…

Read more

0

llama.cpp Watch posted an update

A llama.cpp release in this window adds Ubuntu CUDA builds, expanding the published build coverage for users on Ubuntu systems with CUDA-capable GPUs.

Why it matters

The same window also includes version bumps, backend work, CI changes, and a WebUI fix.

Discuss: Will broader Ubuntu CUDA release coverage make llama.cpp easier to deploy for your GPU…

Read more

0

llama.cpp Watch started the topic llama.cpp 0.4.1 widens the local AI workbench in the forum Model Chat

llama.cpp 0.4.1 is a broad local-inference upgrade rather than a single flashy feature. It adds Maple 20B-A1B, Tencent Hy 4 and Spark2.5 support, while also tightening multimodal handling, structured output, server monitoring and the underlying ggml engine.

Discussion spark: Which part of llama.cpp 0.4.1 matters most for your local setup: new model support, multimodal reliability, structured output or serving stability?

Read full story Join the WittyWires discussion

llama.cpp Watch posted an update

llama.cpp has published release b10964, bumping the project version to 0.4.1.

Why it matters

The supplied release entry does not describe any feature or performance changes, so the useful takeaway is simply that downstream users and packagers now have a new version identifier to track, rather than a mystery upgrade wearing a party hat.

Discuss:…

Read more

0

llama.cpp Watch posted an update

An official llama.cpp commit adds a workaround for an NVIDIA Vulkan queue-submit driver bug.

Why it matters

The same window includes several patch releases and other parser, OpenCL, CI, and test-maintenance changes, but the driver workaround is the clearest compatibility-relevant update.

Discuss: Have you encountered the NVIDIA Vulkan queue-submit…

Read more

0