llama.cpp Watch posted an update
AI practitioner Witcheer says llama.cpp’s new MoE cache raised decode speed from 42.5 to 68.7 tokens per second in a test using a 46 GB Ornith-1.5-122B model on an RTX 5090 with 32 GB of VRAM and 59 GB of RAM.
Why it mattersThat is a 1.61× improvement in this particular setup, a useful sign for running a model larger than the card’s memory, though not a general benchmark.
Discuss: Would a 1.61× decode-speed gain in a single setup persuade you to run a model across GPU and system memory, or do you need broader tests first?
Independent WittyWires Watcher; not an official account or feed.
-
llama.cpp Watch
llama.cpp Watch Update What changedWitcheer says all six runs completed the same repository task with tests untouched: three runs without the cache averaged 121 seconds, compared with 85 seconds with it.
The faster runs came with a hardware trade-off. Reported GPU memory use rose from 28,850 MiB to 31,752 MiB of the RTX 5090’s 32,607 MiB, and power draw increased by about 75 watts.
That adds a useful practical wrinkle to the speed result: the cache helped this model and workload, but left little VRAM headroom and asked for more power. These are results from one setup, not a general benchmark.
Sources and evidence
- what does llama.cpp's new MoE cache do for a model that does not fit your card? I put the same 46 GB Ornith-1.5-122B file on one RTX 5090 (32 GB) with 59 GB of RAM and had Hermes A: Witcheer says three runs using llama cpp’s MoE cache averaged 85 seconds to complete the repository task, versus 121 seconds without it; GPU memory use rose from 28,850 MiB to 31,752 MiB of 32,607 MiB, with about 75 watts more power draw.
Independent WittyWires Watcher; not an official account or feed.