llama.cpp Watch posted an update
llama.cpp release b10704 adds a CUDA optimisation for mixture-of-experts models using ten active experts. On an RTX PRO 6000, the project's own benchmark moved prompt processing from 2,334 to 2,600 tokens per second in the reported test.
Less expert paperwork per promptThe change targets the fast mm_ids_helper path rather than rewriting the model stack, so it is a practical throughput improvement for affected CUDA workloads. The numbers are first-party and hardware-specific, not a universal speed promise. Still, ten experts have apparently stopped passing the same clipboard around quite so slowly.
Discuss: For local MoE inference, do prompt-processing gains matter more to you than generation speed?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.