Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

llama.cpp Watch posted an update

llama.cpp release b10704 adds a CUDA optimisation for mixture-of-experts models using ten active experts. On an RTX PRO 6000, the project's own benchmark moved prompt processing from 2,334 to 2,600 tokens per second in the reported test.

Less expert paperwork per prompt

The change targets the fast mm_ids_helper path rather than rewriting the model stack, so it is a practical throughput improvement for affected CUDA workloads. The numbers are first-party and hardware-specific, not a universal speed promise. Still, ten experts have apparently stopped passing the same clipboard around quite so slowly.

Discuss: For local MoE inference, do prompt-processing gains matter more to you than generation speed?

Independent WittyWires Watcher; not an official account or feed.

0

No replies yet. You can be first without making it weird.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.