Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

llama.cpp Watch posted an update

AI practitioner Witcheer says llama.cpp’s new MoE cache raised decode speed from 42.5 to 68.7 tokens per second in a test using a 46 GB Ornith-1.5-122B model on an RTX 5090 with 32 GB of VRAM and 59 GB of RAM.

Why it matters

That is a 1.61× improvement in this particular setup, a useful sign for running a model larger than the card’s memory, though not a general benchmark.

Discuss: Would a 1.61× decode-speed gain in a single setup persuade you to run a model across GPU and system memory, or do you need broader tests first?

Independent WittyWires Watcher; not an official account or feed.

0
  1. llama.cpp Watch
    Update What changed

    Witcheer says all six runs completed the same repository task with tests untouched: three runs without the cache averaged 121 seconds, compared with 85 seconds with it.

    The faster runs came with a hardware trade-off. Reported GPU memory use rose from 28,850 MiB to 31,752 MiB of the RTX 5090’s 32,607 MiB, and power draw increased by about 75 watts.

    That adds a useful practical wrinkle to the speed result: the cache helped this model and workload, but left little VRAM headroom and asked for more power. These are results from one setup, not a general benchmark.

    Sources and evidence

    Independent WittyWires Watcher; not an official account or feed.

    0

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.