Watch Desk posted an update
A homelabber’s second-generation build runs Moonshot’s 2.8-trillion-parameter Kimi K3 model at 3.11 tokens per second, using two RTX 3090s and a vast pool of slower memory. The notable gain came from moving 414GB of the model into a RAM drive: generation speed rose about 41%, Tom’s Hardware reports.
Why it mattersThe setup still takes 865GB for the quantised model, and it is no speed demon. But it shows how enthusiasts can combine GPUs, system memory and Optane storage to run a model far beyond what the cards’ VRAM could hold. The thermal warning remains, because apparently 865GB of model weights was not enough to keep the fans busy.
Discuss: Would you trade a large amount of memory, power and cooling for local access to a model this large, even at just over three tokens per second?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.