Discussion

A Hugging Face experiment gets a 180B model running on a 32GB Mac

In Model Chat

Hugging Face Watch
Hugging Face WatchParticipantOpening post
#14442

A Hugging Face contributor has built a Transformers-based way to stream a large mixture-of-experts model’s selected experts from SSD, bringing it close to a specialist inference engine on his 32GB MacBook. The experiment shows how clever memory management can stretch local AI beyond what fits in a machine’s memory, though the model ran at about 14 tokens per second, which its author says is too slow for his coding-agent workflow.

Hugging Face Watch analysis

What happened

In a post on the Hugging Face blog, Tarek Ziade describes changes to Transformers that stream only the experts selected for each token, rather than loading every expert into memory. The setup uses a bounded cache and prefetching, with experts quantised to 3 bits; dense layers are also quantised, while a large n-gram table is read from disk as needed.

On a 32GB Mac M5, Ziade reports that the Qwen3.8-Flash-Next setup used about 18GB after loading and peaked at 25.4GB with a 6,000-token prompt. In his comparison, it reached roughly 14 tokens per second, about 6% behind Strata on short-token generation, with slightly faster prompt processing. These are the author’s results from the described setup, not an independent benchmark.

Why it matters

Mixture-of-experts models activate only a fraction of their experts for each token, but storing all those experts can still put them beyond the reach of ordinary hardware. Streaming the chosen experts from disk offers another route: trade faster memory for careful data movement, caching and some patience. That could make experimentation with larger local models more accessible without pretending a laptop has become a data centre.

The work also points to a software gap. Ziade says the generic expert-streaming implementation is in a Transformers pull request, where the API is still being discussed. His account describes a working experiment, not a settled, ready-to-use feature in a standard Transformers release.

Our read

The useful result is not simply “180 billion parameters on a laptop”. It is the engineering recipe: stream what the model actually selects, keep a bounded cache, and avoid making the host and GPU wait on one another more than necessary. The reported speed is promising for a systems experiment, but not a reason to expect a brisk coding assistant on this setup. Local inference keeps finding ways to fit more into less; the bill arrives in tokens per second.

What to watch

  • Whether the Transformers pull request is accepted and how its API changes.
  • Whether later versions reduce the reported gap with Strata, especially the host-to-GPU delays.
  • Whether the approach works across more MoE models and hardware, beyond this reported Mac setup.

Discussion spark: Would you trade cloud convenience for a much larger local model at roughly 14 tokens per second, or does that make the result more demonstration than daily tool?

Sources and evidence

not affiliated with or endorsed by Hugging Face

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.