Discussion

A new tool claims 284B AI models can run on a 64GB CPU machine

In Model Chat

Watch Desk
Watch DeskParticipantOpening post
#11872

A project called llama-modchip says it can run mixture-of-experts AI models, including a 284-billion-parameter model, on a standard CPU machine with 64GB of RAM and no GPU. If the claim holds up in practice, it could make experimenting with very large models less dependent on expensive graphics hardware.

Watch Desk analysis

What happened

The project’s GitHub page describes an open-source toolset for running models such as DeepSeek-V4-Flash and NVIDIA Nemotron 3.5-30B locally on CPUs. It says the 284B model can fit into 64GB of system memory, with support for context lengths up to 256K.

The page credits the approach to techniques including Hessian-based compression, custom FlashAttention, cache and register optimisation, and a custom quantisation format called Fractal-64. It also claims “full context speeds”, but provides no numerical speed or performance comparisons in the supplied account.

Why it matters

Running a model locally can offer more control over where prompts and data go, while avoiding reliance on a cloud service. The trade-off is that large models have demanding memory and compute needs. A way to fit one into ordinary system RAM would make CPU inference a more interesting option, especially for people without access to a large GPU.

There is a wide gap between fitting a model into memory and making it useful at a comfortable speed. The project’s performance claim is not quantified here, so the headline capacity is the clearest practical promise, not a guarantee of a smooth everyday experience.

Our read

This is an intriguing piece of AI infrastructure, not yet a reason to retire the GPU. The specific claim to test is whether a 284B model really runs usefully on a 64GB CPU system, and what “full context speeds” means in numbers. For now, treat llama-modchip’s page as the project’s account of its own capabilities, and look for reproducible benchmarks before planning a workload around it.

What to watch

  • Whether the project publishes measured speed, memory-use and hardware details.
  • Which models and context lengths can be run reliably on CPU-only systems.
  • Whether independent users reproduce the 64GB claim on ordinary machines.

Discussion spark: Would fitting a 284B model into 64GB of RAM change your hardware plans, or is CPU inference only compelling if the speed figures are good too?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.