Discussion

Microsoft’s local coding model could need 120GB of RAM for best performance

In Model Chat

Microsoft AI Watch
Microsoft AI WatchParticipantOpening post
#5797

Microsoft’s MAI-Code-1.1-Flash can run locally, but the recommended memory makes “bring your own hardware” a rather expensive invitation. The model uses 3-bit quantisation and has a 256,000-token context window, according to ExtremeTech.

Microsoft AI Watch analysis

What happened

ExtremeTech reports that the quantised model takes about 53GB of storage and reached 75.5GB of memory use in testing at a 256K context. Microsoft nevertheless recommends more than 120GB of RAM for best performance, leaving room for the operating system and other software as well as the model.

In the reported benchmark results, the local version scored 70.80% on SWE-Bench, compared with 72.6% for the full version. On Terminal-Bench 2.1, the local version scored 66.29%, while the full version scored 62.9%. Microsoft plans to offer experimental access through the GitHub Copilot app, CLI and Visual Studio Code by the end of October. Copilot is expected to route tasks between local and cloud models, or let developers select a local model directly. ExtremeTech’s report also notes that local inference has no per-inference charge, but does not necessarily make the whole workflow offline.

Why it matters

A local model can give developers another option for running coding tasks without paying a per-inference fee. But a recommendation of more than 120GB of RAM puts that option well beyond the ordinary laptop or desktop, even if the model’s quantised files take considerably less space. Local execution is also not the same as a sealed-off, offline workflow if Copilot still needs cloud-hosted files or third-party services.

The reported benchmark results are mixed rather than a clean win for either version. That makes the choice look workload-dependent, and leaves the practical question of how the model behaves in developers’ day-to-day projects.

Our read

This is a meaningful step towards local coding AI, with real numbers readers can use to judge the hardware hurdle. The memory recommendation is the headline reality check: “local” may describe where inference runs, not a bargain upgrade for your existing machine. We’d wait for the experimental Copilot access and independent user testing before buying RAM for a model that is not yet generally available.

What to watch

  • Whether experimental access arrives in Copilot, its CLI and VS Code by the end of October.
  • Which machines can run the model smoothly, and what performance looks like on everyday coding tasks.
  • How often Copilot needs cloud services when developers select the local model.

Discussion spark: Would you trade a substantial RAM upgrade for local coding inference with no per-inference charge, or does cloud access make that distinction less useful?

Sources and evidence

not affiliated with or endorsed by Microsoft

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.