Four local LLMs for modest hardware, and what each is good at

Four local language models offer different strengths on ordinary or low-powered hardware, according to MakeUseOf writer Jayric Maning, who says he tested models on an 8 GB RTX 3060. The useful takeaway is not that one small model rules them all, but that the right choice depends on whether you want coding, creative writing or an always-on assistant.

Watch Desk analysis

What happened

Maning recommends four models, all of which he says are available through LM Studio and Ollama. Qwen3.8-9B-Distill is his general-purpose pick: he says it improves on its Qwen3.5-9B base in general knowledge and structured instruction-following, while losing about 1.5 percentage points on GSM8K maths accuracy. He also says it supports function calling and vision with an additional file.

For coding, Maning favours Ornith-1.5-9B, which uses a self-improvement loop to plan, test and revise solutions. He says it handled small projects, but warns that the process can use more tokens and context than a regular chat model. For creative writing, his choice is Qwythos-9B-v2; in his tests it approached 30 tokens per second with a roughly 68K context window, after he lowered KV-cache precision to Q51.

The fourth option, Llama-3.2-3B-Instruct, is aimed at low-power or CPU-only machines. Maning says it works for tasks such as summarising, following instructions and querying a local knowledge base, though it does not match the larger models on reasoning and general knowledge. He recommends trying Q4KM builds where available, and testing settings on the hardware you actually have. Read Maning’s full guide.

Why it matters

Local AI is often presented as a hardware arms race. These examples show a more useful question: what can a particular machine handle, and what do you need the model to do? A 3B model may be the sensible choice for a home server that needs to stay on all day, while a coding model’s extra planning may be worth the heavier context use for small projects.

The comparisons and performance figures are Maning’s own tests, not a standardised head-to-head benchmark. Still, the article gives readers concrete models and starting settings to try, rather than the usual advice to buy a more expensive graphics card and hope for the best.

Our read

Start with the job, then pick the model: coding, creative writing and a modest always-on assistant are three different workloads. Treat Maning’s settings as a starting point, not a magic recipe; local performance depends on the model build and your hardware.

What to watch

• Whether the named models remain available in LM Studio and Ollama.
• How the models perform on readers’ own hardware and everyday tasks.
• Whether newer small models improve capability without demanding more memory or patience.

Discussion spark: If you had to choose one local model for your own machine, would you prioritise coding ability, general usefulness or low power use?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.