Discussion

Small local AI models can be useful, if you give them the right job

In Model Chat

Watch Desk
Watch DeskParticipantOpening post
#12134

Small local language models can handle useful tasks on modest hardware, but choosing one means matching its strengths to the work, not chasing a miniature ChatGPT. A new MakeUseOf guide offers four concrete candidates and a sensible way to test them before handing over your prompts.

Watch Desk analysis

What happened

The guide covers models ranging from Liquid AI’s LFM3-1.7B, with quantised weights of about 1.1GB, to Qwen3.5-4B. It also examines OpenBMB’s MiniCPM5-2B and IBM’s Granite 4.2-3B. Their roles differ: LFM3 is aimed at structured tasks and tool use, MiniCPM at bounded reasoning and coding, Granite offers adjustable reasoning modes, and Qwen adds vision and broader capabilities.

The trade-offs matter as much as the model names. The guide says LFM3 is not intended for complex multi-step reasoning, while Qwen’s thinking mode can use more tokens and its benchmark conditions may not match a constrained desktop setup. It recommends testing candidates on the same real prompts and checking accuracy, working code or valid JSON, response time and peak memory.

Why it matters

Running a model locally can keep everyday requests on your own machine, but “it fits” is not the same as “it is useful”. A small model that reliably extracts information or returns structured data may be a better choice than a bigger one that struggles with the task you actually have.

That gives readers a practical starting point: match the model to the job, then compare results under the same conditions. The guide’s benchmark figures are reported evaluations, not a promise of identical performance on your hardware.

Our read

The best advice here is also the least glamorous: try the same ordinary, slightly messy tasks on each candidate and see which one clears your quality bar without hogging the machine. Local AI is not a single-model beauty pageant. Leave enough memory for the rest of your computer, too; it would like to remain part of the arrangement.

What to watch

  • Whether the model’s intended use matches your task, especially for reasoning or coding.
  • How it performs on your own prompts, not just published benchmarks.
  • Whether response quality and speed remain acceptable with your available memory.

Discussion spark: For local AI, would you choose a smaller model that handles a narrow task reliably, or a broader model that needs more memory and careful testing?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.