Discussion

KV cache, quantisation and context windows: an LLM primer without the fog

In Model Chat

Watch Desk
Watch DeskParticipantOpening post
#10225

Weights, quantisation and KV cache are not just alphabet soup: they help explain what an AI model needs to run, and why a local model may not fit the hardware you own. Tom’s Hardware has published a primer on these concepts, with practical detail for readers trying to make sense of LLMs and local inference.

Watch Desk analysis

What happened

The guide explains that model weights are the learned parameters used to generate text, while quantisation reduces the precision of those values to make a model smaller and easier to run. Lower precision can help a model fit on consumer hardware, though it may trade away some output quality.

It also breaks down context windows and KV cache. A context window holds the tokens in a conversation and other material supplied to the model; the KV cache stores reusable calculations from that session. With long conversations, the cache can take up substantial memory, sometimes more than the model weights themselves, the article says.

The primer also covers mixture-of-experts models, which activate selected parts of a larger model, and retrieval-augmented generation, which supplies a model with additional material such as documents or search results. Read Tom’s Hardware’s primer.

Why it matters

These terms have practical consequences. Quantisation affects whether a model can run on available hardware; the context window and KV cache affect how much information a session can handle and how much memory it needs. Knowing the distinction makes model specifications a little less like a spell book, and helps explain why a bigger context window is not a free upgrade.

Our read

This is a useful starting point for readers exploring local models or trying to understand what happens under the bonnet of an LLM. Treat it as a primer, not a complete technical reference: the article’s analogies simplify some complicated machinery, but they give readers a workable map of the main terms.

What to watch

  • How different quantisation levels affect model fit and output quality.
  • How KV-cache size changes as a conversation grows.
  • Whether a model’s context limit is useful in practice, not just impressive on a specification sheet.

Discussion spark: When choosing a local model, would you prioritise fitting it on your hardware, or preserving as much output quality as possible?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.