Ollama and llama.cpp now support models that return a choice and associated probabilities instead of generating a written answer. That gives developers a way to run structured AI classification locally, with potential cost and privacy benefits, while leaving an important question: do the probabilities deserve to be trusted?
Watch Desk analysis
What happened
Brief IA reports that Ollama and llama.cpp support the /v1/systemone endpoint, which is compatible with TypeSafe. Its account names Nimble and Tev1 for Ollama, and Laya, Kev, Julia-1 and OpenJev for llama.cpp. The report gives an example request that asks a model to route a double-billing complaint to one of several support teams. That is an illustration, not a test on real customer messages.
The endpoint accepts a question and a set of allowed responses. Instead of producing prose for an application to interpret, a model selects among those options and associates probabilities with them. Brief IA says a recent llama.cpp version can start a Laya-GGUF server with llama serve -hf ggml-org/Laya-GGUF.
Why it matters
For repetitive tasks such as routing support requests or flagging spam, a structured choice may be easier for software to use than a paragraph that needs parsing. Running the model locally could also avoid a paid API call and keep the input on the machine running it. Those are practical possibilities, not demonstrated savings or proof of accuracy.
The probabilities need particular care. A confident-looking score is not automatically a well-calibrated measure of correctness, and the source advises testing on real messages before automating customer support. The machine can give you a neat number; neatness is not the same thing as evidence.
Our read
This is a useful expansion of local inference beyond chat: models can be put to work making constrained choices, not just composing answers. Developers can try the supported endpoint and models for bounded tasks, but should validate both decisions and probability calibration before letting the results trigger consequential actions.
What to watch
- Whether Ollama and llama.cpp document setup and compatibility requirements for the endpoint.
- How the named models perform on real classification tasks, beyond illustrative examples.
- Whether developers publish calibration results and safeguards for decisions that need human review.
Discussion spark: For routine support triage, would you trust a locally run model to route requests automatically after testing, or keep a person in the loop for every decision?
Sources and evidence
- Ollama and llama.cpp Welcome Decision Models – Brief IA (20 November 2026, 13:31 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.