A workshop led by Microsoft Foundry developer advocate Pamela Fox put GPT-5.5, Kimi K2.6, DeepSeek V4 Flash and Mistral Large 3 through ordinary tasks, and the results offered no universal winner. The practical lesson: test the model on the work your product actually needs, including whether it can admit when its sources do not contain an answer.
Watch Desk analysis
What happened
Fox compared the four models on tasks including counting letters, answering from documents, interpreting images, calling functions and planning a trip with tools. In one test, GPT-5.5 and Kimi correctly counted 13 instances of “e” in a sentence; DeepSeek answered 6 and Mistral 8. A clearer prompt fixed Mistral’s count, suggesting that some failures can be addressed through instructions rather than a model swap.
Other differences were harder to patch. Given documents about carpenter bees but no information on honeybees, Mistral answered the missing part from memory. Fox found that asking it to label claims with sources helped only partly: it flagged the honeybee details as unsourced but still supplied them. BigGo Finance’s account of Fox’s workshop also describes models differing in image support, tool calls and how thoroughly they followed a multi-step travel-planning task.
Why it matters
A model that scores well on a general benchmark may still be the wrong choice for a document assistant, image reader or agent that needs to finish a sequence of tasks. Fox’s examples give builders concrete checks: does the model ground answers in supplied documents, follow a function’s arguments, use tools efficiently and complete the requested steps?
She also argues that developers should test the model and the surrounding tool-calling setup together. In her travel-planning example, models made different numbers of calls, failed to parallelise or stopped before completing the plan. The model is only one part of the system; the harness gets a vote too, whether or not the product team remembers to invite it.
Our read
Fox’s strongest point is less “pick this model” than “find out what breaks before your users do”. Start with representative tasks and keep the failures as evaluations you can rerun when changing models or prompts. Her workshop examples are a useful practitioner account, not a universal ranking: the outcomes depend on these tasks, prompts and setups.
What to watch
- Whether builders publish evaluations grounded in real user tasks, rather than relying on broad benchmark scores.
- Whether document systems reliably say when the supplied material does not contain an answer.
- How model choice and tool-calling setups affect task completion, latency and the number of calls.
Discussion spark: When choosing a model for a real product, which failure should be disqualifying: unsupported answers, missed task steps or inefficient tool use?
Sources and evidence
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.