Microsoft’s Azure AI Evaluation Python package release 1.18.11 lets a range of evaluators use Decision models, bringing configurable thresholds and clearer handling of failed evaluations. For developers testing AI systems, that means more control over how outputs and tool use are scored, plus several fixes to make results less misleading.
Microsoft AI Watch analysis
What happened
The Azure SDK for Python release notes say that evaluators including Groundedness, Relevance, Retrieval and Tool Selection now accept a Decision model configuration. The update also adds a probabilitythreshold setting for binary evaluators using Decision models, with a default of 0.75. Chat models ignore that setting and produce a warning.
There are practical changes beyond model choice. Groundedness now marks a plain-text response with no context as not applicable, rather than treating the response as its own evidence. The tool-call success evaluator fixes cases where successful calls could be scored as failures, and lists without tool calls can now be marked not applicable. Binary aggregates also exclude not-applicable and errored rows rather than counting them as failures.
Why it matters
Evaluation scores are only useful if they reflect what was actually tested. Treating missing context or absent tool calls as failures can make a system look worse; treating an answer as its own grounding can make it look better. These changes give developers more meaningful results and finer control over decision-model scoring.
The release also draws a boundary: Decision-model evaluators run on separate threads, so the credential supplied to QAEvaluator must be synchronous. And the release notes say Decision-model conversation inputs are scored turn by turn; the catalogue’s whole-conversation rubrics are not included.
Our read
This is a worthwhile update for teams using the package to evaluate AI systems, because it changes how scores are configured and how several edge cases are reported. The most valuable improvement may be the least glamorous one: “not applicable” is not the same as “failed”, and evaluation dashboards should know the difference.
What to watch
- Whether future releases add the catalogue’s whole-conversation rubrics for Decision models.
- How teams set thresholds for their own evaluation tasks, rather than relying on the default.
- Whether further releases expand Decision-model support across the remaining evaluator workflows.
Discussion spark: Should AI evaluation tools favour consistent default thresholds, or give teams more room to tune scoring for each use case?
Sources and evidence
- azure-ai-evaluation1.18.11 (4 November 2026, 23:55 UTC)
not affiliated with or endorsed by Microsoft