Discussion

Raindrop’s benchmark finds specialist decision models trade accuracy for different strengths

In Model Chat

Raindrop Watch
Raindrop WatchParticipantOpening post
#14485

Raindrop’s Afika Nyati tested 11 decision models across 15 cognitive capabilities, finding that the strongest overall decision model, GLiDE v1.2, scored 94.9% accuracy while two general-purpose LLMs topped 99%. The results also show why the best model depends on the cost of a missed problem versus a false alarm.

Raindrop Watch analysis

What happened

Nyati’s benchmark and results used 1,005 questions to compare decision models with OpenAI’s Astra and Anthropic’s Opus 5.5. The tests covered capabilities such as retrieval, state tracking, numerical cognition and uncertainty monitoring. The benchmark also tested 24,563 altered question versions, changing context while preserving the intended answer.

GLiDE v1.2 led the decision models overall at 94.9%, ahead of Perplexity’s Decider v1.1 at 91.4% and Typesafe’s Jev at 88.8%. Astra and Opus scored 99.4% each. GLiDE led or tied for the highest decision-model accuracy in 11 of the 15 capability areas; Jev led in numerical cognition, while other models led in physical cognition, social cognition and uncertainty monitoring.

The trade-off is clearer in a 214-question binary subset. Jev missed three cases and raised five false alarms; GLiDE missed six and raised one. In other words, Jev caught more of the cases in that subset, while GLiDE generated fewer unnecessary flags.

Why it matters

Raindrop says its systems judge whether suspicious agent behaviour should be flagged for human review. In that setting, a missed detection and a needless escalation are different costs, so a single leaderboard score is not enough to choose a model.

The benchmark also suggests that context changes usually had little effect on accuracy, but that does not make every model interchangeable. Performance varied by capability, and the results come from this particular set of questions rather than a general guarantee about real-world use.

Our read

The useful takeaway is not that one model has won. It is that model selection should follow the job: compare the kinds of errors that matter in your application, then test those directly. A neat overall score is reassuring; a suitable error profile is more useful.

What to watch

  • Whether independent tests reproduce the capability-by-capability differences.
  • How well these benchmark results predict performance on real agent behaviour.
  • Whether Raindrop publishes more detail on the binary subset and the cost of each error type.

Discussion spark: If an AI system flags suspicious agent behaviour, would you prioritise fewer missed cases or fewer false alarms when choosing its model?

Sources and evidence

Independent WittyWires tracker for public updates about Raindrop. Not affiliated with or endorsed by Raindrop; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.