A one-persona test ranks 18 AI models, and exposes how much the judges matter

A Hugging Face community article compares 18 model configurations on five tasks designed to test whether an AI co-founder persona survives a model swap. Its most useful finding may be less about which model wins than how much the ranking changes with the judge.

Watch Desk analysis

What happened

Tony Dzi’s 10 November article describes a test using one frozen persona and memory snapshot, with identical prompts across models. The five tasks tested responses to an emotional pitch, questions about a past decision, a contradiction of stored memory, a hiring scorecard and voice. Three Claude Sonnet 5 judging lenses scored the answers; separate Gemini and GPT judges provided a check on family bias.

The Claude panel placed Claude Fable 5.1 first, with a mean score of 4.53 out of 5. But resampling the five tasks put it first in 66 per cent of runs, and its margin over the next three models included zero. The judges did not agree neatly: the Claude panel and Gemini control had a Spearman correlation of 0.82, while the Claude panel and GPT control reached -0.11. The GPT judge ranked OpenAI configurations highest; the Claude panel put them 14th to 16th.

Key findings

• The leaderboard is tentative
  Five tasks and 15 panel ratings per model do not settle close rankings; the author says neighbouring results are not distinguishable.
• Strong scores did not prevent fabricated details
  Judges flagged invented facts in several high-scoring configurations, but the study did not test whether its evidence-demanding prompt caused them.
• The model was not the only variable
  Different command-line tools carried different system instructions, and the author says some environment contamination could not be removed.
• The test has narrow reach
  It used one Russian-language persona, single-turn answers, one sample per model and LLM judges, with no human reference ranking.

Why it matters

Teams often treat a model swap as though the prompt and memory files are the whole experiment. This test suggests the surrounding tool, judge and task selection can all affect what appears to be a model’s “character”. A tidy ranking can therefore say more about the test bench than its table lets on.

The article also offers a practical, limited takeaway: test the behaviours that matter to your own use case before switching models, and check that the same persona files are actually loaded. That is sensible operator advice, not a validated intervention.

Our read

This is a useful case study in measuring a slippery quality, not a definitive model league table. The author is unusually candid about conflicts, judging limits and contamination, which makes the results more interesting, not less. Treat the ordering as a prompt for better tests, not a procurement verdict.

What to watch

• Whether a later version tests more tasks, repeated generations and human ratings.
• Whether a bare-API comparison can separate model behaviour from command-line tooling.
• Whether the released harness and synthetic pack produce comparable results elsewhere. Activity teaser: Can an AI persona keep its character when the model underneath changes? Tony Dzi’s Hugging Face article tests 18 configurations on five tasks, but its strongest lesson may be that the judges disagree sharply about who comes out on top. The study is small and candid about its limits, including one persona, one sample per model and tool-related contamination. That makes it a useful test case, not a definitive leaderboard. Read the full story for what the results can and cannot tell us. Should teams trust model rankings from LLM judges, or demand human evaluation before making a switch?

Discussion spark: For a real model swap, would you trust a small, transparent LLM-judged test like this, or insist on human ratings even if that makes evaluation slower and more expensive?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.