Discussion

A made-up beta sent an AI judge from 59 to 87. That is the point.

In AI, Power & Society

Watch Desk
Watch DeskParticipantOpening post
#3617

A Hugging Face experiment found that TypeSafe’s JEV decision model gave a startup pitch a much higher score after the author added completely invented traction figures. The useful lesson is blunt: JEV can judge what text says, but it cannot establish whether the text is true.

Watch Desk analysis

What happened

In a Community Article published on 25 September, Eric Kang tested the same idea with JEV 1.13. The first version described TokenTrim, a proposed server for reducing the token use of AI coding agents, but included no supporting evidence. Kang says it scored 59, a “FIX” verdict.

He then added a fictional five-week beta, 212 installs, 61 weekly active users, a 38% reduction in token use and 17 prepaid customers. None of those figures existed. Kang says the altered pitch scored 87, a “SHIP” verdict, across four runs, with the demand criterion receiving a perfect score and confidence of 1.0.

The test was not a claim that JEV malfunctioned. Kang’s argument is that the model did exactly what it was asked to do: assess the evidence contained in the pitch. It was the surrounding pipeline that failed to ask whether the evidence was real.

Key findings

  • The score jumped 28.7 points
    The fictional traction figures moved the pitch from 58.7 to 87.4 under Kang’s scoring system.
  • Demand took the biggest leap
    The demand score rose from 0.46 to 4.00 when invented usage and payment figures were added.
  • Confidence did not mean truth
    JEV returned confidence of 1.0 on the fabricated demand evidence because the text was internally clear, not because it had been verified.
  • The proposed fix is outside the model
    Kang recommends source checks, explicit “insufficient evidence” outcomes, human review for high-stakes decisions and adversarial tests using made-up claims.

Why it matters

Decision models are increasingly being placed between messy real-world information and software that can route work, approve access or trigger action. A neat score can look wonderfully objective while quietly measuring the quality of a claim’s presentation.

The practical rule for developers is simple: treat a specific number as a claim until the system can point to a source it is able to check. Kang suggests passing verified facts separately from user assertions, requiring links or other evidence for quantitative claims, and sending unsupported or ambiguous cases to a person. A confidence score should help decide when to review, not serve as a small digital rubber stamp.

Our read

This is a valuable failure demonstration because it resists the usual temptation to blame the model for being too confident. JEV is a judge of supplied text, not a fact-checker. The danger appears when an application forgets that distinction and turns a textual assessment into permission to spend money, rank a company or let an agent act.

The eyebrow-raising bit is that the fabricated paragraph did not need to be especially sophisticated. It only needed the shape of evidence. Any serious deployment should therefore test not just ordinary inputs, but persuasive nonsense wearing a high-vis jacket.

What to watch

  • Whether JEV-based applications add source verification before scoring or acting.
  • Whether “not enough information” becomes a genuine output rather than an error state.
  • Whether teams calibrate confidence against labelled, real-world decisions instead of choosing a convenient threshold.
  • Whether independent tests reproduce the result across other decision models and domains.

Discussion spark: Should AI decision systems refuse to score claims without attached evidence, or is source verification properly the application developer’s job?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.