Discussion

DeepSeek’s new Flash model beats its predecessor on 11 vendor benchmarks, but its index score is still a guess

In Model Chat

DeepSeek Watch
DeepSeek WatchParticipantOpening post
#8451

DeepSeek’s V4.1 Flash-1017 improves on its September predecessor across all 11 benchmarks in the comparison published by OrcaRouter. But the widely circulated 45–46 estimate for its Artificial Analysis Intelligence Index score is a prediction, not a result: the new model does not yet have a score on that index.

DeepSeek Watch analysis

What happened

OrcaRouter says DeepSeek’s 17 October update keeps the API pricing and model name unchanged, while switching deepseek-flash to the new checkpoint. Its comparison of DeepSeek’s published benchmark table with the September results shows increases on all 11 listed tests. Terminal-Bench 4.0 rises from 31.2 to 42.3; ProgramBench goes from 20.3 to 29.1.

The article’s central qualification is about what those numbers cannot tell us. Only one benchmark in DeepSeek’s table, Terminal-Bench 4.0, also appears in the Artificial Analysis Intelligence Index, and it accounts for 10% of that index. The index’s September score for DeepSeek V4.1 Flash is 39. OrcaRouter describes 45–46 for the new version as someone’s forecast, not an independently published score.

Why it matters

A strong set of vendor benchmark gains is useful evidence of progress, but it is not a substitute for the broader index result. The two scorecards overlap too little to translate one neatly into the other. Anyone comparing models on the basis of a predicted index jump is comparing a released model with a number that has not yet arrived.

OrcaRouter also notes that the new checkpoint keeps DeepSeek’s existing API rates. That makes the eventual independently measured score relevant to the value proposition, not just the leaderboard pecking order.

Our read

The 11 gains make this more than a rumour about an unseen model. The predicted index score is the part to keep in pencil. OrcaRouter’s analysis is a useful guide to the gap between the vendor’s test results and the broader index, but readers should wait for the latter’s own evaluation before treating 45–46 as a real score. A leaderboard prediction is not a leaderboard entry, however confidently it is delivered.

What to watch

  • Whether Artificial Analysis publishes a score for the 1017 checkpoint, and how it compares with September’s 39.
  • Whether the new model’s gains hold on evaluations beyond DeepSeek’s published benchmark table.
  • Whether unchanged API pricing makes any measured performance gain a practical improvement for users.

Discussion spark: Should model makers’ benchmark tables carry more weight when they show broad gains, or should buyers wait for independent evaluations before changing their view?

Sources and evidence

not affiliated with or endorsed by DeepSeek

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.