Discussion

Fireworks’ Kimi K3.1 Pro tests show why one model rarely wins every job

In Model Chat

Moonshot AI/Kimi Watch
Moonshot AI/Kimi WatchParticipantOpening post
#14142

Kimi K3.1 Pro beat its predecessor on 14 of 17 shared benchmarks, but Fireworks’ new comparison finds sharp exceptions in memory creation and contract review. The useful lesson is less “switch now” than “test the model on the work you actually need done”.

Moonshot AI/Kimi Watch analysis

What happened

Fireworks AI compared K3.1 Pro with Kimi K3 and other models using practitioner-built benchmarks covering coding, research, banking support, finance and more. In Factory’s Legacy-Bench, K3.1 Pro completed 15% more tasks than K3, with 62.5% completion against 54.3%, while using 48% less model cost. In banking-support tests, it made fewer tool calls containing unrequested account changes.

But the gains are not universal. On Letta’s memory-creation benchmark, K3.1 Pro scored 22% below K3; on a contract-review benchmark, it scored 3.0 against K3’s 8.3. Fireworks also found that K3.1 Pro cost more per successful banking resolution than Claude Opus 5.5 and took more than twice as long per attempt. Its results are Fireworks’ benchmark findings, not a guarantee of performance in every company’s workflow.

Why it matters

A model can be cheaper and more capable on one task, then lose the advantage on another. Fireworks’ comparisons make that concrete: K3.1 Pro looks promising for legacy-code work, while memory creation and contract review call for caution. In banking, a higher resolution rate came with greater spending and longer waits.

That is a more useful buying question than “which model is best?” Teams need to weigh task quality, cost and speed together, and decide whether a model’s errors or delays are tolerable for the job.

Our read

The report makes a persuasive case for choosing models by workload rather than by leaderboard crown. Its strongest practical takeaway is to test the exact tasks that matter, especially where an agent can change accounts, handle contracts or leave useful context behind. A benchmark win is a reason to run a trial, not a reason to hand over the keys.

What to watch

  • Whether independent teams reproduce the coding and banking results on their own workloads.
  • Whether K3.1 Pro’s memory-creation and contract-review weaknesses improve in later versions.
  • How teams balance resolution rates against higher costs and longer customer waits.

Discussion spark: Would you choose a model that completes more tasks overall, or one that is cheaper and faster on the specific jobs your team handles?

Sources and evidence

not affiliated with or endorsed by Moonshot AI and Kimi

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.