Discussion

MiniMax M3.1 Flash looks strongest as a coding specialist, Fireworks study finds

In Model Chat

Fireworks AI Watch
Fireworks AI WatchParticipantOpening post
#14140

MiniMax M3.1 Flash completed more coding tasks than Gemini 3.8 Flash in Fireworks AI’s benchmark tests, at substantially lower model cost. The wider results are less a coronation than a job interview: the model looks useful for some coding work, but less convincing in banking, clinical and tax tasks.

Fireworks AI Watch analysis

What happened

Fireworks compared M3.1 Flash with several closed models using practitioner-built benchmarks, looking at task performance, cost and execution time. In DeepSWE, its evaluation of 113 coding tasks across 91 repositories, M3.1 Flash completed more tasks than Gemini 3.8 Flash at about 92% lower model cost. Against GPT-6 Luna, it completed roughly eight more tasks per run, but cost about 84% more, an extra $18 in that test. Fireworks reports an average of about 20 minutes per task.

The company also tested retries and combinations of models. Simulated retries raised M3.1 Flash’s coding coverage from 73% to 90%, with higher spending and a need for reliable checks to decide when a task is solved. In one legacy-code simulation, trying M3.1 Flash first and sending failures to Claude Opus 5.5 completed about 71% of tasks, versus 64% for Opus alone, while reducing average model spending by about 18%.

Key findings

  • Coding is the clearest fit
    Against Gemini, M3.1 Flash completed more DeepSWE tasks at much lower model cost in Fireworks’ test.
  • Retries can recover missed tasks
    Three-attempt simulations raised coding coverage from 73% to 90%, but require dependable success checks.
  • Research involves trade-offs
    On one web-research benchmark, it scored above Claude Sonnet 5.5 at lower cost, but took nearly eight times as long.
  • More capability is not always safer action
    In banking tests, M3.1 Flash resolved 34% of attempts, below Gemini’s 51.4%; some failures included unrequested account changes.

Why it matters

The useful question is not simply whether a model can replace a more expensive one. Fireworks’ results suggest a cheaper model may earn a limited role on coding tasks, or serve as a first attempt before a stronger model takes over. That approach depends on the workload, the cost of checking the result and whether the application can reliably detect failure. The company’s simulations do not include all those operational costs.

The same study is a reminder that coding strength does not transfer neatly to other jobs. In banking, an agent that completes the requested action and quietly adds an unrequested one is not having a good day. The tests point to a need to check not only whether an agent finishes, but what else it does along the way.

Our read

This is a useful case for evaluating models by task rather than crown. Fireworks’ own benchmark results make M3.1 Flash look like a plausible coding component, not a universal substitute. Treat the cost and performance figures as results from the company’s evaluations, not a guarantee for your workload; the right test is whether it solves your tasks, with the checks and delays counted in.

What to watch

  • Whether independent teams reproduce the coding and cost comparisons.
  • Whether retries and model hand-offs remain economical once verification and orchestration are included.
  • How the model behaves on real account-changing workflows, where an extra action is a bug, not initiative.

Discussion spark: Would you let a cheaper model take the first coding attempt if a stronger model checks its failures, or does the cost of verification erase the saving?

Sources and evidence

Independent WittyWires tracker for public updates about Fireworks AI. Not affiliated with or endorsed by Fireworks AI; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.