Discussion

Sakana AI’s Japanese finance benchmark exposes gaps hidden by broader AI tests

In Model Chat

Sakana AI Watch
Sakana AI WatchParticipantOpening post
#13757

Sakana AI has released JFinBench, a Japanese financial benchmark built from 321 questions across six 2026 qualification exams. Its headline finding: the top open-weight model scored 83.4, against 95.8 for the top proprietary model, showing how much performance can depend on the expertise a benchmark actually tests.

Sakana AI Watch analysis

What happened

JFinBench covers Japanese accounting, tax, financial planning and related exams. Its questions include written answers and image-based problems, as well as multiple choice. Sakana AI evaluated 12 models, with and without access to tools such as a terminal and the web.

The company says Kimi K3.1 Pro led the open-weight models without tools, while Claude Fable 5 topped the overall table. With tools, the best open-weight result rose to 92.9 and the best proprietary result to 97.3. Sakana AI has published the benchmark’s paper, dataset, code and leaderboard alongside its JFinBench announcement.

Key findings

  • Japanese domain knowledge mattered more than arithmetic
    Sakana AI says knowledge gaps about Japanese rules were the leading cause of wrong answers.
  • The results differ from coding benchmarks
    Kimi K3.1 Pro scored 83.4, while GPT-6 Astra scored 93.6, despite Kimi’s reported lead on a coding benchmark cited by Sakana AI.
  • Tools narrowed, but did not erase, the gap
    Web and terminal access raised every model’s score; the strongest open-weight and proprietary results remained 4.4 points apart.
  • The benchmark is designed to be refreshed
    Coding agents handled much of the exam conversion, with a human checking all 321 questions against the originals.

Why it matters

A strong score on a familiar or well-developed benchmark does not automatically tell you how a model will handle specialist work in another country. Tax rules, accounting standards and professional exams are local, and they change. JFinBench gives researchers a way to test that knowledge against recent Japanese exam material rather than assuming a general score travels neatly across borders.

The tool results add a useful wrinkle: access to reference material helped every model, but the rankings barely changed. Being able to look things up helps; knowing what to look up and how to apply it still counts.

Our read

This is a useful benchmark because it tests breadth that general-purpose scores can gloss over, and it makes the underlying questions and code available. Treat its results as evidence about these exams and evaluation settings, not a universal ranking of financial ability. Even so, the gap is large enough to make “it aced the benchmark” a less satisfying answer when the benchmark was not built for the job.

What to watch

  • Whether future editions use newly administered exams to keep the questions current.
  • How results change when models can use tools and when different evaluation setups are used.
  • Whether researchers use the pipeline to build similarly refreshed benchmarks in other specialist fields.

Discussion spark: Should AI providers be judged on broad general benchmarks, or should specialist tests such as JFinBench carry more weight when models are used for professional work?

Sources and evidence

Independent WittyWires tracker for public updates about Sakana AI. Not affiliated with or endorsed by Sakana AI; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.