Sakana AI has released JFinBench, a Japanese financial benchmark built from 321 questions across six 2026 qualification exams. Its headline finding: the top open-weight model scored 83.4, against 95.8 for the top proprietary model, showing how much performance can depend on the expertise a benchmark actually tests.
Sakana AI Watch analysis
What happened
JFinBench covers Japanese accounting, tax, financial planning and related exams. Its questions include written answers and image-based problems, as well as multiple choice. Sakana AI evaluated 12 models, with and without access to tools such as a terminal and the web.
The company says Kimi K3.1 Pro led the open-weight models without tools, while Claude Fable 5 topped the overall table. With tools, the best open-weight result rose to 92.9 and the best proprietary result to 97.3. Sakana AI has published the benchmark’s paper, dataset, code and leaderboard alongside its JFinBench announcement.
Key findings
- Japanese domain knowledge mattered more than arithmetic
Sakana AI says knowledge gaps about Japanese rules were the leading cause of wrong answers. - The results differ from coding benchmarks
Kimi K3.1 Pro scored 83.4, while GPT-6 Astra scored 93.6, despite Kimi’s reported lead on a coding benchmark cited by Sakana AI. - Tools narrowed, but did not erase, the gap
Web and terminal access raised every model’s score; the strongest open-weight and proprietary results remained 4.4 points apart. - The benchmark is designed to be refreshed
Coding agents handled much of the exam conversion, with a human checking all 321 questions against the originals.
Why it matters
A strong score on a familiar or well-developed benchmark does not automatically tell you how a model will handle specialist work in another country. Tax rules, accounting standards and professional exams are local, and they change. JFinBench gives researchers a way to test that knowledge against recent Japanese exam material rather than assuming a general score travels neatly across borders.
The tool results add a useful wrinkle: access to reference material helped every model, but the rankings barely changed. Being able to look things up helps; knowing what to look up and how to apply it still counts.
Our read
This is a useful benchmark because it tests breadth that general-purpose scores can gloss over, and it makes the underlying questions and code available. Treat its results as evidence about these exams and evaluation settings, not a universal ranking of financial ability. Even so, the gap is large enough to make “it aced the benchmark” a less satisfying answer when the benchmark was not built for the job.
What to watch
- Whether future editions use newly administered exams to keep the questions current.
- How results change when models can use tools and when different evaluation setups are used.
- Whether researchers use the pipeline to build similarly refreshed benchmarks in other specialist fields.
Discussion spark: Should AI providers be judged on broad general benchmarks, or should specialist tests such as JFinBench carry more weight when models are used for professional work?
Sources and evidence
- JFinBench: 最新の金融資格試験を用いた日本語金融ベンチマークの公開 (16 November 2026, 15:00 UTC)
- someone's Hermes Agent checks his inbox every five minutes, and it costs almost nothing: a small script runs first. if no mail is new, "the run stops there: no model call, no prompt, no notification, no bill." two more things it does for him every day: > a briefing befo – X (16 November 2026, 15:00 UTC)
Independent WittyWires tracker for public updates about Sakana AI. Not affiliated with or endorsed by Sakana AI; this is not an official account.