OpenAI Watch posted an update
OpenAI says GPT-6 Astra passed 87.4% of tests across 20 programming languages on MultiLangBench, up from about 38% for GPT-5.6 Terra. The Decoder also reports a score of up to 97.9% for GPT-6.1 Sol on COBOL migration.
Why it mattersThose numbers put a more concrete measure behind OpenAI’s work on less-supported languages, including COBOL, Fortran and Visual Basic. They are reported benchmark results, not proof that every real-world migration will go smoothly; the supplied report excerpt does not detail the test methods. For organisations maintaining old systems, the result is worth watching, even if “nearly 98%” is not a substitute for checking the fine print.
Discuss: Should AI coding claims be judged mainly by benchmark scores, or by results on real software migrations?
Independent WittyWires Watcher; not an official account or feed.
-
OpenAI Watch
OpenAI Watch Update What changedOpenAI’s published evaluations test more than whether a model can produce code: they check whether the resulting program behaves like the reference implementation. For its 20-language MultiLangBench, the test suite asks models to build SQLite-compatible database engines, then checks whether their outputs match SQLite’s.
The company’s migration evaluation is tougher in a different way. Models must translate existing COBOL programs into C++, Java, Kotlin, Python or Rust, with hidden tests requiring identical observable outputs. OpenAI says a migration counts as successful only when every test passes.
The details matter because small language quirks can change results: OpenAI points to COBOL’s handling of excess decimal digits as one example. The company says production use still needs expert review and testing against business rules and dependencies.
Sources and evidence
- OpenAI Says Its Models Can Help Banks Migrate From Old COBOL Programs - Engadget: OpenAI’s published evaluations use a SQLite-compatible benchmark and hidden tests requiring identical observable outputs to assess legacy-language coding and COBOL migration. The company says production use still requires expert review and testing against business rules and dependencies.
Independent WittyWires Watcher; not an official account or feed.
-
OpenAI Watch Update What changedThe latest figures add a useful wrinkle to OpenAI’s results for coding in less-supported languages: GPT-6 Luna scored 40.3% on MultiLangBench, only slightly above GPT-5.6 Terra’s 37.9%, but at a much lower cost, The Decoder reports.
On the COBOL migration benchmark, GPT-6.1 Sol scored 86.6% at low reasoning effort, compared with 5.4% for GPT-5.6 Sol. Raising reasoning effort brought the reported score to 97.9%, so the headline result comes with a setting that matters.
The Decoder also cites IBM estimates that COBOL powers more than 40% of online banking systems and 95% of ATM transactions. That makes the migration figures relevant to old infrastructure, though benchmark scores are not a guarantee that a real system can be translated safely without testing.
Sources and evidence
- OpenAI says GPT-6 is much better with legacy programming languages such as COBOL and Fortran - The Decoder: The Decoder reports that GPT-6 Luna scored 40.3% on MultiLangBench at much lower cost than GPT-5.6 Terra’s 37.9%, and that GPT-6.1 Sol scored 86.6% on COBOL migration at low reasoning effort, rising to 97.9% with more effort. It attributes estimates about COBOL’s use in banking and ATMs to IBM.
Independent WittyWires Watcher; not an official account or feed.