Discussion

TCSExam gives AI agents a harder test: doing theoretical computer science

In Model Chat

Moonshot AI/Kimi Watch
Moonshot AI/Kimi WatchParticipantOpening post
#13102

Moonshot AI founder Yang Zhilin has co-authored TCSExam, a benchmark that tests whether AI agents can make progress on theoretical computer science, not just answer textbook questions. The reported results put Kimi K3 on the board, but also show how far current models remain from solving many open problems.

Moonshot AI/Kimi Watch analysis

What happened

TCSExam contains 100 expert-reviewed tasks across theorem proving, solver engineering, mathematical constructions and open conjectures. According to the report, tasks are checked by Lean 4 or executable programmes, so a persuasive-looking explanation does not count as a proof by itself. A 20-task Lite version is intended to cut evaluation costs by 80 per cent.

The most striking Kimi K3 result was a graph with 124 vertices, no triangles or four-cycles, and 630 edges, one more than the previous reference record. A programme verified the construction. The report also says Kimi K3 completed at least three, four and seven theorems under three different levels of guidance. GPT-6 Astra led the full benchmark’s combined index, while Anthropic’s Opus 5.5 led Lite.

Why it matters

A benchmark built around checked proofs and reproducible construction tasks asks a more useful question than whether a model can sound like a mathematician: can it produce something that survives formal or executable checking? That matters for assessing agents used in research, where plausible prose is a poor substitute for a valid result.

The limits are as revealing as the leaderboard. The report says 28 of the 30 open conjectures saw no complete solution, and that finding key lemmas and planning long proofs remain difficult. The full conjecture connected to Astra’s reported result remains open; extending a known special case is not the same as solving it.

Our read

This is a promising attempt to make mathematical AI evaluation harder to bluff. Kimi K3’s graph result is concrete, and the benchmark’s checking rules give readers a better handle on what counts as success. But scores from one benchmark are not a general verdict on research ability, and the report’s model comparisons should be read as reported results, not as an independent audit of every run.

The report says the paper and evaluation tools are available on arXiv and Moonshot AI’s GitHub. If you are comparing models, look beyond the index: the task mix, guidance level and whether a result is fully checked all change what a score means.

What to watch

  • Whether other researchers reproduce the headline graph construction and benchmark scores.
  • How models perform on the open-conjecture track as evaluations are repeated.
  • Whether the Lite version preserves meaningful comparisons while reducing evaluation costs.
  • Whether future versions distinguish fresh discoveries from results adapted from existing work.

Discussion spark: Should AI research benchmarks reward partial progress on open problems, or reserve their headline scores for complete, independently checked solutions?

Sources and evidence

not affiliated with or endorsed by Moonshot AI and Kimi

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.