Stanford and Google DeepMind researchers have built TURTLE, a system that evolves tool-use benchmark tasks to make them harder for AI agents. In tests using Gemma 4 31B-it, the share of hard tasks in AppWorld rose by 165%, addressing a growing problem: capable models can outpace the tests meant to measure them.
Google DeepMind Watch analysis
What happened
TURTLE modifies existing tasks by adding filters, extra steps or hidden components, then checks that a solution remains possible. The research, described in a paper dated 19 October 2026, uses three cooperating agents: a proposer based on DeepMind’s AlphaEvolve, a solver, and a feedback generator that guides later changes.
For a task to count as solvable, the solver had to succeed at least half the time across a minimum of ten attempts. Researchers tested the system on AppWorld, τ-Bench and AutomationBench, covering tasks such as email, payments and collaboration. In AppWorld, the share of hard tasks rose by 165% from a starting point of 19.8%; nearly all of 735 evolved tasks landed in the 40% to 60% solver-success range. Crypto Briefing reported the findings.
Why it matters
A benchmark that models quickly master can stop telling researchers much about the next improvement. TURTLE offers a way to generate tougher, still-solvable tasks rather than relying entirely on people to write and check each new challenge by hand. Keeping many tasks near the middle of the difficulty range may also make progress easier to measure than a test where nearly every model aces the questions.
Our read
This is a useful answer to a real measurement problem: make the exam adapt, but do not make it impossible. The result is promising, though the reported experiments used one solver model, Gemma 4 31B-it. Tasks tuned around that model may not land at the same difficulty for others. An evolving benchmark is only as useful as the range of systems it can fairly measure.
What to watch
- Whether TURTLE is tested with a wider range of solver models.
- Whether its evolved tasks are released for other researchers to use.
- Whether difficulty stays comparable as the system adapts across models and benchmark environments.
Discussion spark: Should benchmark tasks be evolved against a reference model, or does that risk measuring differences from that model rather than general tool-use ability?
Sources and evidence
- Stanford and Google DeepMind build TURTLE to make AI benchmarks evolve – Crypto Briefing (8 November 2026, 23:42 UTC)
not affiliated with, endorsed by, or operated by Google or Google DeepMind