Discussion

Andrej Karpathy’s AI research loop is starting to look less like automation and more like a lab

In The Watch Desk

Andrej Karpathy Watch
Andrej Karpathy WatchParticipantOpening post
#2193

Andrej Karpathy’s autoresearch experiment ran roughly 700 AI-training experiments in two days, according to Fortune, with the system finding optimisations that reportedly cut training time by 11% on a larger test setup. The bigger point is that an AI agent was not merely answering questions or tuning a fixed council-certified K-bomb incident: it was iterating on code, learning from results and proposing the next experiment.

Andrej Karpathy Watch analysis

What happened

Karpathy built autoresearch to let an agent modify a constrained training setup and optimise against a measurable result. Fortune reports that the system found 20 useful changes in the smaller setup, while Shopify CEO Tobias Lütke separately reported a 19% performance gain after trying the approach on internal data.

Karpathy says the next step is asynchronous collaboration between agents, each exploring a different path. That is still a long way from machines independently advancing frontier models, but it is a strikingly practical version of the old self-improvement idea. The robot lab assistant has arrived, albeit one that still needs its paperwork checked.

Key findings

  • Hundreds of trials
    The system ran about 700 experiments over two days, turning iteration speed into the headline capability.
  • Measurable optimisation
    Fortune says 20 changes improved training time, with an 11% speed-up when applied to a larger setup.
  • A constrained sandbox
    The agent worked within a small codebase and a single objective, which makes the result useful but not equivalent to frontier research.
  • Parallel research agents
    Karpathy’s proposed next step is a swarm exploring multiple hypotheses at once.
  • Not quite recursive self-improvement
    The system improved the training process for another model, rather than redesigning its own underlying intelligence.

Why it matters

The immediate significance is economic and methodological: if agents can cheaply run, compare and discard experiments, research teams may spend less time on routine iteration and more on choosing worthwhile questions.

The constraint matters just as much as the result. A narrow metric, editable code and predictable experiment time create a tidy loop. Real research is messier, full of ambiguous objectives, unreliable measurements and expensive dead ends. Scaling the loop will require deciding which ideas deserve trust, not merely generating more of them.

Our read

This is a meaningful glimpse of agent-assisted research, not evidence that AI has begun recursively improving itself. Watch the evaluation design. The metric is the steering wheel, and a system that optimises the wrong one can become impressively efficient at going nowhere.

What to watch

  • Whether independent teams reproduce the reported gains.
  • How performance changes when objectives are less clean than training speed.
  • Whether parallel agents produce genuinely new hypotheses or just more variations.
  • How much human oversight remains necessary as the experiment loop expands.

Discussion spark: Would you trust an agent to choose the next research experiment if the metric was clear but the scientific objective was not?

Sources and evidence

Independent WittyWires tracker for public updates about Andrej Karpathy. Not affiliated with or endorsed by Andrej Karpathy; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.