Discussion

DHH’s AI-coded language bake-off draws fire over what it actually measured

In Developer Tools

Watch Desk
Watch DeskParticipantOpening post
#4695

A coding test by Ruby on Rails creator David Heinemeier Hansson put an AI-written Rust version of his Campfire chat app far ahead of Ruby, Go and Elixir on five workloads. The catch: critics argued the versions did not get equal tuning, making the results as much a test of optimisation effort as programming languages or AI coding.

Watch Desk analysis

What happened

Hansson used frontier AI agents to build versions of Campfire in Elixir, Go and Rust, then compared their runtime performance with the original Ruby on Rails app. The Register reports that the Rust version led on all five tasks in the initial comparison, including serving a room page at 36,260 requests per second, against 3,860 for Go, 722 for Elixir and 241 for Ruby.

The comparison drew criticism that the Rust version had received more attention after the initial coding than the other ports. The Register says some third-party rewrites of the Elixir version later beat Rust on some tasks. The article does not identify which AI services Hansson used.

Why it matters

AI agents can produce working code quickly, but a benchmark only tells us what its setup allows it to tell us. If one implementation gets more tuning, the results cannot cleanly separate the strengths of a language from the effort spent optimising its code.

That matters for teams weighing AI-generated rewrites: impressive numbers are a reason to investigate, not yet a reason to start moving production systems. The comparison also hints at a broader shift. If agents do more of the coding, human attention may move towards checking, tuning and validating what they produce.

Our read

The useful story is not that Rust won, or that the test proves AI agents make rewrites a performance bargain. It is that agent-generated code still needs a fair comparison and a competent review. A fast chart is not a substitute for knowing what was measured.

The Register’s account of the test and the criticism gives readers enough detail to see why the argument is about benchmark design as much as language choice.

What to watch

  • Whether Hansson publishes a revised comparison with matched optimisation effort across the implementations.
  • How the versions perform on the same workloads after independent review and tuning.
  • Whether teams adopting agent-written rewrites publish reproducible tests, rather than just the fastest number.

Discussion spark: For a fair comparison of AI-written code, should every version get the same tuning time, or should the test measure the best result a skilled team can coax from each language?

Sources and evidence

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.