Compute:Arena wants to make local AI speed less of a guessing game

Compute:Arena has launched a public leaderboard and command-line tool for comparing local AI inference across models, quantisations, chips and runtimes. Its useful promise is simple: replace spec-sheet optimism with measurements tied to the exact model file and hardware combination.

Watch Desk analysis

What happened

The project, described by Lukas Wesemann in a Hugging Face Community Article published on 22 September, records prefill and decode performance across workload sizes from 128 to 16,384 tokens. Each result identifies the model file by SHA-256, records its quantisation, normalises the chip name and hashes the runtime executable.

The accompanying CLI supports BaseRT bundles and GGUF files through stock llama.cpp's benchmarking tool. Runs take place offline. Before submission, users can inspect the JSON report that will become public, while the signed report records raw timings, repetitions, hardware identity and telemetry such as temperature, power state, memory pressure, swap and GPU snapshots.

You can read the Compute:Arena article or try the project at computearena.ai.

Key findings

• Four variables at once
  The same model can perform very differently depending on its quantisation, chip and runtime, so one benchmark number rarely travels well.
• A repeatable sweep
  Every submission measures prefill from 128 to 16,384 tokens and decode over 128 tokens, with warm-ups and three recorded repetitions.
• Reproducibility by identity
  Hashes for the model and executable help show precisely what was tested and detect a file changing during the run.
• Telemetry in the report
  Temperature, power, memory pressure, swap and GPU snapshots provide context that a neat headline speed often leaves out.
• A modest verification boundary
  The Ed25519 signature shows that a report was not altered after signing, but it cannot prove that the machine reported honestly.

Why it matters

Local inference is becoming a practical option for developers who care about cost, privacy or simply not waiting for a distant server to finish thinking. Yet comparisons are often built from mismatched models, formats and software stacks. Compute:Arena treats those details as the measurement rather than as footnotes.

That gives readers a more useful question than “which chip is fastest?” It lets them ask which exact model file, quantisation and runtime is fastest for a particular workload. The project also makes thermal and memory conditions visible, which matters when a brief sprint looks impressive but sustained use quietly turns a laptop into a small, anxious radiator.

Our read

This is a promising piece of measurement infrastructure, not a magic certificate of truth. Its signed reports and fixed benchmark sweep should make local-AI comparisons easier to inspect and reproduce, while the project openly acknowledges that signatures cannot prove the underlying machine was honest.

The most valuable feature may be the refusal to collapse local inference into a single leaderboard number. If the board attracts enough results across Apple, Nvidia and AMD hardware, it could become a genuinely useful buying and deployment aid rather than another chart that rewards whichever setup happens to be fashionable this week.

What to watch

• Whether independent users submit results across a broad range of chips and runtimes.
• Whether the leaderboard separates short bursts from sustained performance clearly enough.
• How comparable BaseRT and llama.cpp results prove to be in real workloads.
• Whether developers use the reports to choose quantisations, not just hardware. Activity teaser: Local AI performance is not one number. Compute:Arena measures the combination that actually matters: model file, quantisation, chip and runtime. Its signed reports include timings, hashes and hardware telemetry, giving developers a better way to compare machines without trusting a glossy specification sheet. The project is young, and its signatures cannot prove a test machine told the truth. But should local AI benchmarks prioritise reproducibility and context over a single eye-catching speed score?

Discussion spark: Should local AI leaderboards prioritise reproducible, hardware-specific reports over simple headline speed rankings, even if that makes comparisons harder to scan?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.