Six billion AI requests reveal the surprisingly simple rules behind faster model serving

Researchers and Chutes have released a public dataset containing more than 6.12 billion production LLM requests, offering an unusually detailed look at how real-world AI traffic behaves. The most useful finding is strikingly practical: 99% of repeat requests arrive within 15 minutes, meaning serving systems may not need exotic cache strategies to capture much of the available reuse.

Watch Desk analysis

What happened

The dataset, published through Agentic System’s open data hub, covers a year of Chutes traffic across 9,174 models. It includes request metadata, token counts and information about prefix-cache reuse, giving researchers material for studying the less glamorous but increasingly important machinery underneath AI applications.

The accompanying account says the traffic averages 27 billion tokens a day. Its analysis reports that simple least-recently-used cache eviction often performs as well as or better than more complex policies, while cache-aware routing improves token hit rates without creating major load imbalance. It also identifies a gradual fall in output lengths, which the source links to growing automated use of language models.

Key findings

• 6.12 billion requests:
  A large production trace gives researchers something better than a toy workload for testing serving systems.
• 99% of repeats within 15 minutes:
  Short-lived cache strategies may capture most repeated traffic.
• 9,174 models:
  The data reflects a broad serving environment rather than one carefully behaved model.
• Simple eviction holds up:
  Least-recently-used caching often matches or beats more elaborate alternatives, according to the source.
• Shorter outputs over time:
  The dataset points towards more automated, task-focused model use.

Why it matters

Inference costs are shaped by what happens after a model is released: repeated prompts, cached prefixes, routing decisions and the sheer amount of text generated. A large production trace can help engineers test those systems against reality rather than a neat spreadsheet of invented traffic.

The apparent lesson is wonderfully unfashionable. Better AI infrastructure may sometimes come from applying simple policies consistently, not from adding another layer of cleverness. A 15-minute reuse window could be more valuable to an operator than a beautifully complicated cache policy that needs its own onboarding programme.

Our read

This is a strong computing story because it exposes concrete behaviour in the infrastructure serving modern AI. The dataset could help researchers build more realistic benchmarks and help operators decide where optimisation effort is worth spending.

The evidence is an attributed account on the dataset website, not an independently reproduced research paper in the supplied material. Readers should therefore treat the figures as a useful public claim and inspect the data, methodology and access conditions before turning them into production assumptions.

What to watch

• Whether independent researchers reproduce the reported cache and routing findings.
• How representative Chutes traffic is of commercial, private and enterprise AI workloads.
• Whether shorter outputs continue as automated model use expands.
• Whether the dataset enables serving benchmarks that better predict real deployment costs.

Discussion spark: Should AI infrastructure teams favour simple, well-tested caching policies, or invest in complex optimisation when traffic patterns vary so sharply between services?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.