Aleph Alpha finds German benchmark translations can change what AI tests measure

A German benchmark can end up testing whether a model can decode a bad translation, rather than the skill its score claims to measure. Aleph Alpha says its review of 30 translated LLM benchmarks found translation errors, grammatical clues and culturally specific questions that can all distort results.

Aleph Alpha Watch analysis

What happened

In a post published on 27 October, Aleph Alpha describes examining German translations of popular AI benchmarks. The company highlights three problems: mistranslated words or questions, grammar that changes or gives away an answer, and questions that remain rooted in American culture despite being put into German.

One example turns “honey”, both a term of affection and something spread on toast in English, into “Honig”. The literal translation loses the affectionate meaning, so the riddle no longer tests the same thing. Elsewhere, translating “consumer” as “Verbraucher” gives an ecological question an economic-sounding answer; Aleph Alpha says “Konsument” is the relevant term.

The post says that in half of the machine-translated evaluations the authors checked as native speakers, roughly a third of examples had at least one of the translation problems they describe. Aleph Alpha argues that such changes can make scores incomparable with the English originals.

Why it matters

Benchmark scores are often used to compare models, but a score is only useful if the test measures what people think it measures. A German version that introduces broken grammar, cultural assumptions or a different question may reward language-repair skills and guesswork alongside the intended ability.

That does not make every translated benchmark worthless. It does mean that a headline score needs context: which language version was used, how its questions were checked, and whether the test still measures the same mixture of knowledge and skills.

Our read

This is a useful reminder that evaluation is not just a spreadsheet with a stern expression. Aleph Alpha’s examples show how small translation choices can change a question’s meaning or even reveal its answer. The company is making the case for German-language evaluation built for German speakers and contexts, rather than relying on translated English tests.

The findings come from Aleph Alpha’s own review, so the reported error rates should be read as the authors’ assessment of the translations they examined. But the underlying point is clear: if the test changes in translation, the score may change meaning too.

What to watch

• Whether benchmark makers publish quality checks for translated versions.
• Whether German-language evaluations are developed and validated beyond direct translation.
• Whether model comparisons make clear which language and version of a benchmark they use.

Discussion spark: Should translated AI benchmarks be treated as comparable with their English originals, or should each language have its own purpose-built tests?

not affiliated with or endorsed by Aleph Alpha