JevMan puts AI decision models in Pac-Man’s firing line

A Pac-Man-inspired benchmark is testing whether AI decision models can make a string of good choices under pressure, not merely produce one clever answer. Its reported results suggest that speed alone is a poor guide to which model will play well.

Watch Desk analysis

What happened

JevMan gives decision models information about a maze and asks them to choose a direction at each junction. The benchmark records performance across 100 games per model, with an average score used for ranking. IBM Think published the account on 15 October, drawing on answers from Felix Wunderlich, Opper’s chief growth officer and a co-builder of the game.

Wunderlich says Jev and GPT-6 Luna survived for about 61 seconds in the reported runs, while Jev ate the most ghosts. But the leaderboard also shows that the fastest answers did not necessarily win: the five fastest entries included four of the lowest scores. As of 14 October, Wunderlich said a community submission called SemIf led the leaderboard with 3,210 points, ahead of Opper’s own runs.

Why it matters

The test gives developers a more practical question than “which model sounds smartest?” Can it make repeated choices reliably, quickly enough for the task, and at a price worth paying? Wunderlich says every model in the reported runs cost less than two US cents per game, and that one regular chat model missed the two-second deadline on two-thirds of its moves and could not be ranked.

That is useful evidence about this particular game, not a universal ranking of decision models. The benchmark’s own design is the point: a model gets the same information at each junction, and one bad turn can cost a life. Reliability across many decisions matters more than one spectacular run.

Our read

A game is a lively way to make a dry technical distinction visible: fast is not the same as good, and one good answer is not the same as dependable performance. JevMan looks most useful as a repeatable comparison tool, not a crystal ball for how a model will behave in every real-world workflow. Developers should test candidates on their own tasks too; Pac-Man has yet to become a customer-support department.

What to watch

• Whether the open-source benchmark attracts more models and independent submissions.
• Whether rankings hold up as the game’s tasks or decision deadlines change.
• Whether benchmark performance predicts results on practical software decisions.

Discussion spark: Should developers trust a benchmark that rewards repeated decisions in a game, or only tests built around the real task they need a model to perform?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.