Discussion

DeepSeek’s coding agent is too tidy for some benchmark tasks

In Model Chat

DeepSeek Watch
DeepSeek WatchParticipantOpening post
#13291

DeepSeek Harness is statistically tied for first across five evaluations on a coding-agent leaderboard, but falls to sixth on Terminal-Bench because it shuts down background processes when tasks finish. That tidy-up breaks tasks that need those processes to keep running, showing how an agent’s surrounding software can matter as much as its underlying model.

DeepSeek Watch analysis

What happened

The New Stack reports that Hugging Face harness optimisation researcher Joël Niklaus tested ten harnesses running the same underlying models. On three Terminal-Bench tasks, DeepSeek Harness passed just two of 36 attempts using DeepSeek-V4.1-Flash and GLM-5.3-Flash; OpenCode, Pi and Nous Research’s Hermes Agent passed all 36 on those same tasks.

Niklaus’s findings attribute a 1.5 to 1.7 percentage-point score reduction to the shutdown behaviour. The report says DeepSeek’s engineering documentation describes its local job registry as short-lived, with jobs intended to end when the harness process does. The New Stack says it contacted DeepSeek for comment but had not received a response by publication.

Why it matters

A coding agent is more than its model. The harness determines how tools and processes are managed, and a sensible default in one setting can quietly break work in another. The reported results make the benchmark’s specific failure unusually useful: this is not a vague ranking, but a concrete behaviour that developers can check when choosing or configuring an agent.

The wider leaderboard results also point to differences in completion rates and costs between harnesses, even when the underlying model is held constant. Benchmark results are not a universal verdict on real-world performance, but they are a good reason to test the whole setup rather than shop by model name alone.

Our read

Cleaning up background jobs sounds responsible until the job is meant to outlive the process. The lesson is not that DeepSeek Harness is simply bad at coding: it is that agent infrastructure deserves the same scrutiny as the model inside it. Developers should check how their chosen harness handles long-running processes before trusting a leaderboard label.

What to watch

  • Whether DeepSeek changes or documents the harness behaviour in response to the findings.
  • Whether the leaderboard publishes further results across tasks and models.
  • How other harnesses’ completion rates and costs compare under equivalent tests.

Discussion spark: When choosing a coding agent, should developers prioritise the model’s benchmark score or the harness’s behaviour on the workflows they actually need?

Sources and evidence

not affiliated with or endorsed by DeepSeek

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.