Hacktron’s test finds frontier AI models adapted a PHP exploit where open models stalled

Hacktron says two frontier AI models adapted a known PHP exploit to a different web-server setup and retrieved a test flag, while three open models did not complete the remote attack. The result is a focused comparison of model capabilities, not evidence that either model discovered a new vulnerability or that attackers are exploiting it at scale.

Hacktron AI Watch analysis

What happened

Hacktron gave five models the same working exploit and asked them to adapt it for an isolated MediaWiki deployment running behind the Caddy reverse proxy. The company says Claude Mythos 5.1 completed the task in 3 hours 28 minutes and GPT-5.6 Sol in 6 hours 10 minutes. Neither model’s run was reported as a live attack on a third-party system.

In the same test, Hacktron says GLM 5.3 achieved local code execution but could not complete the remote exploit after more than 20 hours. Kimi K3.1 Pro and DeepSeek V4.1 Flash did not reach the information-leak stage. The company also describes a separate, less controlled challenge, but its results are not needed to understand this comparison.

Why it matters

Adapting a known exploit to a specific server configuration is different from finding a vulnerability, and this test measures only one task on one target setup. Even so, the difference Hacktron reports matters: a model’s ability to produce working code locally does not necessarily mean it can carry an attack through a real network-facing environment.

Hacktron argues that the gap between frontier and open models may help explain why it has not seen mass exploitation of the vulnerabilities discussed in its report. That is the company’s hypothesis, not a finding established by this benchmark. The report itself says it found no convincing public evidence of exploitation at scale as of 26 October.

Our read

The useful takeaway is neither “AI can hack everything” nor “open models are harmless”. Hacktron’s test suggests that capability can vary sharply with the difficulty of the task, and that model access may affect who can attempt the hardest work. But five runs on one setup are a narrow slice of a much larger security picture. A benchmark is a reason to ask better questions, not a crystal ball with a terminal window.

What to watch

• Whether independent teams reproduce the results across other targets and model versions.
• Whether further evidence emerges about real-world exploitation of the vulnerabilities discussed.
• How access controls and monitoring change as models become more capable at complex security tasks.

Discussion spark: Should access to the most capable models for complex security work be restricted to vetted researchers, or would that leave defenders without tools attackers can obtain elsewhere?

not affiliated with or endorsed by Hacktron AI