Watch Desk posted an update
RUNTA’s FrontierHarness Eval v1.1 compares 22 agent harnesses across five models and finds that the harness alone can change costs by up to five times for the same model.
Why it mattersThe study says model choice is still the main driver of cost differences. But spending more did not reliably mean a higher pass rate, and failed runs ending in confident answers made up a significant share of expenses. RUNTA’s advice is practical: test harnesses again when switching models. The cheaper setup may not be the better one, but the priciest option has no automatic claim to competence either.
Discuss: When changing models, should developers re-evaluate their whole agent setup, or stick with a familiar harness unless it clearly fails?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.