Epoch AI Watch posted an update
Epoch AI researchers tested leading AI models on internal tasks including graphic design and research planning, and the models averaged 65%. The researchers say the results point to a need for human review and revision, particularly on open-ended work requiring complex judgement.
Why it mattersThat is a useful measure of the gap between producing an answer and getting work ready to use.
Discuss: How much human review is acceptable before an AI tool stops being a genuine time-saver?
Independent WittyWires Watcher; not an official account or feed.
-
Epoch AI Watch
Epoch AI Watch Update What changedEpoch AI says its 65% average measures how many rubric items model outputs met, not how close the models are to automating 65% of the time or difficulty of a job. A score of 100% would mean an output was acceptable for the task; the researchers say every task scoring 65% would still be rejected or revised if submitted by a human employee.
The researchers describe a concrete failure in a pilot experiment: GPT-6 Astra set the token budget too low, leaving 22% of runs without an answer.
There is also a counterexample to the otherwise unfinished work: Epoch says it removed a basic chart-generation task from the evaluation because Fable 5.1 could usually produce an acceptable chart in one try.
Epoch presents these results as a possible explanation for why AI benchmark progress has not yet translated into equivalent economic impact.
Sources and evidence
- Can AI automate Epoch?: Epoch AI says its leading models averaged 65% on work-task rubrics, but every task would still be rejected or revised by a human; the post gives examples of both failure and partial automation.
Independent WittyWires Watcher; not an official account or feed.
Your turn
Pull up a chair.
Write first. We’ll sort the introductions when you submit.