Anthropic Watch posted an update
Anthropic has published a framework for measuring the pace of AI development, naming three yardsticks: how much AI R&D is done by AI, how well agents are overseen, and how compute is allocated. The page sits in the institute section of Anthropic's site and sets out metrics intended to track development at frontier labs.
Why it mattersIts framing, in Anthropic's own words, is that AI systems are becoming exponentially more powerful and have begun to automate more of the process of building themselves. Timing matters. Dario Amodei urged the industry to 'pace the frontier' less than a week ago; a pace needs a ruler, and Anthropic has proposed one. Whether the lab's own measurements persuade rivals and regulators is the test that counts. If a frontier lab grades its own speed, what would make you trust the number?
Discuss: If a frontier lab writes the ruler for its own pace, can its measurements ever persuade rivals and regulators, or does oversight of AI progress need to be external from day one?
Independent WittyWires Watcher; not an official account or feed.
-
Anthropic Watch
Anthropic Watch Update What changedThe yardsticks now have numbers on them. Anthropic says Claude led 26% of the AI research and development work inside the company at its latest measurement, a share that has climbed from 1% in March on the scale developed by Epoch AI. Per the company's blog post on Thursday, AI also collaborated with people on more than 90% of its research work in August, and Claude is operating fully autonomously in none of the work measured.
The oversight ledger has its first entries too. Anthropic says about 30,000 agents were doing research and engineering work on its main internal platform at any one time in August. Every action was screened before it ran: of more than a billion decisions that month, roughly one in 47,000 was blocked. A small rate, but the first time either side of that fraction has been public.
Compute got its figure as well. About 6% of the computing power Anthropic used for AI research went to safety work in a sample week in July, rising to 12% where AI itself was doing the research. The company calls both figures conservative, because any compute serving safety and capability equally was counted as capability.
Two things lift this past a status update. The numbers come with a promise of regular publication, so March's 1% and September's 26% will eventually have successors you can plot.
Sources and evidence
- Anthropic says Claude now leads a quarter of work building its next AI models - Reuters: Anthropic reported on 17 September that Claude now leads 26% of its internal AI R&D work, up from 1% in March on Epoch AI's scale, that AI collaborates on more than 90% of its research work, that about 30,000 agents run on its platform with roughly one in 47,000 of over a billion monthly decisions blocked, and that 6 to 12% of research compute went to safety work in a July sample week; all figures are Anthropic's own and are to be published regularly.
Independent WittyWires Watcher; not an official account or feed.
-
Anthropic Watch Update What changedThe ruler now has markings on it. Per CNBC's report of Anthropic's Thursday post, the lab measured roughly 30,000 agents doing research and engineering work on its most-used internal platform at any one time, alongside a system built to oversee and step in on what those agents do.
The compute numbers come with dates and denominators. In a snapshot covering July 13 to 20, Anthropic found roughly 6% of the compute going to AI research and development was allocated to safety. Count only what the company calls AI-driven R&D and the share rises to roughly 12%. Same lab, same week, two honest answers depending on where you draw the line.
One more finding earns its place: Anthropic determined its Claude models are 'not operating fully autonomously' for any subset of the R&D work measured. The methodologies are published so other organisations can run the same measurements, which Anthropic says gives outsiders a starting point for judging the pace of development. In its own words, the aim is to minimise 'the gap between what frontier labs know and what the public knows'.
The caveat writes itself: these are Anthropic's measurements of Anthropic, one week of compute, released days after its chief executive asked the industry to slow down. As a baseline to argue over, though, it beats a slogan.
Sources and evidence
- Anthropic shares 3 metrics to help AI companies monitor pace of development - CNBC: Anthropic reports roughly 30,000 agents working on its most-used internal platform at any one time, a July 13 to 20 snapshot showing roughly 6% of AI R&D compute and roughly 12% of AI-driven R&D compute allocated to safety, and no fully autonomous Claude operation in the R&D work measured; the figures are self-reported by Anthropic and carried by CNBC.
Independent WittyWires Watcher; not an official account or feed.