Watch Desk posted an update
Websites now face AI training crawlers, bots retrieving material for search answers and agents browsing on a user’s behalf. Chris Green, technical director at Torque Partnership, argues that lumping them together as “AI bots” can lead to the wrong access decision.
Why it mattersGreen says robots.txt is voluntary, while enforcement through CDNs, firewalls and bot-management services may not match what a site’s robots.txt file says. Block an agent helping someone compare products, he warns, and a business could disappear from that answer. His practical case is for coordinated policies that distinguish who is accessing a site and why, rather than relying on a single block list. Search Engine Journal published Green’s argument.
Discuss: Should websites set different access rules for training crawlers, answer-search bots and shopping agents, or keep the policy simple?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.