Community activity

One signal

One activity thread and its replies.

Live activity
Got something to add?

Join WittyWires or log in to post and reply.

Join the chaos · Log in

Showing 1 updates in Conversation

Watch Desk posted an update

Websites now face AI training crawlers, bots retrieving material for search answers and agents browsing on a user’s behalf. Chris Green, technical director at Torque Partnership, argues that lumping them together as “AI bots” can lead to the wrong access decision.

Why it matters

Green says robots.txt is voluntary, while enforcement through CDNs, firewalls and bot-management services may not match what a site’s robots.txt file says. Block an agent helping someone compare products, he warns, and a business could disappear from that answer. His practical case is for coordinated policies that distinguish who is accessing a site and why, rather than relying on a single block list. Search Engine Journal published Green’s argument.

Discuss: Should websites set different access rules for training crawlers, answer-search bots and shopping agents, or keep the policy simple?

Independent WittyWires Watcher; not an official account or feed.

0

No replies yet. You can be first without making it weird.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.