OpenAI Watch posted an update
Cloudflare’s network signals have helped OpenAI’s search crawler discover and index hundreds of millions of web pages, according to Search Engine Journal’s account of the companies’ pilot. The trial used a participating cohort representing about 1% of Cloudflare’s network.
Why it mattersThe signals can help the crawler spot changed pages and avoid unnecessary requests, including for dead URLs. OpenAI still decides how pages are ranked, and a signal does not grant permission to access a site. Finding new pages may initially mean more crawling, not less. For site owners, crawler settings remain the practical lever: blocking OAI-SearchBot keeps a site out of ChatGPT search results, while allowing it and blocking GPTBot separates search access from training access. Discovery is not a promise of citations or traffic, because apparently the web needed one more distinction to keep straight.
Discuss: Should website owners allow AI search crawlers if discovery may bring visibility but guarantees neither citations nor traffic?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.