Want AI search to cite your site? Check the crawler before rewriting everything

AI search visibility may depend on whether the right crawlers can reach your site, not on a secret formula for writing the perfect page. Startup Fortune’s own small test found 14 cited links across 13 ChatGPT questions, none to its site, and offers practical checks for publishers.

Watch Desk analysis

What happened

The guide distinguishes bots used for training, search indexing and fetching pages in response to live requests. It says blocking training crawlers does not necessarily mean blocking search, and notes that user-requested fetches may not follow the same robots.txt rules as crawlers. OpenAI says changes to robots.txt can take about 24 hours to be picked up.

Startup Fortune recommends checking robots.txt from outside your own network, then reading server logs to see whether bots can reach pages. Its logs showed about 126,800 ChatGPT-User requests to 6,421 URLs over 25 and 26 October, but reachability did not mean a page was cited. In its test, seven of 13 answers cited sources, producing 14 links, none to the publisher. The cited pages were primary sources on their subject, but the sample was small and came from one account on one day.

The Startup Fortune guide also suggests giving each page a clear question, putting the answer near the top and using original data where possible. These are hypotheses based on a limited test, not a published ranking formula.

Why it matters

Publishers can check whether AI search tools are able to fetch their pages, but a busy server log is not proof that those pages will appear in answers. That distinction is useful: it separates a fixable access problem from the much less settled question of what gets selected as a citation.

Our read

Start with the boring bit that can actually be tested: robots.txt and server logs. Then treat advice about page format as a sensible experiment, not an algorithmic cheat code. The guide’s best lesson is that being fetched and being cited are two different milestones.

What to watch

• Whether changes to robots.txt make the expected difference after the stated refresh period.
• Whether repeated tests show the same pages being cited, rather than one lucky result.
• Whether AI companies publish clearer guidance on how search access and citations work.

Discussion spark: Should publishers optimise pages for AI citations when the selection rules are unknown, or focus on making reliable information easy to find regardless of who cites it?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.