ChatGPT may stop at the page behind a link, but Firecrawl’s new guide sets out a way to give it a larger site to work with: crawl the pages, keep their URLs with the text, then ask for a site-wide summary. The practical distinction is useful: one link is not a website crawl, however confidently the chatbot answers.
Firecrawl Watch analysis
What happened
In a blog published on 29 October, Firecrawl explains how to use its ChatGPT plugin to crawl a site, return page URLs and summaries, and ask for an overview. Its example prompt limits a crawl to 20 pages under a specific path. Firecrawl says each crawled page costs one credit; its free plan includes 1,000 credits a month.
For larger sites, the guide recommends saving crawled pages to text files and adding them to a ChatGPT Project. It says a text file can hold up to two million tokens, while Projects allow between five and 40 files depending on plan. The guide also describes a separate route for Claude users through Firecrawl’s connector. Read Firecrawl’s guide.
Why it matters
The instructions address a common mismatch: asking a chatbot about an entire site when it may only receive one page, or miss content that loads with JavaScript. Crawling and keeping each page’s URL alongside its text gives the model more material and makes it easier to trace a summary back to a source page.
The guide is from Firecrawl, which sells the crawling service it recommends. Its workflow and plan limits are the company’s own descriptions, not a comparison of every available tool. Still, the useful takeaway is concrete: for a one-off question, try a page; for a section or a large site, gather the pages first and preserve their URLs.
Our read
This is a practical how-to, not a magic “read the whole internet” button. Start with a small, clearly bounded crawl, ask for page-by-page summaries with URLs, and check that the answer accounts for the pages you expected. If you will return to the material, a text file in a Project may be tidier than rebuilding the context in every chat.
What to watch
- Whether the plugin’s crawl and URL limits work as described for the sites readers use.
- How often JavaScript-rendered pages or other content are missed in real-world crawls.
- Whether the resulting summaries stay faithful to the linked source pages.
Discussion spark: For research across a large website, would you trust a crawler-and-summary workflow, or rather collect and check the pages yourself?
Sources and evidence
- Source update (29 October 2026, 00:00 UTC)
not affiliated with or endorsed by Firecrawl