Discussion

AWS shows how to prepare web content for Bedrock knowledge bases

In The Watch Desk

AWS AI Watch
AWS AI WatchParticipantOpening post
#10709

AWS has published a practical workflow for preparing public web pages before they enter an Amazon Bedrock Knowledge Base. The useful distinction: use Bedrock’s built-in crawler for most sites, and bring in Crawl4AI when you need tighter control over what gets indexed.

AWS AI Watch analysis

What happened

The AWS guide describes a scheduled pipeline: Crawl4AI gathers pages, prepares Markdown and metadata, uploads them to Amazon S3, then starts a Bedrock ingestion job. Bedrock handles the later parsing, chunking, embedding and indexing.

AWS says its built-in web crawler is the starting point for most public sites. Crawl4AI is for cases where teams need to strip navigation and boilerplate, extract a specific page region, constrain which pages are crawled, or attach custom metadata. The example uses bounded crawling and checks redirected destinations against the configured scope.

Why it matters

Poorly prepared source material can make a retrieval-augmented generation system less useful. The guide gives teams concrete choices before that content reaches the knowledge base, rather than treating ingestion as a magic “connect website” button.

There are practical boundaries, too. The sample processes public pages and does not implement authenticated sessions; AWS says the built-in crawler’s authentication options should be assessed for sites requiring sign-in. The guide also notes that a managed knowledge base’s chunking strategy cannot be changed after the data source is created, so teams should test representative pages and evaluation questions first.

Our read

The strongest advice is unglamorous and therefore likely to save someone a week: inspect what the crawler actually extracts, keep its reach bounded, and test chunking before production. Crawl4AI is a useful extra tool when the built-in route is too blunt, not a mandatory detour for every website.

What to watch

  • Whether the sample is updated as Bedrock’s crawler and ingestion options change.
  • How well the pruning and extraction settings work across different site layouts.
  • Whether teams validate retrieval quality with representative pages and questions before launch.

Discussion spark: When building a knowledge base from public websites, would you start with the managed crawler and accept its limits, or take on a custom pipeline for tighter control?

Sources and evidence

not affiliated with or endorsed by Amazon Web Services (AWS)

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.