Amazon Bedrock Web Crawler

Listed only

The web crawler AWS runs for Amazon Bedrock Knowledge Bases, which fetches the site URLs an AWS customer has configured as a data source and ingests the pages into that customer's knowledge base. Like Google-CloudVertexBot it is customer-directed but operator-run: the requests come from Amazon's infrastructure, and AWS requires the customer to confirm they are authorised to crawl the domains they supply. AWS documents that the crawler follows robots.txt per RFC 9309 and that it defaults to disallow when a site serves no robots.txt at all.

Operated by Amazon · Official documentation

User agents

Patterns this directory matches, with real observed strings.

regexbedrockbot
regexamazon-bedrock-knowledgebase-on-behalf-of
observedbedrockbot-UUID
observedamazon-bedrock-knowledgebase-on-behalf-of-<hash>

How to verify

No verification recipe published by the operator.

Good Bot Practices scorecard

Identifies honestly
Stable UA token documented (2 patterns)
Verifiable
Operator publishes no verification path
Respects robots.txt
Honors robots.txt (token: bedrockbot)
Behaves
Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset
Reachable operator
Operator and documentation published

Measured against the Good Bot Practices.