AI crawlers 43

Allen Institute for AI · AI crawlers

The Allen Institute for AI's crawler, used to build open training datasets such as Dolma. It crawls selected domains for web content that is then used to train open language models. Ai2 runs it under two user agents, a plain AI2Bot and an Ai2Bot-Dolma variant that names the dataset, both pointing at the same crawling notice. Ai2 publishes no IP ranges, ASN, or reverse-DNS pattern, and its crawling notice offers user-agent filtering as the only documented control.

Aleksma Ai, Inc. (Lyrenth) · AI crawlers

The crawler behind Lyrenth, an index of the public web that serves each page to software agents as a single structured "AIDocument". The operator states the crawler is an indexer and not a model trainer, that it implements RFC 9309 robots.txt including Crawl-delay, Sitemap and HTTP 429/503 backoff, that it fails closed when robots.txt cannot be read, and that it fetches at most once every two seconds per domain across the whole fleet. The bot page offers three independent ways to confirm a request: a published address feed, Web Bot Auth request signatures, and forward-confirmed reverse DNS under lyrenth.com.

Amazon · AI crawlers

The web crawler AWS runs for Amazon Bedrock Knowledge Bases, which fetches the site URLs an AWS customer has configured as a data source and ingests the pages into that customer's knowledge base. Like Google-CloudVertexBot it is customer-directed but operator-run: the requests come from Amazon's infrastructure, and AWS requires the customer to confirm they are authorised to crawl the domains they supply. AWS documents that the crawler follows robots.txt per RFC 9309 and that it defaults to disallow when a site serves no robots.txt at all.

Amazon · AI crawlers

The web crawler AWS runs for Amazon Quick, which fetches the website content an AWS customer has configured as a data source and indexes it into that customer's knowledge base. Like the Bedrock and Q Business crawlers it is customer-directed but operator-run. AWS documents that it respects the robots.txt protocol, honours user-agent, allow, disallow and crawl-delay fields, reads robots.txt per host, and that it crawls anyway when robots.txt cannot be fetched because of blocking, parsing errors or timeouts.

Amazon · AI crawlers

Amazon's web crawler used to improve Amazon's products and services, including training AI models. Amazon publishes the crawler's source addresses on its own bot page as a dated snapshot; they are recorded here verbatim.

ByteDance · AI crawlers

ByteDance's web crawler, used to gather training data for its AI models. ByteDance publishes no IP ranges, ASN, or reverse-DNS pattern for it.

Common Crawl Foundation · AI crawlers

Common Crawl's crawler that builds the freely available Common Crawl web archive, widely reused as AI training data by third parties.

Anthropic · AI crawlers

Anthropic's crawler that indexes pages to improve search-style answer quality in Claude products, distinct from ClaudeBot's training-data crawl.

Anthropic · AI crawlers

Anthropic's web crawler collecting publicly available web data for training Claude models. Anthropic publishes its crawler source IPs as a machine-readable feed.

Cloudflare · AI crawlers

The crawler behind Cloudflare AI Search, which indexes website content so it can be searched. Cloudflare documents that it only crawls a website the customer owns — the domain must exist in the same Cloudflare account and be selected as an AI Search data source.

Cloudflare · AI crawlers

The crawler behind the /crawl endpoint of Cloudflare's Browser Run (Browser Rendering) developer product, which crawls third-party websites on behalf of Cloudflare customers building applications on the platform. Its user agent is not configurable, and every request is signed with Web Bot Auth HTTP message signatures that site owners can verify against Cloudflare's published key directory.

Upstash · AI crawlers

Context7's documentation crawler, an Upstash project. It reads public documentation pages so that AI coding agents can retrieve and cite them, and the operator states the content is used for AI search with links back to the source rather than for model training. Requests are signed with HTTP Message Signatures (RFC 9421) under the Web Bot Auth scheme.

Diffbot · AI crawlers

Diffbot's general-purpose web crawler, used to build its Knowledge Graph and power its structured-extraction APIs. Diffbot documents robots.txt behavior but publishes no IP list, ASN, or reverse-DNS pattern.

DigitalOcean, LLC · AI crawlers

The crawler DigitalOcean runs when a customer adds a website URL as a data source for a DigitalOcean Knowledge Base, which indexes the content so it can power retrieval-augmented generation for that customer's own agents. The documentation states the crawler indexes up to 5,500 pages per job and skips inaccessible or disallowed links to keep indexing jobs bounded, follows HTML links depending on the behaviour the customer selects, indexes text and certain image types, and ignores videos and navigation links.

Firecrawl · AI crawlers

The fetcher behind Firecrawl's scraping and crawling API, which retrieves web pages on behalf of the developers and AI agents calling that API. Firecrawl's crawl endpoint honours robots.txt by default; ignoring it is an enterprise option that must be enabled by the operator. Firecrawl documents that it does not use a fixed set of outbound IP addresses.

ProRata.ai · AI crawlers

ProRata.ai's crawler for Gist.ai. It systematically fetches licensed and publicly accessible content to power the retrieval-augmented generation model behind Gist Answers, following internal and external links to discover new and changed pages, and may also download the CSS and JavaScript a page needs for accurate content extraction. ProRata documents that the crawler only visits content that is either freely accessible without credentials or explicitly licensed, spaces out its requests, and publishes the two addresses it crawls from, one for production and one for testing.

Google · AI crawlers

Google Cloud crawler that fetches sites on the site owners' request when building Vertex AI Agents. Crawling preferences addressed to its user agent have no effect on Google Search or other Google products.

OpenAI · AI crawlers

OpenAI's web crawler that gathers publicly available data used to train OpenAI's models.

Hive · AI crawlers

ImageSift's crawler, operated by Hive. It scrapes publicly available images across the web to support Hive's ImageSift reverse-image-search and web intelligence products. Standard robots.txt directives are respected.

Moonshot AI · AI crawlers

Moonshot AI's search crawler for Kimi, which analyses pages for relevance and builds the index behind Kimi's search features. Blocking it removes a site from Kimi search results. Listed separately from KimiBot because Moonshot documents it as an independently configurable crawler with its own robots.txt token and its own published address list.

Moonshot AI · AI crawlers

Moonshot AI's crawler for the Kimi assistant, gathering content that may be used to train Kimi's foundation models. Moonshot documents robots.txt control per crawler and publishes a machine-readable list of the addresses its crawlers use, while noting the ranges are dynamic and may change.

Klaviyo · AI crawlers

The crawler behind Klaviyo's Kai Customer Agent. It fetches publicly available pages only from the domains and URLs a Klaviyo customer has explicitly connected to their own account — their store plus any help centre, documentation portal, blog or news source they add — and indexes them so Klaviyo can answer shopper questions and generate content. Klaviyo documents that it does not bypass authentication, paywalls or access controls, that it slows down on 429 and 503 responses with Retry-After, and that it deletes indexed content within days once a page is removed or disallowed.

Leipzig University Natural Language Processing Group · AI crawlers

The LCC crawler is operated by Leipzig University to collect web text for the Leipzig Corpora Collection, a set of linguistic corpora used in natural language processing research.

Liner · AI crawlers

Web crawler operated by Liner, an AI search and research assistant, which gathers pages from the public web and indexes them so Liner's search and research agents can retrieve and cite them. Liner's crawler page documents the user agent, states that access is controlled through robots.txt, and gives worked examples for both a full disallow and path-scoped allow rules. Liner also publishes the crawler's addresses as a JSON feed under the heading "IP address ranges", noting the pool may change over time.

Linkup · AI crawlers

Linkup's web crawler. It discovers and retrieves publicly accessible pages to build and maintain the Linkup search index, which powers search and web grounding for AI applications. Linkup publishes the CIDR ranges its crawler originates from and asks site owners to check both the source IP and the user-agent token, since user agents can be spoofed.

MaCoCu project (Jožef Stefan Institute) · AI crawlers

Crawler for the CEF-funded MaCoCu project, which collects, curates and enriches monolingual and parallel web text to build language corpora for under-resourced languages. The project publishes no IP ranges, so requests cannot be verified.

Meta · AI crawlers

Meta's crawler for AI training data and content indexing across Meta products. Verified by ASN lookup (AS32934); Meta publishes no IP feed.

Meta · AI crawlers

Meta's crawler that navigates the web to improve the quality of Meta AI search results, analysing page content for relevance and accuracy in Meta AI responses. Verified by ASN lookup (AS32934); Meta publishes no IP feed.

Mistral AI · AI crawlers

Mistral AI's indexing crawler. It crawls the web automatically to build the index behind Mistral search, which answers user questions in Vibe. Mistral documents that content collected by this crawler is not used for generative AI training of any kind, and publishes the crawler's addresses as a JSON feed.

Mistral AI · AI crawlers

Mistral AI's training crawler. It collects web content to help build the datasets used to train Mistral's generative AI models, and is documented as separate from both the search-index crawler and the user-triggered fetcher. Mistral states webmasters can disallow this user agent in robots.txt.

National Institute of Information and Communications Technology (NICT) · AI crawlers

The web crawler of the Universal Communication Research Institute at Japan's National Institute of Information and Communications Technology. It collects web pages for NICT's research and development of information processing technologies including multilingual translation, information analysis and artificial intelligence, and NICT states that collected information and research products may be provided to third parties for joint or further research. The crawler page documents a load policy that caps connections per unit of time across a host's several addresses or hostnames.

Noah Wire · AI crawlers

The crawler powering Noah Wire's search and research engine for journalists and editors. It scans publicly accessible news sources — press releases, newsrooms, blogs and announcement feeds — and the operator states it does not scrape, store or republish full articles, producing linked summaries instead. The bot page says requests are signed with HTTP Message Signatures, but publishes no key directory URL, so no verification recipe is recorded.

OpenAI · AI crawlers

OpenAI's crawler that indexes pages to power search results inside ChatGPT. It is distinct from GPTBot and is not used to gather training data.

Webz.io · AI crawlers

Webz.io's legacy crawler user agent (formerly "Omgilibot"), used to collect web, forum, and news content for its data feeds. Webz.io's current public documentation describes successor crawlers ("webzio" / "webzio-extended") and no longer documents this UA string or any IP verification method for it.

Perplexity · AI crawlers

Perplexity's crawler that indexes pages to surface and link websites in Perplexity search results. Perplexity states it is not used to train models.

Qualified · AI crawlers

Qualified's crawler. It crawls customer websites to power Qualified's AI products, such as its AI SDR, during initial setup, on major feature releases and on a schedule the customer's Qualified representative sets. The operator publishes the three source addresses its crawlers request from and asks site owners to allow-list them.

Reflection AI · AI crawlers

The web crawler of Reflection AI, which visits publicly available pages to help build and improve the company's open models. Its bot page is unusually complete for an AI crawler: it names the single user-agent token the crawler uses, tells site owners to match on that token rather than the whole string, publishes the crawler's address ranges as a JSON feed, and states that the addresses reverse-resolve under the operator's own domain.

SB Intuitions Corp. · AI crawlers

Crawler operated by SB Intuitions (a SoftBank AI subsidiary) that collects web pages for AI development and information analysis, including training of its Sarashina language models. SB Intuitions publishes no IP ranges, so requests cannot be verified.

ScalePost Corporation · AI crawlers

ScalePost's on-demand fetcher. The operator documents that it fetches a publicly available page with a single GET request when a customer asks for an analysis of how that content is being crawled, cited and referenced by AI systems, that it does not execute JavaScript, submit forms or follow links out of the page body, and that it retains only extracted metadata. ScalePost states it fetches robots.txt before any URL on a host and obeys both its own token and the wildcard, honours Crawl-Delay across its parallel workers, and backs off on Retry-After.

Parallel Web Systems · AI crawlers

Parallel's indexing crawler, which collects publicly available web content to build and maintain the search index behind Parallel's web APIs. The operator documents the robots.txt token ShapBot and publishes the crawler's addresses as a JSON feed.

Webz.io Ltd. · AI crawlers

Second crawler in Webz.io's crawler pair, which performs ethical validation on the data collected by Webzio and tags it as usable or not usable for AI and machine-learning training. Webz.io publishes no IP ranges.

Yandex · AI crawlers

Yandex's robot for processing robots.txt so that site owners can keep already indexed page content out of the AI responses shown in Yandex Search. Yandex's robot table states it applies to pages that have already been indexed by the primary crawler and that it makes no indexing requests of its own.

You.com · AI crawlers

You.com's crawler that indexes pages for its AI-powered search product. You.com documents a dedicated IP range, a reverse-DNS pattern, and support for signed-request verification via Web Bot Auth.