AI assistants 15

Amazon · AI assistants

Amazon's on-demand fetcher supporting user actions, such as responding to Alexa queries that need up-to-date information; Amazon states it does not crawl content for generative AI model training. Amazon publishes its live-crawl source addresses on a dedicated page as a dated snapshot; they are recorded here verbatim.

OpenAI · AI assistants

OpenAI's user-triggered fetcher, used when a ChatGPT user or a GPT Action asks the assistant to visit a specific page. It does not crawl autonomously.

Anthropic · AI assistants

Anthropic's user-triggered fetcher, used when a person asks Claude to visit a specific web page (e.g. via tool use in a conversation).

Diffbot · AI assistants

Diffbot's user-triggered fetcher, which Diffbot documents as being used by requests originating on behalf of a human user browsing a URL through Diffbot software, in response to that person's input. Diffbot's robots.txt guidance covers its web crawls; it makes no separate statement about these user-initiated fetches, and publishes no IP list, ASN or reverse-DNS pattern.

DuckDuckGo · AI assistants

DuckDuckGo's real-time fetcher for DuckDuckGo Search's AI-assisted answers, which cite their sources. DuckDuckGo states the data is not used to train AI models, and publishes a machine-readable list of the bot's IP addresses.

Google · AI assistants

The fetcher used by AI agents hosted on Google infrastructure to navigate the web and perform actions on behalf of a user who asked for them. Google publishes a dedicated IP range list for it and signs a subset of its requests with Web Bot Auth under the agent.bot.goog identity. As a user-triggered fetcher it generally ignores robots.txt rules.

Google · AI assistants

Google tools and product features that fetch a specific page because an end user asked for it (Google Read Aloud, Site Verifier, Gemini Notebook, Chrome Web Store, Google Messages, Pinpoint, Publisher Center, and similar), rather than autonomous crawling for search indexing. They generally ignore robots.txt because a human requested the fetch.

Moonshot AI · AI assistants

Moonshot AI's user-triggered fetcher for Kimi, which retrieves a page when a person asks Kimi to summarise it or answers a question needing live web retrieval. Moonshot states it is not used for automated bulk crawling and that, because the actions are user-triggered, robots.txt rules may not directly apply.

Meta · AI assistants

Meta's on-demand fetcher that retrieves a single link at a user's request to support agentic AI features (e.g. an AI assistant navigating a page a user asked about), rather than broad indexing. Because its fetches are requested by a user, Meta documents that this crawler may bypass robots.txt rules. Verified by ASN lookup (AS32934).

Mistral AI · AI assistants

Mistral AI's user-triggered fetcher. When someone asks Vibe a question, it may visit a web page to help answer and link to the source in its response. Mistral documents that it is not used for automatic crawling of the web, nor to collect content for generative AI training, and publishes the addresses it fetches from.

Mozilla (Tabstack) · AI assistants

Fetcher for Tabstack, Mozilla's developer-facing platform for programmatic, AI-driven interaction with web content. Every request carries a dedicated user agent, and the operator documents that Tabstack respects robots.txt rules addressed to it, stops immediately on a disallowed path, fails fast rather than retrying, and caches robots.txt results to reduce follow-up requests.

Perplexity · AI assistants

Perplexity's user-triggered fetcher, used when a user's question requires visiting a specific web page to produce an accurate answer.

SB Intuitions Corp. · AI assistants

SB Intuitions' user-triggered fetcher. The operator documents that when someone asks its Sarashina service a question, this agent visits websites on that person's behalf to improve the quality of the search results it reasons over, that answers may include links to the pages it visited, and that what it retrieves is not used for AI development. That last point distinguishes it from SBIntuitionsBot, which the same page says is used for AI development and information analysis.

Parallel Web Systems · AI assistants

Parallel's user-triggered fetcher. It identifies itself when Parallel accesses content on behalf of a user, giving content owners visibility into user-initiated requests. The operator documents that it is not used for automatic crawling and is intended to provide visibility rather than to be managed through robots.txt, which governs ShapBot instead.

Yandex · AI assistants

A Yandex robot that proxies user actions taken on Yandex services: it sends requests in response to button clicks and downloads pages for online translation, so its requests to a site follow from something a person just did. It does not follow the general robots.txt rules.