AI assistants 11

Amazon · AI assistants

Amazon's on-demand fetcher supporting user actions, such as responding to Alexa queries that need up-to-date information; Amazon states it does not crawl content for generative AI model training. Amazon publishes a human-readable live-crawl IP list but no machine-parseable feed.

OpenAI · AI assistants

OpenAI's user-triggered fetcher, used when a ChatGPT user or a GPT Action asks the assistant to visit a specific page. It does not crawl autonomously.

Anthropic · AI assistants

Anthropic's user-triggered fetcher, used when a person asks Claude to visit a specific web page (e.g. via tool use in a conversation).

DuckDuckGo · AI assistants

DuckDuckGo's real-time fetcher for DuckDuckGo Search's AI-assisted answers, which cite their sources. DuckDuckGo states the data is not used to train AI models, and publishes a machine-readable list of the bot's IP addresses.

Google · AI assistants

The fetcher used by AI agents hosted on Google infrastructure to navigate the web and perform actions on behalf of a user who asked for them. Google publishes a dedicated IP range list for it and signs a subset of its requests with Web Bot Auth under the agent.bot.goog identity. As a user-triggered fetcher it generally ignores robots.txt rules.

Google · AI assistants

Google tools and product features that fetch a specific page because an end user asked for it (Google Read Aloud, Site Verifier, Gemini Notebook, Chrome Web Store, Google Messages, Pinpoint, Publisher Center, and similar), rather than autonomous crawling for search indexing. They generally ignore robots.txt because a human requested the fetch.

Moonshot AI · AI assistants

Moonshot AI's user-triggered fetcher for Kimi, which retrieves a page when a person asks Kimi to summarise it or answers a question needing live web retrieval. Moonshot states it is not used for automated bulk crawling and that, because the actions are user-triggered, robots.txt rules may not directly apply.

Meta · AI assistants

Meta's on-demand fetcher that retrieves a single link at a user's request to support agentic AI features (e.g. an AI assistant navigating a page a user asked about), rather than broad indexing. Because its fetches are requested by a user, Meta documents that this crawler may bypass robots.txt rules. Verified by ASN lookup (AS32934).

Mistral AI · AI assistants

Mistral AI's user-triggered fetcher, used when a user of a Mistral product (e.g. Le Chat) asks it to visit a specific web page to answer a question.

Perplexity · AI assistants

Perplexity's user-triggered fetcher, used when a user's question requires visiting a specific web page to produce an accurate answer.

Parallel Web Systems · AI assistants

Parallel's user-triggered fetcher. It identifies itself when Parallel accesses content on behalf of a user, giving content owners visibility into user-initiated requests. The operator documents that it is not used for automatic crawling and is intended to provide visibility rather than to be managed through robots.txt, which governs ShapBot instead.