The web's bots,
verified.

AddSearch · Search engines

Crawler for AddSearch's hosted site-search service. It indexes the pages of customer sites so AddSearch can serve search results for them, obeys robots.txt rules written for the AddSearchBot token, and AddSearch publishes the fixed addresses it crawls from for allowlisting.

Microsoft · Search engines

Microsoft's crawler for Bing Ads / Microsoft Advertising. It crawls ads and follows through to the advertised landing pages for quality control, with both desktop and mobile variants.

Algolia · Search engines

Algolia's Crawler, which visits a customer's pages, extracts search-relevant content, and pushes it to Algolia search indices. Algolia documents a fixed User-Agent token and a single static egress IP address for allowlisting.

Amazon · Search engines

Amazon's advertising crawler. It scans web pages that request ads from Amazon's advertising systems, collecting page content for Amazon's classification systems to maintain brand safety and improve ad relevance.

Amazon · Search engines

The web crawler AWS runs for Amazon Kendra, its enterprise search service, which fetches the site URLs an AWS customer has configured as a data source and indexes them for that customer. Like the Bedrock knowledge-base crawler it is customer-directed but operator-run, and AWS states that customers must only point it at their own pages or pages they are authorised to index, and that using it to aggressively crawl sites the customer does not own is not acceptable use.

Amazon · Search engines

Amazon's product crawler, which collects publicly available product details from Amazon selling partner, brand and retailer websites to improve the accuracy and completeness of product information in Amazon's store. Amazon documents that it honours robots.txt user-agent and disallow directives, with changes taking up to 24 hours to apply, and that it does not support crawl-delay, nofollow or noindex. Amazon signs every request from this crawler with HTTP message signatures, publishing the Ed25519 key directory the Signature-Agent header points at.

Amazon · Search engines

The web crawler AWS runs for Amazon Q Business, which fetches the websites an AWS customer has connected as a data source and indexes them into that customer's Q Business application. It is customer-directed but operator-run: AWS states the customer must only index their own webpages or webpages they have authorisation to index, and documents the robots.txt token site owners can use to allow or stop it.

Amazon · Search engines

Amazon's crawler used to improve search experiences in Amazon products and services; Amazon states it does not crawl content for generative AI model training. Amazon publishes its source addresses on a dedicated page as a dated snapshot; they are recorded here verbatim.

Direqt · Search engines

Direqt's search crawler, which discovers links and metadata to surface in Direqt's search features. The operator states it is not used to crawl content for model training, documents the robots.txt token Anomura, and publishes the two addresses the crawler requests come from on its crawler documentation page, saying it will update that page if more are added.

Apple · Search engines

Apple's web crawler, used to index content for Siri, Spotlight Suggestions, and Safari's search features.

Atlassian · Search engines

Crawler behind the Teamwork Graph "custom website" connector, which indexes a site an Atlassian customer has connected so its content is searchable in Rovo. Atlassian documents the robots.txt token and publishes the connector's egress ranges under the rovo-crawler product in its IP-ranges feed.

Baidu · Search engines

Baidu's primary web crawler, fetching pages for Baidu Search indexing.

Microsoft · Search engines

Microsoft crawler, listed among Bing's crawlers, that fetches pages to generate video previews shown in Bing. It runs desktop and mobile variants and is verifiable through the same reverse-DNS check as Bing's other crawlers.

Microsoft · Search engines

Microsoft's primary web crawler, fetching pages for Bing Search indexing.

Microsoft · Search engines

Microsoft fetcher used to generate page snapshots for previews in Bing search results. Runs both desktop and mobile variants.

Bublup · Search engines

Bublup's content-discovery bot. It fetches pages to build the database behind Bublup's suggestion engine, reading page title, description and related images. Bublup states its crawling IPs are not fixed and documents reverse-DNS verification against the bublup.com domain instead.

Channel3 · Search engines

Channel3's product crawler. It visits publicly accessible product detail pages and collects images, titles, descriptions, prices, availability and variants for Channel3's product catalogue, which AI apps and agents query to route shoppers back to the original site. The operator tells site owners not to hard-code its addresses and to verify it by forward-confirmed reverse DNS instead.

Cludo · Search engines

Crawler for Cludo's hosted site-search product. It scans the domains a customer configures, follows links and sitemap URLs, and extracts page content and metadata into that customer's search index. Cludo documents that site owners can steer it with robots.txt rules addressed to the cludo token, and publishes the two proxy addresses — one per cluster, EU and US — that all of its services, crawlers included, make outbound requests from.

Cốc Cốc · Search engines

Cốc Cốc's web crawler, fetching pages for the Vietnamese Cốc Cốc search engine. Separate web and image sub-bots share the same verification domain.

Kakao Corp. · Search engines

Web-document crawler for Kakao's Daum search service in Korea. It collects pages so they can appear in Daum web search results, and Kakao's help centre states that the robot adheres to the robots exclusion standard when collecting documents from all sources, honouring both robots.txt rules written for the Daum token and noindex/none/nosnippet robots meta tags. The same help centre notes a blocked page's title can still appear in results when other pages link to it, because that text comes from the anchor text of the linking page rather than from a fetch of the blocked page.

deepnoc GmbH · Search engines

Research crawler operated by deepnoc GmbH that parses and stores public web page content so that new search engines can query it without running their own crawling infrastructure. No IP ranges are published.

Doofinder · Search engines

Fetcher for Doofinder's e-commerce site-search service. It retrieves the product data feed and page content of the shops that subscribe to Doofinder so their catalogue can be indexed for on-site search. The operator's security FAQ prints the user agent it sends and the three regional addresses it requests from, offering them as an alternative to putting credentials in a protected feed URL.

DuckDuckGo · Search engines

DuckDuckGo's web crawler, used to improve DuckDuckGo's search results.

Exa · Search engines

Exa's search crawler, which discovers and indexes pages on the public web so that people and applications can find, retrieve and cite them through Exa. Every request is signed with HTTP Message Signatures (RFC 9421) under the Web Bot Auth scheme, so a request can be verified cryptographically regardless of its source IP address.

FindFiles.net · Search engines

The crawler behind the FindFiles.net file search engine. It checks that publicly accessible files users search for are still available, using multiplexed HTTP/2 HEAD requests with a minimum ten-second interval per server, and may download images, videos and executables for classification and safety checks. Purpose-specific user agents are published for its link checking, favicon fetching, virus scanning and image classification agents, which all share one crawler host and address.

iontic GmbH (fleebs.com) · Search engines

Crawler for fleebs.com, a German real-time search engine. The operator's bot page documents two identities: one that searches for new information and one that analyses the pages it finds. It documents robots.txt blocking with the FleebsBot token and notes that pages indexed before a block may take time to drop out of the index.

Freespoke · Search engines

Freespoke's search crawler, which fetches pages to build the index behind the Freespoke search engine. The operator documents the crawler's user agent, the robots.txt token FreespokeCrawler and support for the Crawl-delay directive.

Geedo · Search engines

GeedoShopProductFinder is the crawler for Geedo, a product-search engine that indexes publicly accessible product pages from online stores and follows robots.txt rules.

Google · Search engines

Google's crawler for Google Shopping surfaces, including the Shopping tab in Google Search and shopping.google.com. It crawls product and store pages to populate shopping results.

Google · Search engines

Google crawler used by Search testing tools such as the URL Inspection tool in Search Console and the Rich Result Test. Fetches pages on demand to show how Google Search renders them; it has no effect on ranking.

Google · Search engines

A group of Google crawlers that operate under separate agreements between the crawled site and a specific Google product (AdsBot, AdSense, Google Safety scanning, and similar). Several members of this group ignore the wildcard robots.txt rule and must be targeted explicitly.

Google · Search engines

Google's primary web crawler, fetching pages for Google Search indexing.

Google · Search engines

Google's image crawler, which fetches images for Google Images and image features in Search. It is a distinct crawler from Googlebot with its own robots.txt token, though rules for Googlebot also apply to it.

Google · Search engines

Google's video crawler, which fetches video content for Google Search video features. It is a distinct crawler from Googlebot with its own robots.txt token, though rules for Googlebot also apply to it.

Google · Search engines

Google's generic crawler family (GoogleOther, GoogleOther-Image, GoogleOther-Video), used by various Google product teams for fetching publicly accessible content. Robots.txt rules addressed to it do not affect Google Search.

360 Search (Qihoo 360) · Search engines

360Spider is the web-search crawler of 360 Search (so.com), the search engine operated by Chinese internet company Qihoo 360. The operator also runs 360Spider-Image and 360Spider-Video for image and video search.

Babbar (Ibou) · Search engines

The crawler behind Ibou, a French conversational search engine. It discovers publicly accessible pages and indexes them so Ibou can cite and link back to the sites it answers from. The operator documents a 5-second politeness delay per host, publishes its address ranges as JSON, and documents forward-confirmed reverse DNS as the reliable way to tell a genuine request from a spoofed user agent.

IONOS · Search engines

IONOS Crawler (IonCrawl) continuously crawls publicly accessible domains to generate insights into how they are used, which IONOS uses to improve and expand its hosting products. IONOS documents reverse-DNS verification and deletes crawled data after 60 days.

PT Epithre Teknologi Indonesia · Search engines

The crawler that builds the index behind Ison Search, an Indonesian web search engine that also grounds cited AI answers linking back to the original page. Epithre states it trains no model on what the crawler reads. Crawling is paced per site rather than per crawler process: it starts at two requests per second, speeds up only while responses stay fast and error-free, never exceeds about ten per second or four concurrent connections, halves on slow responses and 5xx, and drops to the minimum and pauses on 429 or 403, with robots.txt Crawl-delay always taking precedence. Pages whose text appears only under JavaScript are rendered in a headless browser that sends the same token and follows the same rules. Epithre says it submits no forms, attempts no authentication, reads a CAPTCHA or bot challenge as a refusal rather than solving or retrying past it, and fetches nothing behind a paywall.

Jooble · Search engines

The crawler for Jooble, a job-search aggregator that indexes job listings published across the web. Jooble's bot page states that it uses a web crawler which identifies itself as JoobleBot. No IP ranges are published.

Kagi · Search engines

Kagi's web crawler, fetching pages for the paid Kagi search engine. Kagi publishes a fixed, small set of source IPs rather than a CIDR feed, each paired with a kagibot.org hostname.

Kernel · Search engines

Kernel's search crawler, which fetches pages to build search indexes and retrieval databases. The operator documents it as the Kernel-operated identity, distinct from the user-directed Kernel Agent, and signs its requests with HTTP Message Signatures (RFC 9421) under the Web Bot Auth scheme using the Signature-Agent https://search.bot.kernel.sh.

LINE · Search engines

LINE's crawler. It fetches web pages to support LINE's search features and content services, and adheres to the Robots Exclusion Protocol outlined in robots.txt.

Lyonl Search · Search engines

The crawler used by Lyonl Search to discover and refresh public web documents for its index, with separate identities for web, image and news crawling so site owners can control each class independently. The operator documents robots.txt and crawl-delay support, conditional fetches with ETag and Last-Modified, and that the crawler does not submit forms or bypass authentication. It states that crawler address ranges are not currently published, so no verification recipe is recorded.

Magnet.me · Search engines

Magnet.me's web crawler. It discovers and refreshes career-related pages — vacancies, events and employer information — for the Magnet.me careers platform. The operator documents that it may render JavaScript, follows the canonical version of a page, does not submit forms and adjusts its crawl speed automatically, and it publishes the addresses it crawls from as both a JSON and a plain-text list.

Marginalia Search · Search engines

The crawler behind Marginalia Search, an independent search engine focused on older, text-heavy, and non-commercial web pages. Identifies itself by a bare URL rather than a conventional bot token.

METAJob · Search engines

MetaJobBot is the focused web crawler operated by the METAJob job meta-search engine. It searches websites for job listings to include in the METAJob index.

Mojeek · Search engines

Mojeek's web crawler, fetching pages for the independent Mojeek search engine index.

MotoMiner · Search engines

MotoMiner's vehicle-listing crawler. It primarily crawls automotive dealership websites and adds their vehicle detail pages to the MotoMiner index. The operator documents robots.txt and crawl-delay support, honours bot-specific noindex meta directives, throttles outbound requests against defined traffic thresholds, and publishes the address the crawler currently requests from.

Naver · Search engines

Naver's web crawler, fetching pages for Naver Search indexing across South Korea's largest search engine. Naver documents two ways to confirm a request is really its search robot: a reverse-DNS lookup that resolves under naver.com, and a published JSON list of the IP ranges its search robots use.

NestDaddy Technologies · Search engines

Web crawler behind the NestDaddy search engine. The operator's webmaster page documents that it extracts page title, meta description, headings, language and link structure to build the index, filters adult and abusive content out of results, and fully respects robots.txt including the Crawl-delay directive. It states that the crawler runs from a distributed network whose current ranges are available only on request.

netEstate GmbH · Search engines

The crawler netEstate GmbH runs for its information-extraction products. The operator documents two uses: an Impressums-Crawler that locates a site's imprint page and extracts postal addresses, contact data and company names for address verification, and a Jobcrawler that finds job postings on employer websites. netEstate states that the crawler observes robots.txt, the robots meta element and common opt-out wordings, and that it returns only a hash of any email address rather than the address itself.

Openindex · Search engines

Openindex operates a web-crawling cluster (Apache Nutch on an Apache Hadoop cluster) for research and development of universal and focused search engines. The crawler respects robots.txt and the Crawl-delay directive and identifies itself in the User-Agent.

Paqle · Search engines

Paqle's news crawler. It indexes news articles to discover who is mentioned in the media, crawling at most one page per minute per site and tuning its rate to how often a site publishes. Paqle documents that the crawler falls back to a site's Googlebot robots.txt rules when no Paqlebot rules are present, and that its addresses reverse-resolve under paqle.net.

Huawei · Search engines

Huawei's web crawler, fetching pages for the Petal Search engine used in Huawei's mobile services and for Huawei Assistant content recommendations. Huawei documents reverse-DNS verification against aspiegel.com and petalsearch.com hostnames.

Pipl · Search engines

PiplBot is the web-indexing crawler operated by Pipl, an identity and people-search service. It retrieves and indexes publicly available information, including content behind searchable databases, to build Pipl's people-search index.

Quantcast · Search engines

Quantcast's advertising crawler. It crawls websites to extract page content for interest-based audience categorization and to run quality-assurance checks on advertisement landing pages.

Qwant · Search engines

Qwant's web crawler, fetching pages to build and refresh the index that powers the privacy-focused Qwant search engine. Qwant runs it under two documented forms — the plain crawler and a Qwantbot-news variant dedicated to Qwant News — and the user agent carries a worker id identifying the machine that made the request. Older crawls identify as Qwantify.

SISTRIX · Search engines

The crawler for the Seekport search engine, operated by SISTRIX (Bonn, Germany). It fetches pages to build Seekport's search index and follows the Disallow directives in robots.txt.

Allen Institute for AI (Ai2) · Search engines

SemanticScholarBot is the crawler for Semantic Scholar, an academic search engine built by the Allen Institute for AI. It crawls the web to find scholarly PDFs and metadata for indexing.

Seznam.cz · Search engines

Seznam's web crawler, fetching pages for the Seznam.cz search engine used primarily in the Czech Republic. Seznam publishes the full list of addresses the crawler uses as JSON, and documents that every SeznamBot address carries a fulltextrobot-*.seznam.cz reverse-DNS record.

Zoovu (Germany) GmbH · Search engines

Crawler for the Site Search 360 hosted site-search product, operated by Zoovu (Germany) GmbH, formerly SEMKNOX. It indexes the pages of the customer sites it is pointed at, following links from the configured root domains and skipping external hosts. Robots.txt is not applied automatically: the control panel offers a "Fill Blacklist from robots.txt" action that imports a site's disallow rules into the crawl blacklist, and the crawler honours noindex robots meta tags. The operator publishes the addresses it crawls from for firewall allowlisting.

Sogou · Search engines

Sogou Spider is the web crawler for Sogou, a major Chinese search engine (owned by Tencent). It crawls and indexes web pages, news, and images to build Sogou's search index.

The Stepstone Group · Search engines

The Stepstone Group's job-listing crawler. Its crawler page states that it processes only publicly available information, observes robots.txt directives and keeps intervals between requests to avoid loading servers, and publishes the user agent it sends for transparency.

Stract · Search engines

The crawler for Stract, an open-source search engine. Stract documents the crawler's user agent and robots.txt behavior but publishes no IP list or reverse-DNS verification method.

Elasticsearch B.V. (Swiftype) · Search engines

The web crawler behind Swiftype's site-search product, now operated by Elasticsearch B.V. Unlike a general search-engine crawler it only visits sites that Swiftype customers have asked it to crawl, in order to build a search index for those sites. The operator documents that Swiftbot obeys every restriction it finds in robots.txt, including a full-site disallow, and that it looks for the token Swiftbot.

TinEye · Search engines

TinEye-bot is the web crawler operated by TinEye, the reverse image search engine built by Idee Inc. It crawls the web to discover and index images for TinEye's image-matching search service.

Trovit · Search engines

Trovit's web crawler, which discovers new and updated pages to add to the Trovit classifieds search index. Trovit documents that it does not fetch most sites more than once per second and that it can be blocked with the trovitBot robots.txt token.

urlsuma.de (Gerhard Stöbe) · Search engines

Crawler for urlsuma.de, a German general-purpose web search engine under construction. The operator documents that it re-reads robots.txt without caching before every single fetch, makes about two requests per visit (robots.txt plus one page), executes no JavaScript, and treats a 4xx response as a permanent block. Current crawler addresses are shared on request only.

U.S. General Services Administration (Technology Transformation Services) · Search engines

The crawler for SearchGov (formerly Search.gov), the search service GSA's Technology Transformation Services runs for U.S. federal, state and local government websites. It indexes publicly accessible content only — the operator tells agencies not to register domains that serve secure content such as intranets — and agencies are told to allow it in robots.txt and to allowlist its two published source addresses. GSA's guidance names the user agent and the addresses but offers no feed or reverse-DNS method, so the addresses are recorded as static ranges.

Webz.io Ltd. · Search engines

Crawler for Webz.io (formerly Webhose.io), a web-data provider that collects publicly available content to build structured data feeds. Webz.io publishes no IP ranges, so requests cannot be verified.

Wepch · Search engines

The crawler for Wepch, an independent privacy-focused search engine still in development. Its operator page states the project's purpose, gives a contact address for crawl complaints, and documents the robots.txt token site owners can use to crawl-delay or disallow it.

Yahoo · Search engines

Slurp is Yahoo Search's web crawler. It crawls and indexes pages for Yahoo Search results and also gathers content for Yahoo News, Finance, and Sports.

LY Corporation (Yahoo! JAPAN) · Search engines

Web crawler operated by Yahoo! JAPAN (LY Corporation) to collect pages for its Japanese search index and related services. It identifies itself with Y!J-prefixed user-agent tokens and follows the Robots Exclusion Protocol.

Yandex · Search engines

Yandex's blog-search robot. Yandex's robot table documents it as the blog search robot that indexes post comments, and lists it as following robots.txt directives.

Yandex · Search engines

Yandex's indexing robot for content in languages other than Russian, feeding the international yandex.com search index. Yandex's robot table documents that it can index content when there is no explicit robot-specific restriction, and lists it among the robots that do not follow the general robots.txt rules written for arbitrary robots.

Yandex · Search engines

Yandex robot that loads resources such as JavaScript and CSS needed to render pages during crawling for Yandex Search.

Yandex · Search engines

Yandex's favicon fetcher. Yandex's robot table documents it as downloading a site's favicon file for display in search results, and lists it as one of the Yandex robots that do not follow robots.txt directives.

Yandex · Search engines

A Yandex robot that Yandex's robot table describes as the mobile devices robot, listed alongside YandexImages and recorded as following the general robots.txt rules.

Yandex · Search engines

Yandex's image-indexing robot, which fetches images so they can be displayed in Yandex Images. Yandex documents it as a separate robot from YandexBot and lists it as following robots.txt directives.

Yandex · Search engines

The robot used by Yandex Market, Yandex's shopping service. Yandex's robot table lists two versions with different robots.txt behaviour — version 1.0 follows the general rules and version 2.0 does not — so this entry records the more permissive of the two.

Yandex · Search engines

Yandex's multimedia-indexing robot. Yandex's robot table documents it as a separate robot from YandexBot, describes it as indexing multimedia data, and lists it as following robots.txt directives.

Yandex · Search engines

Yandex's mobile-layout robot, which fetches pages to determine whether their layout is suitable for mobile devices. It advertises an iPhone Safari prefix followed by its own YandexMobileBot token, and Yandex lists it among the robots that do not follow the general robots.txt rules written for arbitrary robots.

Yandex · Search engines

The Yandex information card robot, which fetches pages used to build the object cards Yandex shows in search results. Yandex's robot table records that it follows the general robots.txt rules.

Yandex · Search engines

The Yandex information card robot that downloads dynamic data, listed separately from YandexOntoDB. Yandex's robot table records that it does not follow the general robots.txt rules.

Yandex · Search engines

Yandex's screenshot robots, which fetch a page so a picture of it can be taken. Yandex documents two of them: YandexScreenshotBot for the desktop rendering and YandexMobileScreenShotBot for the mobile one. Neither follows the general robots.txt rules.

Yandex · Search engines

Yandex's sitelinks checker, which fetches the pages a site offers as sitelinks in search results to confirm they are still available. Yandex's robot table records it as taking the general robots.txt rules into account.

Yandex · Search engines

The Yandex Business robot, which fetches pages for Yandex's business directory listings. Yandex's robot table records that it follows the general robots.txt rules.

Yandex · Search engines

The robot of Yandex Verticals, the classifieds group that covers Auto.ru, Yandex Realty, Yandex Jobs and Yandex Reviews. Yandex's robot table records it as taking the general robots.txt rules into account.

Yandex · Search engines

The Yandex search verticals robot, listed separately from YandexVerticals in Yandex's robot table. Yandex records that it follows the general robots.txt rules.

Yandex · Search engines

Yandex's video-indexing robot, which fetches pages and video content for display in Yandex video search. Yandex documents it as a separate robot from YandexBot and lists it as following robots.txt directives.

Yandex · Search engines

A second Yandex robot that indexes videos for display in Yandex video search, listed separately from YandexVideo. Unlike YandexVideo, Yandex's robot table records that it does not take the general robots.txt rules into account.

Yandex · Search engines

Yandex's primary web crawler, fetching pages for Yandex Search indexing.

Allen Institute for AI · AI crawlers

The Allen Institute for AI's crawler, used to build open training datasets such as Dolma. It crawls selected domains for web content that is then used to train open language models. Ai2 runs it under two user agents, a plain AI2Bot and an Ai2Bot-Dolma variant that names the dataset, both pointing at the same crawling notice. Ai2 publishes no IP ranges, ASN, or reverse-DNS pattern, and its crawling notice offers user-agent filtering as the only documented control.

Aleksma Ai, Inc. (Lyrenth) · AI crawlers

The crawler behind Lyrenth, an index of the public web that serves each page to software agents as a single structured "AIDocument". The operator states the crawler is an indexer and not a model trainer, that it implements RFC 9309 robots.txt including Crawl-delay, Sitemap and HTTP 429/503 backoff, that it fails closed when robots.txt cannot be read, and that it fetches at most once every two seconds per domain across the whole fleet. The bot page offers three independent ways to confirm a request: a published address feed, Web Bot Auth request signatures, and forward-confirmed reverse DNS under lyrenth.com.

Amazon · AI crawlers

The web crawler AWS runs for Amazon Bedrock Knowledge Bases, which fetches the site URLs an AWS customer has configured as a data source and ingests the pages into that customer's knowledge base. Like Google-CloudVertexBot it is customer-directed but operator-run: the requests come from Amazon's infrastructure, and AWS requires the customer to confirm they are authorised to crawl the domains they supply. AWS documents that the crawler follows robots.txt per RFC 9309 and that it defaults to disallow when a site serves no robots.txt at all.

Amazon · AI crawlers

The web crawler AWS runs for Amazon Quick, which fetches the website content an AWS customer has configured as a data source and indexes it into that customer's knowledge base. Like the Bedrock and Q Business crawlers it is customer-directed but operator-run. AWS documents that it respects the robots.txt protocol, honours user-agent, allow, disallow and crawl-delay fields, reads robots.txt per host, and that it crawls anyway when robots.txt cannot be fetched because of blocking, parsing errors or timeouts.

Amazon · AI crawlers

Amazon's web crawler used to improve Amazon's products and services, including training AI models. Amazon publishes the crawler's source addresses on its own bot page as a dated snapshot; they are recorded here verbatim.

ByteDance · AI crawlers

ByteDance's web crawler, used to gather training data for its AI models. ByteDance publishes no IP ranges, ASN, or reverse-DNS pattern for it.

Common Crawl Foundation · AI crawlers

Common Crawl's crawler that builds the freely available Common Crawl web archive, widely reused as AI training data by third parties.

Anthropic · AI crawlers

Anthropic's crawler that indexes pages to improve search-style answer quality in Claude products, distinct from ClaudeBot's training-data crawl.

Anthropic · AI crawlers

Anthropic's web crawler collecting publicly available web data for training Claude models. Anthropic publishes its crawler source IPs as a machine-readable feed.

Cloudflare · AI crawlers

The crawler behind Cloudflare AI Search, which indexes website content so it can be searched. Cloudflare documents that it only crawls a website the customer owns — the domain must exist in the same Cloudflare account and be selected as an AI Search data source.

Cloudflare · AI crawlers

The crawler behind the /crawl endpoint of Cloudflare's Browser Run (Browser Rendering) developer product, which crawls third-party websites on behalf of Cloudflare customers building applications on the platform. Its user agent is not configurable, and every request is signed with Web Bot Auth HTTP message signatures that site owners can verify against Cloudflare's published key directory.

Upstash · AI crawlers

Context7's documentation crawler, an Upstash project. It reads public documentation pages so that AI coding agents can retrieve and cite them, and the operator states the content is used for AI search with links back to the source rather than for model training. Requests are signed with HTTP Message Signatures (RFC 9421) under the Web Bot Auth scheme.

Diffbot · AI crawlers

Diffbot's general-purpose web crawler, used to build its Knowledge Graph and power its structured-extraction APIs. Diffbot documents robots.txt behavior but publishes no IP list, ASN, or reverse-DNS pattern.

DigitalOcean, LLC · AI crawlers

The crawler DigitalOcean runs when a customer adds a website URL as a data source for a DigitalOcean Knowledge Base, which indexes the content so it can power retrieval-augmented generation for that customer's own agents. The documentation states the crawler indexes up to 5,500 pages per job and skips inaccessible or disallowed links to keep indexing jobs bounded, follows HTML links depending on the behaviour the customer selects, indexes text and certain image types, and ignores videos and navigation links.

Firecrawl · AI crawlers

The fetcher behind Firecrawl's scraping and crawling API, which retrieves web pages on behalf of the developers and AI agents calling that API. Firecrawl's crawl endpoint honours robots.txt by default; ignoring it is an enterprise option that must be enabled by the operator. Firecrawl documents that it does not use a fixed set of outbound IP addresses.

ProRata.ai · AI crawlers

ProRata.ai's crawler for Gist.ai. It systematically fetches licensed and publicly accessible content to power the retrieval-augmented generation model behind Gist Answers, following internal and external links to discover new and changed pages, and may also download the CSS and JavaScript a page needs for accurate content extraction. ProRata documents that the crawler only visits content that is either freely accessible without credentials or explicitly licensed, spaces out its requests, and publishes the two addresses it crawls from, one for production and one for testing.

Google · AI crawlers

Google Cloud crawler that fetches sites on the site owners' request when building Vertex AI Agents. Crawling preferences addressed to its user agent have no effect on Google Search or other Google products.

OpenAI · AI crawlers

OpenAI's web crawler that gathers publicly available data used to train OpenAI's models.

Hive · AI crawlers

ImageSift's crawler, operated by Hive. It scrapes publicly available images across the web to support Hive's ImageSift reverse-image-search and web intelligence products. Standard robots.txt directives are respected.

Moonshot AI · AI crawlers

Moonshot AI's search crawler for Kimi, which analyses pages for relevance and builds the index behind Kimi's search features. Blocking it removes a site from Kimi search results. Listed separately from KimiBot because Moonshot documents it as an independently configurable crawler with its own robots.txt token and its own published address list.

Moonshot AI · AI crawlers

Moonshot AI's crawler for the Kimi assistant, gathering content that may be used to train Kimi's foundation models. Moonshot documents robots.txt control per crawler and publishes a machine-readable list of the addresses its crawlers use, while noting the ranges are dynamic and may change.

Klaviyo · AI crawlers

The crawler behind Klaviyo's Kai Customer Agent. It fetches publicly available pages only from the domains and URLs a Klaviyo customer has explicitly connected to their own account — their store plus any help centre, documentation portal, blog or news source they add — and indexes them so Klaviyo can answer shopper questions and generate content. Klaviyo documents that it does not bypass authentication, paywalls or access controls, that it slows down on 429 and 503 responses with Retry-After, and that it deletes indexed content within days once a page is removed or disallowed.

Leipzig University Natural Language Processing Group · AI crawlers

The LCC crawler is operated by Leipzig University to collect web text for the Leipzig Corpora Collection, a set of linguistic corpora used in natural language processing research.

Liner · AI crawlers

Web crawler operated by Liner, an AI search and research assistant, which gathers pages from the public web and indexes them so Liner's search and research agents can retrieve and cite them. Liner's crawler page documents the user agent, states that access is controlled through robots.txt, and gives worked examples for both a full disallow and path-scoped allow rules. Liner also publishes the crawler's addresses as a JSON feed under the heading "IP address ranges", noting the pool may change over time.

Linkup · AI crawlers

Linkup's web crawler. It discovers and retrieves publicly accessible pages to build and maintain the Linkup search index, which powers search and web grounding for AI applications. Linkup publishes the CIDR ranges its crawler originates from and asks site owners to check both the source IP and the user-agent token, since user agents can be spoofed.

MaCoCu project (Jožef Stefan Institute) · AI crawlers

Crawler for the CEF-funded MaCoCu project, which collects, curates and enriches monolingual and parallel web text to build language corpora for under-resourced languages. The project publishes no IP ranges, so requests cannot be verified.

Meta · AI crawlers

Meta's crawler for AI training data and content indexing across Meta products. Verified by ASN lookup (AS32934); Meta publishes no IP feed.

Meta · AI crawlers

Meta's crawler that navigates the web to improve the quality of Meta AI search results, analysing page content for relevance and accuracy in Meta AI responses. Verified by ASN lookup (AS32934); Meta publishes no IP feed.

Mistral AI · AI crawlers

Mistral AI's indexing crawler. It crawls the web automatically to build the index behind Mistral search, which answers user questions in Vibe. Mistral documents that content collected by this crawler is not used for generative AI training of any kind, and publishes the crawler's addresses as a JSON feed.

Mistral AI · AI crawlers

Mistral AI's training crawler. It collects web content to help build the datasets used to train Mistral's generative AI models, and is documented as separate from both the search-index crawler and the user-triggered fetcher. Mistral states webmasters can disallow this user agent in robots.txt.

National Institute of Information and Communications Technology (NICT) · AI crawlers

The web crawler of the Universal Communication Research Institute at Japan's National Institute of Information and Communications Technology. It collects web pages for NICT's research and development of information processing technologies including multilingual translation, information analysis and artificial intelligence, and NICT states that collected information and research products may be provided to third parties for joint or further research. The crawler page documents a load policy that caps connections per unit of time across a host's several addresses or hostnames.

Noah Wire · AI crawlers

The crawler powering Noah Wire's search and research engine for journalists and editors. It scans publicly accessible news sources — press releases, newsrooms, blogs and announcement feeds — and the operator states it does not scrape, store or republish full articles, producing linked summaries instead. The bot page says requests are signed with HTTP Message Signatures, but publishes no key directory URL, so no verification recipe is recorded.

OpenAI · AI crawlers

OpenAI's crawler that indexes pages to power search results inside ChatGPT. It is distinct from GPTBot and is not used to gather training data.

Webz.io · AI crawlers

Webz.io's legacy crawler user agent (formerly "Omgilibot"), used to collect web, forum, and news content for its data feeds. Webz.io's current public documentation describes successor crawlers ("webzio" / "webzio-extended") and no longer documents this UA string or any IP verification method for it.

Perplexity · AI crawlers

Perplexity's crawler that indexes pages to surface and link websites in Perplexity search results. Perplexity states it is not used to train models.

Qualified · AI crawlers

Qualified's crawler. It crawls customer websites to power Qualified's AI products, such as its AI SDR, during initial setup, on major feature releases and on a schedule the customer's Qualified representative sets. The operator publishes the three source addresses its crawlers request from and asks site owners to allow-list them.

Reflection AI · AI crawlers

The web crawler of Reflection AI, which visits publicly available pages to help build and improve the company's open models. Its bot page is unusually complete for an AI crawler: it names the single user-agent token the crawler uses, tells site owners to match on that token rather than the whole string, publishes the crawler's address ranges as a JSON feed, and states that the addresses reverse-resolve under the operator's own domain.

SB Intuitions Corp. · AI crawlers

Crawler operated by SB Intuitions (a SoftBank AI subsidiary) that collects web pages for AI development and information analysis, including training of its Sarashina language models. SB Intuitions publishes no IP ranges, so requests cannot be verified.

ScalePost Corporation · AI crawlers

ScalePost's on-demand fetcher. The operator documents that it fetches a publicly available page with a single GET request when a customer asks for an analysis of how that content is being crawled, cited and referenced by AI systems, that it does not execute JavaScript, submit forms or follow links out of the page body, and that it retains only extracted metadata. ScalePost states it fetches robots.txt before any URL on a host and obeys both its own token and the wildcard, honours Crawl-Delay across its parallel workers, and backs off on Retry-After.

Parallel Web Systems · AI crawlers

Parallel's indexing crawler, which collects publicly available web content to build and maintain the search index behind Parallel's web APIs. The operator documents the robots.txt token ShapBot and publishes the crawler's addresses as a JSON feed.

Webz.io Ltd. · AI crawlers

Second crawler in Webz.io's crawler pair, which performs ethical validation on the data collected by Webzio and tags it as usable or not usable for AI and machine-learning training. Webz.io publishes no IP ranges.

Yandex · AI crawlers

Yandex's robot for processing robots.txt so that site owners can keep already indexed page content out of the AI responses shown in Yandex Search. Yandex's robot table states it applies to pages that have already been indexed by the primary crawler and that it makes no indexing requests of its own.

You.com · AI crawlers

You.com's crawler that indexes pages for its AI-powered search product. You.com documents a dedicated IP range, a reverse-DNS pattern, and support for signed-request verification via Web Bot Auth.

Aleksma Ai, Inc. (Lyrenth) · AI assistants

Lyrenth's on-demand fetcher. It identifies itself separately from the AIWebIndex crawler when a customer asks the index for a URL that is not in it yet, so site owners can tell scheduled crawling and on-demand fetches apart in their logs. The operator states both user agents honour robots.txt the same way, and the crawler's address feed covers this user agent too.

Amazon · AI assistants

Amazon's on-demand fetcher supporting user actions, such as responding to Alexa queries that need up-to-date information; Amazon states it does not crawl content for generative AI model training. Amazon publishes its live-crawl source addresses on a dedicated page as a dated snapshot; they are recorded here verbatim.

OpenAI · AI assistants

OpenAI's user-triggered fetcher, used when a ChatGPT user or a GPT Action asks the assistant to visit a specific page. It does not crawl autonomously.

Anthropic · AI assistants

Anthropic's user-triggered fetcher, used when a person asks Claude to visit a specific web page (e.g. via tool use in a conversation).

Diffbot · AI assistants

Diffbot's user-triggered fetcher, which Diffbot documents as being used by requests originating on behalf of a human user browsing a URL through Diffbot software, in response to that person's input. Diffbot's robots.txt guidance covers its web crawls; it makes no separate statement about these user-initiated fetches, and publishes no IP list, ASN or reverse-DNS pattern.

DuckDuckGo · AI assistants

DuckDuckGo's real-time fetcher for DuckDuckGo Search's AI-assisted answers, which cite their sources. DuckDuckGo states the data is not used to train AI models, and publishes a machine-readable list of the bot's IP addresses.

Google · AI assistants

The fetcher used by AI agents hosted on Google infrastructure to navigate the web and perform actions on behalf of a user who asked for them. Google publishes a dedicated IP range list for it and signs a subset of its requests with Web Bot Auth under the agent.bot.goog identity. As a user-triggered fetcher it generally ignores robots.txt rules.

Google · AI assistants

Google tools and product features that fetch a specific page because an end user asked for it (Google Read Aloud, Site Verifier, Gemini Notebook, Chrome Web Store, Google Messages, Pinpoint, Publisher Center, and similar), rather than autonomous crawling for search indexing. They generally ignore robots.txt because a human requested the fetch.

Moonshot AI · AI assistants

Moonshot AI's user-triggered fetcher for Kimi, which retrieves a page when a person asks Kimi to summarise it or answers a question needing live web retrieval. Moonshot states it is not used for automated bulk crawling and that, because the actions are user-triggered, robots.txt rules may not directly apply.

Meta · AI assistants

Meta's on-demand fetcher that retrieves a single link at a user's request to support agentic AI features (e.g. an AI assistant navigating a page a user asked about), rather than broad indexing. Because its fetches are requested by a user, Meta documents that this crawler may bypass robots.txt rules. Verified by ASN lookup (AS32934).

Mistral AI · AI assistants

Mistral AI's user-triggered fetcher. When someone asks Vibe a question, it may visit a web page to help answer and link to the source in its response. Mistral documents that it is not used for automatic crawling of the web, nor to collect content for generative AI training, and publishes the addresses it fetches from.

Mozilla (Tabstack) · AI assistants

Fetcher for Tabstack, Mozilla's developer-facing platform for programmatic, AI-driven interaction with web content. Every request carries a dedicated user agent, and the operator documents that Tabstack respects robots.txt rules addressed to it, stops immediately on a disallowed path, fails fast rather than retrying, and caches robots.txt results to reduce follow-up requests.

Perplexity · AI assistants

Perplexity's user-triggered fetcher, used when a user's question requires visiting a specific web page to produce an accurate answer.

SB Intuitions Corp. · AI assistants

SB Intuitions' user-triggered fetcher. The operator documents that when someone asks its Sarashina service a question, this agent visits websites on that person's behalf to improve the quality of the search results it reasons over, that answers may include links to the pages it visited, and that what it retrieves is not used for AI development. That last point distinguishes it from SBIntuitionsBot, which the same page says is used for AI development and information analysis.

Parallel Web Systems · AI assistants

Parallel's user-triggered fetcher. It identifies itself when Parallel accesses content on behalf of a user, giving content owners visibility into user-initiated requests. The operator documents that it is not used for automatic crawling and is intended to provide visibility rather than to be managed through robots.txt, which governs ShapBot instead.

Yandex · AI assistants

A Yandex robot that proxies user actions taken on Yandex services: it sends requests in response to button clicks and downloads pages for online translation, so its requests to a site follow from something a person just did. It does not follow the general robots.txt rules.

Buffer · Social previews

The crawler behind link previews in Buffer's social-media management product. When someone shares a link in Buffer it fetches the page's public metadata — title, description, thumbnail and Open Graph or Twitter Card data — so the poster can see how the post will look. Buffer publishes its egress addresses both on that page and as a machine-readable JSON document, and sends X-BufferLinkPreviewBot headers alongside the user agent.

Deskyo · Social previews

Link-preview fetcher for Deskyo, a service that lets its members save web pages. The operator's bot page documents that Deskyo must display information tied to a saved page — its title, description and favicon — which is what this fetcher retrieves, and gives a legal contact for questions about links saved by a member. The page states no robots.txt policy and publishes no addresses.

Discord · Social previews

Discord's crawler that fetches shared links to render embeds in chat. Discord's own docs only specify the User-Agent format required of API clients, not this embed fetcher; no IP ranges, ASN, or rDNS are published.

Embedly · Social previews

Embedly's fetcher, used to generate rich embeds and previews of shared URLs for its customers' apps and sites. Embedly's FAQ publishes a fixed, small set of source IPs rather than a live CIDR feed.

Meta · Social previews

Meta's crawler that fetches shared links from Facebook, Instagram, and Messenger to generate rich link previews. Meta documents one exception to its robots.txt compliance: the crawler might bypass robots.txt when performing security or integrity checks, such as looking for malware or malicious content. Verified by ASN lookup (AS32934); Meta's own docs describe IP-based allowlisting but publish no IP feed.

Fastmail · Social previews

Fastmail's server-side fetcher, used on behalf of its mail users. It proxies images embedded in messages so that no browser cookies, browser user agent or user IP address reach the remote server, and it fetches subscribed ICS and remote CalDAV calendars. Fastmail documents that every user agent it sends contains the string FastmailUA.

Hatena Co., Ltd. · Social previews

The fetchers Hatena's services use to collect information from pages its users link to. Hatena-Favicon identifies a page's favicon, Hatena::Scissors retrieves image thumbnails, HatenaBookmark fetches article information for Hatena Bookmark, Hatena Star associates stars with pages, and Hatena Antenna fetches update differences for the pages a user is monitoring.

Hatena Co., Ltd. · Social previews

Hatena Blog's fetcher. It retrieves a linked page when a blog author asks for it: the :title option of Hatena's URL notation pulls the page title, the :embed option collects the title, summary and favicon used to render a blog card, and the blog import feature fetches images so they can be re-uploaded to Hatena Fotolife.

Iframely · Social previews

Iframely's fetcher, used to generate embeds and link previews of shared URLs for its customers' apps and sites. Iframely publishes live IPv4/IPv6 address lists and documents reverse-DNS resolution to its own domain.

LinkedIn · Social previews

LinkedIn's crawler that fetches shared links to build post link previews. LinkedIn's own robots.txt names the "LinkedInBot" token, but LinkedIn publishes no dedicated bot-documentation page, IP list, ASN, or reverse-DNS pattern for this crawler.

ListGem · Social previews

ListGem's link unfurler. When someone adds a URL to a ListGem list it fetches that single page to read its public metadata; the operator documents that it does no bulk crawling or link spidering. Every request carries an HTTP Message Signature (RFC 9421, Web Bot Auth profile, Ed25519) and the public key directory is published so requests can be verified.

Livelap · Social previews

LivelapBot is the crawler for Livelap, a content discovery app. It fetches pages shared on social media and crawls RSS feeds on a schedule, indexing HTML and media meta tags to build link previews shown in the Livelap app.

Mastodon gGmbH · Social previews

The generic link-preview fetcher built into Mastodon server software (FetchLinkCardService, using the http.rb gem), run independently by every self-hosted instance. There is no single operator, IP range, or ASN to verify against — traffic can originate from any of thousands of instances.

Microsoft · Social previews

Microsoft fetcher, listed among Bing's crawlers, that retrieves pages to generate link previews for Microsoft products and services.

Naver · Social previews

Naver's summary-information robot. When a user attaches or shares a link in Naver Blog, Cafe, SmartEditor and similar services, Blueno fetches the linked page to build the preview, reading Open Graph or Twitter Cards markup first and falling back to summarising the document. Naver documents that it runs on user-initiated requests rather than as a search crawler, caches each summary for a period to avoid repeat visits, and can be confirmed by a forward-confirmed reverse DNS lookup. Its user agent carries a facebookexternalhit prefix, so a pattern written for Meta's link preview fetcher will also match it.

PagePeeker SRL · Social previews

PagePeeker's thumbnailing robot. It captures a screenshot of a page when one of PagePeeker's customers requests a thumbnail for it, refetching a given site at most once every five to seven days.

Pinterest · Social previews

Pinterest's crawler, used both for general web indexing and for validating Pin metadata/broken links. Pinterest documents a reverse-DNS verification method rather than a fixed IP list, since only its US-based traffic uses a stable range.

Reddit · Social previews

Reddit's on-demand fetcher that retrieves a shared URL's metadata to build the link preview shown on a post. Reddit's robots.txt now disallows all crawling and documents no user agent, IP list, or verification method for this fetcher.

Slack · Social previews

Slack's image-proxying fetcher, which retrieves and caches images posted in channels while stripping referrer data. Slack documents the User-Agent string but publishes no IP ranges or other verification method.

Slack · Social previews

Slack's crawler that expands links posted in channels into rich previews by reading oEmbed, Twitter Card, and Open Graph metadata. Slack documents the User-Agent strings but publishes no IP ranges or other verification method.

Snap Inc. · Social previews

Snapchat's link-preview fetcher, which scans the HTML of URLs shared in Snapchat chats to build preview cards from Open Graph or Twitter Card tags. Snap documents that responses are cached for 30 minutes to reduce traffic; no IP ranges or robots.txt behavior are documented.

Start.me · Social previews

Start.me's fetcher. When a user adds a link or an RSS widget to their Start.me start page, this bot retrieves three things from the target site: the page title, the site's favicon, and the contents of any RSS feed the site offers.

Telegram · Social previews

Telegram's fetcher for generating link previews in chats. Telegram's docs document static CIDR ranges only for inbound webhook delivery to bot servers, not for this outbound preview fetcher, so no IP verification is attached here.

X · Social previews

X (formerly Twitter)'s crawler that fetches shared links to render Card previews. X's own developer docs give no IP ranges, ASN, or reverse-DNS verification method; the commonly cited "*.twttr.com" rDNS check and aggregate IP ranges trace only to community write-ups, not official docs.

VK · Social previews

vkShare is VK's link-preview fetcher. When a user shares a URL on the VK social network, it retrieves the target page to build a preview (title, description, and image) for the shared post.

Meta · Social previews

WhatsApp's fetcher that requests a shared URL to build the link preview shown in chat. Meta documents the request's User-Agent format but no IP feed; verified by ASN lookup (AS32934), same as Meta's other crawlers.

Yahoo · Social previews

Yahoo Mail's content-fetch proxy. It retrieves images and other assets embedded in the URLs of messages sent to Yahoo Mail users, so that the content is served through Yahoo rather than fetched directly by the recipient's browser. Yahoo documents the user agent and publishes the address ranges the proxy fetches from.

Yahoo · Social previews

Yahoo Link Preview fetches a page when a Yahoo Mail user includes its URL in an email message, generating a thumbnail, title and description preview. It fetches only user-referenced pages rather than crawling the web.

Yandex · Social previews

A Yandex robot that collects the data used to generate page previews. Yandex's robot table lists it separately from the indexing robots and records that it does not follow the general robots.txt rules.

SmartBear Software · Monitoring

SmartBear's AlertSite runs customer-configured availability and transaction monitors against the sites and APIs their owners point it at, from a published set of monitoring locations. SmartBear tells site owners to filter the traffic by looking for "AlertSite" in the user agent string or by the monitoring location addresses. Some mobile-network locations use dynamic carrier addresses that are not in the published list.

Awario · Monitoring

Awario's social-listening and media-monitoring crawlers, which fetch web pages and RSS feeds to find mentions of the brands and keywords its customers track. Awario states the bots respect robots.txt and honour Crawl-delay, and explicitly asks site owners not to identify them by IP because it uses no consecutive IP blocks.

Microsoft · Monitoring

Synthetic availability-test agents from Microsoft Azure Application Insights that send recurring HTTP requests to user-configured URLs to monitor uptime and responsiveness from multiple Azure regions.

Better Stack · Monitoring

Better Stack's uptime-monitoring probes, which request customer URLs on a schedule from a published, frequently-changing list of monitoring IPs.

BrandVerity · Monitoring

BrandVerity is a brand-protection service that crawls websites and paid search landing pages to detect trademark and affiliate-compliance violations on behalf of its clients.

Brandwatch · Monitoring

magpie-crawler is Brandwatch's web crawler. It downloads publicly available blog, forum, news, and social-media pages to be indexed and analysed by Brandwatch's social-listening platform. It respects robots.txt.

Checkly · Monitoring

Synthetic-monitoring service that runs API and browser checks against customer endpoints from a published, fixed set of static outbound IPs.

Cloudflare · Monitoring

Cloudflare's support-diagnostics fetcher. Cloudflare documents that requests with this user agent are triggered when Cloudflare Support Engineers perform error checks, and by the continuous monitoring that raises alerts in the Cloudflare dashboard.

Cloudflare · Monitoring

Cloudflare's Health Checks probes, which monitor customer origin servers from Cloudflare data centers in regions the customer selects. Cloudflare documents a fixed User-Agent format embedding the first 16 characters of the health check ID, and recommends matching on it; its health-checks docs do not tie probe traffic to the published Cloudflare IP lists, so no CIDR recipe is attached.

Cloudflare · Monitoring

Cloudflare's prefetch bot, an Enterprise speed-optimization feature that pre-populates the CDN cache with content a visitor is likely to request next, using a site-owner-supplied manifest of URLs. Requests carry the CloudFlare-Prefetch user-agent.

Cloudflare · Monitoring

Cloudflare's Load Balancing monitor, which sends health-check requests to customer origin pools at regular intervals to evaluate endpoint health. It embeds the load-balancer pool id in a fixed User-Agent.

Cookiebot (Usercentrics) · Monitoring

Cookiebot's cookie-consent compliance scanner. It crawls the domains its customers have registered, on a roughly monthly schedule, to detect the cookies and tracking technologies in use and generate a cookie declaration. Cookiebot documents that scans run only from a fixed pool of addresses.

CookieHub · Monitoring

CookieHub's cookie-consent compliance scanner. It crawls customer domains on a roughly monthly schedule to detect cookies and tracking technologies in use and generate a compliance report.

Datadog · Monitoring

Datadog Synthetic Monitoring probes that run customer-configured API and browser tests against endpoints. API tests send a "Datadog/Synthetics" User-Agent; browser tests append "DatadogSynthetics" to a browser UA. Datadog publishes probe IP ranges as a machine-readable feed at ip-ranges.datadoghq.com/synthetics.json.

DLC Websites · Monitoring

Dead Link Checker is a hosted service that crawls a submitted website to find broken links and report them, with paid tiers that run automated periodic scans and email the results.

DubBot · Monitoring

The crawler behind DubBot's web-governance platform. It inventories a customer's own website by following every link from a supplied URL and checks the pages for accessibility, broken links, spelling and content-policy problems. DubBot runs it from AWS on static IP addresses that the operator publishes for allowlisting.

Dynatrace · Monitoring

Dynatrace's synthetic monitoring runs browser and HTTP checks against sites its customers configure. Dynatrace always appends a RuxitSynthetic token to the user agent — even when the customer sets a custom one — so that synthetic traffic can be identified in server logs. Checks are user-configured, not crawling, so robots.txt is not part of the documented behaviour.

Fed Reporter Inc. · Monitoring

Scheduled data-retrieval bot that pulls publicly available banking-institution records from the FFIEC National Information Center for Fed Reporter's internal research dashboards. Its documentation states it targets only public data, uses no CAPTCHA bypass, and runs a separate testing agent off-peak.

GreenWave Online Inc. · Monitoring

The website-monitoring robot of the FreeWebMonitoring service. It only checks URLs that registered members have submitted, and the operator documents that all checks originate from a single server address. GreenWave Online also warns that an older 0.1 agent name is forged by an unrelated scanner.

Freshworks · Monitoring

Freshworks' free uptime-monitoring service, which checks customer sites from a small, fixed set of documented monitoring-location IPs rather than a live feed.

Ghost Inspector · Monitoring

Ghost Inspector runs customer-authored browser tests against the sites their owners point it at, from a fixed pool of cloud test runners in 18 regions. Its runners append the literal string "Ghost Inspector" to the browser user agent they run, and the operator publishes the full list of test-runner addresses so site owners can filter the traffic out of their analytics.

Gnowit · Monitoring

Gnowit is an Ottawa-based media and government monitoring company whose crawler continuously fetches news, government, and other public web sources to power real-time monitoring, summarization, and analytics for its clients.

Grafana Labs · Monitoring

Grafana Cloud Synthetic Monitoring public probes, which run customer-configured checks (HTTP, browser, and protocol) from Grafana-run locations. HTTP checks send a documented synthetic-monitoring-agent User-Agent by default (customers can override it). Grafana publishes probe source IPs as a machine-readable feed at allowlists.grafana.com/synthetics.

HetrixTools · Monitoring

Uptime and blacklist monitoring service that probes customer sites from a published list of monitoring-node IPs.

Hydrozen · Monitoring

Hydrozen.io is an uptime and website monitoring service; its checker fetches monitored endpoints from a documented, published set of IP addresses.

Hypefactors · Monitoring

Buck is Hypefactors' media-monitoring crawler. It discovers and indexes web pages by following links, gathering publicly available content for the company's media-monitoring platform. It identifies itself and respects robots.txt.

Automattic · Monitoring

Jetpack's site uptime monitor, operated by Automattic for WordPress sites that enable Jetpack Monitor. It sends an HTTP HEAD request to the site's homepage every five minutes; if that check fails, three further servers in other locations re-check before the site is reported down.

Hatena Co., Ltd. · Monitoring

The prober behind Mackerel's external URL monitoring. Mackerel is Hatena's hosted server-monitoring service; when a customer configures an external URL monitor, Mackerel requests that URL from its own servers on a schedule and alerts the customer when the response fails their conditions. It checks only the URLs a customer has configured and does not crawl or follow links. Hatena publishes the address ranges Mackerel sends from as a machine-readable feed covering both webhook notifications and these monitoring requests.

MediaMonitoringBot · Monitoring

Crawler behind MediaMonitoringBot, a Ukrainian media-monitoring service that indexes news and media publisher pages and delivers mention alerts to PR and communications subscribers. The operator documents that it strictly respects robots.txt allow and disallow rules, publishes its main crawler addresses, and documents a reverse-DNS suffix for verification.

Determ · Monitoring

Mediatoolkitbot is the web crawler operated by Determ (formerly Mediatoolkit), a media monitoring platform. It fetches publicly available web content to match brand mentions and topics that Determ's customers track.

Neticle Technologies · Monitoring

Neticle's in-house crawler for its social and online media-monitoring service. It continuously fetches newly published web content so Neticle can score sentiment around the keywords, brands and products its customers track.

New Relic · Monitoring

New Relic's synthetic-monitoring minions, which probe customer-configured URLs from public minion IPs. New Relic documents an X-Abuse-Info request header identifying the monitor and account; logs also record a NewRelicbot User-Agent. New Relic states the published ranges "are reserved for use by New Relic and cannot be used by anyone else". The published IP list is a JSON object keyed by location, which this directory's feed formats cannot consume, so the ranges are recorded statically instead.

ObservePoint · Monitoring

ObservePoint's web-governance platform runs customer-configured audits and user journeys against their own sites to validate analytics and marketing tags. Requests come from a published set of static addresses that the operator states are used by ObservePoint alone, and most of its browser-based user agents carry the literal name "ObservePoint".

Oh Dear · Monitoring

Website monitoring service that crawls the sites its customers monitor, checking for broken links, mixed content and sitemap validity, and that also verifies outgoing links found on a customer's site, which is why it reaches pages on third-party domains. Oh Dear documents that it throttles itself and follows robots.txt by default, honouring rules written for the OhDear token, though a customer can switch "Respect robots.txt" off for their own monitor. Every user agent it sends starts with OhDear.app and links the feature page explaining the check, and Oh Dear publishes the fixed list of monitoring servers it runs from.

Osano · Monitoring

Osano's web retrieval agent for Compliance Check, which loads pages on a customer's own monitored hostname to assess cookie, consent and privacy-signal behavior. Requests are signed with Ed25519 HTTP Message Signatures (RFC 9421) under the Web Bot Auth scheme with the Signature-Agent https://bots.osano.com, so they can be verified independently of source IP.

SolarWinds · Monitoring

Uptime and page-speed monitoring service that probes customer sites from a published list of probe-server IPs. Owned by SolarWinds.

Pulsetic · Monitoring

Uptime monitoring service that probes the sites its customers configure. Primary checks come from the monitoring engine chosen in the customer's account, and when downtime is detected Pulsetic sends confirmation requests from remote nodes in other countries, so a single monitored site can be hit from several of its addresses. Pulsetic publishes both its IPv4 and IPv6 node lists for allowlisting and documents the user agent it sends.

Amazon Web Services · Monitoring

Amazon Route 53 health checkers, which probe customer-configured endpoints from AWS data centers worldwide to drive DNS failover. AWS publishes the checker source ranges only inside the general ip-ranges.amazonaws.com feed, tagged with the ROUTE53_HEALTHCHECKS service, which this directory's feed formats cannot filter on, so the ranges are recorded here as a snapshot instead. AWS documents that it rarely changes them.

Scrunch AI · Monitoring

Fetcher for Scrunch AI's brand-visibility product. Scrunch documents that it accesses customer websites to diagnose issues with their content and configuration, and that it does not proactively crawl or spider the general internet — requests are made only in response to customer configuration.

SentiOne · Monitoring

SentiOne's social-listening crawler. It indexes user-generated content for the SentiOne Listen platform, which its operator says analyses over 300,000 domains daily. Robots.txt rules written for "sentibot" are honoured, and Yandex-style reverse-DNS plus a published IP list allow verification.

Sentry · Monitoring

Sentry's uptime monitoring bot, which probes customer-configured URLs on a schedule from Sentry's uptime-check infrastructure and raises alerts on failures. Sentry documents a fixed User-Agent and publishes the current uptime-check IP addresses at a machine-readable endpoint.

Zoho · Monitoring

Zoho's infrastructure and website monitoring service, which probes customer sites from 130+ global monitoring locations published as a machine-readable IP list.

Spectate · Monitoring

Spectate's uptime and performance monitoring agent. It requests the URLs a Spectate customer has added to their monitors, from fixed probe locations, to check that the target is reachable. Spectate publishes the addresses those probes request from as a plain-text list grouped by location.

StatusCake · Monitoring

Uptime and page-speed testing service that requests customer URLs from a global network of test-location servers, published as a machine-readable IP list.

Testomato · Monitoring

Testomato's website-monitoring agent. It downloads pages and resources and submits web forms for the checks its customers configure (uptime, error, SSL, response-time, meta-tag and JSON-LD monitoring), on a schedule the customer sets. Testomato publishes a plain-text list of the addresses its monitoring nodes request from.

Trendiction · Monitoring

Trendiction operates a crawler that fetches public news sites, message boards, and blogs to build a search index and feed media-monitoring and market-research analytics for its clients.

Udger s.r.o. · Monitoring

Udger's link checker, which visits the websites of browser, operating-system and robot vendors to confirm they are still reachable while the company maintains its user-agent database. Udger documents the user-agent string it sends but publishes no allowlist of the addresses it crawls from, and states no robots.txt policy for it.

updown.io · Monitoring

Website uptime-monitoring service that checks customer sites from a published JSON array of node IPs. Documents "updown.io" as the common substring of its requests rather than one fixed User-Agent.

Uptime.com · Monitoring

Uptime.com's synthetic-monitoring probes. They request the URLs its customers configure as checks, from probe servers in many locations, including headless-browser transaction checks that execute JavaScript. Uptime.com publishes its probe IP list only inside the authenticated dashboard, so the user agent is the public identifier.

UptimeRobot · Monitoring

Uptime monitoring service that probes customer sites on a schedule from a published list of monitoring IPs.

WEBSPARK (W.A.R.D.) · Monitoring

The monitoring agent behind W.A.R.D. It requests URLs that users have added to their monitoring list and records the HTTP status code so the user can be notified of downtime, at an average frequency of ten minutes or more per URL. The operator documents that its addresses reverse-resolve under the robot.ward.host suffix.

WatchFor · Monitoring

WatchFor's monitoring probe, which checks that websites, APIs, servers and certificates its customers have configured are healthy, from fixed locations worldwide. It performs only the specific check each monitor defines and does not crawl sites, follow links or harvest content. WatchFor publishes a machine-readable address list and forward-confirmed reverse DNS under probe.watchfor.io so site owners can tell a genuine probe from a spoofed user agent, and also signs every probe with an Ed25519 key under the Web Bot Auth profile of RFC 9421 HTTP message signatures.

Yandex · Monitoring

Yandex's availability checker, which downloads pages from websites in different regions of the world to check whether they are reachable for users. Yandex's robot table records that it makes no indexing requests and does not follow the general robots.txt rules.

Yandex · Monitoring

The robot behind Yandex Metrica, Yandex's web analytics product. Yandex documents several versions of it: one downloads site pages to check their availability, including the landing pages of Yandex Direct ads, and another downloads and caches CSS styles so pages can be rendered in Webvisor. The robot table records that it ignores the directives set for it in robots.txt.

Naver · SEO tools

Naver's advertising crawler. It periodically visits the landing pages advertisers register in Naver Search Ads, reading the HTML, content and meta information so Naver can assess page quality and match ads to relevant queries. Naver states this robot deliberately ignores robots.txt for registered landing pages, and tells site owners to verify it by reverse DNS.

Ahrefs · SEO tools

Ahrefs' on-demand site-auditing crawler, distinct from AhrefsBot, used when Ahrefs customers run the Site Audit tool against their own or a competitor's domain. Shares Ahrefs' published IP-range feed.

Ahrefs · SEO tools

Ahrefs' primary web crawler, powering the backlink and keyword database behind the Ahrefs SEO platform and the Yep search engine. Crawls from publicly published IP ranges with a matching reverse-DNS suffix.

Audigent · SEO tools

Audigent's advertising crawler. The operator documents that it collects only the metadata in the header of an HTML page and does not scrape page body content, and that it must be named explicitly in robots.txt because a wildcard rule does not block it.

Audigent · SEO tools

The broader of the two crawlers Audigent documents on its bot page. Unlike AudigentAdBot, which reads only HTML head metadata, Audigent states that AudigentBot can scrape any content available on the page, from both the header and the body, and that it can be blocked with a wildcard robots.txt rule. Audigent says it crawls only sites where it or its advertiser partners may serve ads.

Audisto GmbH · SEO tools

The crawler behind Audisto, a German SaaS platform for technical website analysis and continuous website monitoring. It fetches the accessible URLs of a site so the product can report crawling and indexation errors, hreflang and canonical problems, and internal link-graph issues. The operator runs two crawlers: an Essential crawler that may crawl any site, and a Full crawler that only crawls sites whose ownership has been verified. The bot reference documents automated overload protection — after 10 error responses within a hundred requests the crawler lengthens its delay and reduces parallelism, and relaxes again once the error rate drops below 5 percent — plus a 30-second total request budget and per-content-type response size caps.

Babbar · SEO tools

Barkrowler is the web crawler operated by Babbar (formerly Exensa). It builds and updates Babbar's graph of the web, which powers the company's SEO and link-analysis tools, and applies a politeness delay between requests.

Bombora, Inc. · SEO tools

BomboraBot is Bombora's web crawler. It classifies the content and topics of web pages that carry Bombora's tags so the company can model B2B purchase intent, visiting each tagged page at most once every 30 days.

Botify · SEO tools

Botify's SEO crawler, used by its SiteCrawler product to analyze enterprise websites for search-indexing insights. Botify publishes the addresses its crawls originate from and recommends site owners allow the full list, which it documents as shared by SiteCrawler, AlertPanel and SpeedWorkers. The crawler's user agent is configurable per crawl, so identity rests on the published ranges rather than the agent string alone.

Bright Data · SEO tools

Bright Data's data-collection crawler, documented as the main collection pipeline for its products, with a 24-hour cache layer to avoid re-downloading the same page. The operator states Brightbot is deliberately transparent — a unique user agent plus a single published source subnet — so its traffic can be separated from user traffic. Its documented control mechanism is Bright Data's own collectors.txt file and Web Master console; the page states no robots.txt behaviour.

BuiltWith Pty Ltd · SEO tools

Crawler for BuiltWith's technology-profiling service, which visits sites and analyses publicly visible markup to determine which web technologies they use. BuiltWith publishes no IP ranges, so requests cannot be verified.

Conductor · SEO tools

Conductor's single web crawler. It reads the HTML of pages on sites its customers track, recording on-page elements such as title tags, header tags and other metadata for Conductor's SEO and search-visibility reporting. Conductor publishes the address range it crawls from and will lower the crawl rate on request.

CINC Corp. (株式会社CINC) · SEO tools

The web crawler operated by CINC, a Japanese data-solutions company, to collect the page data behind its marketing and SEO analytics products. Its documented policy is to fetch page body content, header and HTTP status information and the JS/CSS needed to render a page, then store a rendered screen capture. CINC states that it does not follow advertising links, deletes all cookies between requests, and does not load analytics or ad-measurement tags. No robots.txt policy and no IP ranges are published.

seoClarity · SEO tools

seoClarity's page and site audit crawler. Crawls are triggered on demand by seoClarity clients to analyse pages for technical and content issues, and clients can also schedule daily managed page crawls. The operator documents that it obeys robots.txt and Crawl-delay, and that its addresses are dynamic.

Clearscope · SEO tools

The crawler behind Clearscope, a hosted content-optimisation and SEO platform. The operator describes it as a web scraper that retrieves publicly available content from websites in order to power Clearscope's products, naming Content Grade recommendations and Content Inventory specifically. The support page states the crawler is built to be efficient and respectful and to comply with industry standards, and directs site owners whose WAF or host blocks crawlers to allow its published addresses.

Cocolyze · SEO tools

Crawler for Cocolyze's SEO analysis platform, fetching pages of sites its users analyse. Cocolyze publishes no IP ranges, so requests cannot be verified beyond the user agent.

cognitiveSEO · SEO tools

James BOT is the web crawler operated by cognitiveSEO, an SEO toolset. It crawls the web and analyzes links to power the backlink and SEO analysis offered by the cognitiveSEO platform.

Conductor · SEO tools

The crawler behind Conductor Monitoring, the website auditing product formerly sold as ContentKing. The operator documents that it makes HTTP GET requests to customer-owned websites for continuous auditing of performance and search visibility, that it throttles itself when a server responds slowly or serves errors and stops entirely if problems persist, and that it reviews robots.txt and meta robots rules. Conductor publishes the four address subnets its monitoring runs from.

Criteo · SEO tools

Criteo's advertising crawler. It fetches merchant and publisher pages to extract product and content data used for Criteo's commerce and retargeting ads, and respects robots.txt and crawl-delay directives.

DataForSEO · SEO tools

DataForSEO's crawler. It fetches pages to build the backlink and SEO datasets that power DataForSEO's marketing-data APIs, and honours robots.txt and crawl-delay directives.

Dataprovider.com · SEO tools

Dataprovider.com's in-house crawler. It indexes more than 400 million domains each month and structures what it finds into the company's web dataset (business information, technology detection, classifications and risk signals). The operator documents that it follows the robot exclusion protocol and that its crawlers can be identified by a reverse DNS lookup.

DomCop · SEO tools

The crawler behind DomCop's domain-research service. DomCop members ask it to crawl websites of their choice; it reads those pages, follows links to the depth the member chose and checks the domain behind every outgoing link to find expired domains, recording only the domain names of links that lead nowhere. It also reads a domain's robots.txt once per domain to establish whether a site is still live there. The operator documents robots.txt and Crawl-delay support, a minimum three seconds between requests to the same host, HTML-only fetches capped at 2 MB, and the single address the crawler requests from.

Moz · SEO tools

Moz's general-purpose web crawler, distinct from rogerbot, that gathers link data powering the Moz Link Index and Link Explorer. Moz's own help pages document no fixed IP range for it.

Dragon Metrics · SEO tools

Dragon Metrics' SEO crawler, which collects data for the platform's Site Audit and Site Explorer features. Its operator documents that it respects robots.txt using Google's open-source parser, and that it crawls from dynamic IP addresses so it can only be identified by user agent.

Ezoic · SEO tools

Ezoic's crawler family, run by the digital-publisher technology platform of the same name. A desktop and a mobile variant crawl pages to study how sites, search engines and content interact, alongside named subtypes for Core Web Vitals measurement, uptime checks, ads.txt verification, integration checks and page-topic analysis. All variants share the single robots.txt token EzoicBot.

HubSpot · SEO tools

HubSpot's crawler, which fetches customer and external pages to power the SEO recommendations and link analysis in HubSpot's marketing tools. HubSpot publishes its egress ranges tagged by service, including web crawling.

Integral Ad Science · SEO tools

Content-rating and ad-verification crawler operated by Integral Ad Science. It visits web pages to assess content quality and brand safety and to support invalid-traffic detection for advertisers. IAS publishes the public-facing addresses its robots crawl from as a JSON file linked from its indexing policy; the file is in a shape this directory's feed formats cannot consume, so the ranges are snapshotted here instead.

Integral Ad Science · SEO tools

The second crawler Integral Ad Science documents alongside IAS Crawler. IAS says Wombles captures additional information used to refine its invalid traffic models, as distinct from the content-rating work IAS Crawler does. Both robots are covered by the public IP list IAS publishes.

JetOctopus · SEO tools

Cloud SEO crawler and log analyzer that crawls the sites its customers configure. Its regular user agent is the JetOctopus token recorded here; the crawl configuration also offers desktop browser, mobile browser and Googlebot desktop/mobile strings so a customer can see how their own site renders for those clients, and in those modes the requests are not distinguishable by user agent, only by source address. Robots.txt rules are followed when the crawl's "Respect robots rules" setting is on, which is why the flag below is false, and JetOctopus publishes the servers it crawls from so site owners can allowlist them.

Local Profy LLC (LinkChecker.pro) · SEO tools

LinkChecker.pro's backlink-monitoring crawler. It revisits the backlinks that the service's customers track, checking whether each link is still present and how it is marked. The operator documents that the crawler strictly respects robots.txt and Crawl-delay, and publishes the addresses it crawls from.

Authoritas (Analytics SEO Limited) · SEO tools

Linkdexbot is the web crawler for Linkdex, an SEO and search-marketing analytics platform now operated by Authoritas. It gathers link and page data used to power the platform's SEO reporting tools.

Links Indexer (Kalpraj Solutions (OPC) Private Limited) · SEO tools

The crawler behind the Links Indexer URL indexing service. It parses third-party sites to verify their URLs and status for sitemap campaigns, querying pages for metadata and favicons and taking a homepage screenshot. The operator states it never harvests e-mail addresses or content unrelated to sitemap campaigns, runs at most four parallel requests, and publishes the single address it crawls from.

MContextual · SEO tools

MContextual's crawler for contextual advertising. The operator documents that it reads page content so advertisers can build cookieless contextual audiences from a site's subject matter, that it can be blocked with standard robots.txt directives, and that its requests originate from AWS and GCP addresses rather than a published range.

MegaIndex · SEO tools

MegaIndex is an SEO and web-analytics platform whose crawler indexes links across the web to power backlink analysis, keyword tracking, and site audits for its subscribers.

Meta · SEO tools

Meta's crawler that fetches pages for advertising and other business-related products and services, separate from the AI-training and link-preview crawlers. Verified by ASN lookup (AS32934); Meta publishes no IP feed.

Metrics Tools (Andreas Knatz) · SEO tools

Crawler for Metrics Tools, a German SEO analytics service, collecting page data for its visibility and ranking analyses. The operator publishes no IP ranges, so requests cannot be verified beyond the user agent.

Majestic-12 · SEO tools

The crawler behind Majestic's backlink index. Majestic explicitly states it is a community-based distributed crawler with no fixed IP allocation, so requests cannot be verified by IP, ASN, or reverse DNS.

Acquia · SEO tools

Acquia Web Governance (formerly Monsido) crawler, which scans the public websites its customers have configured to run accessibility, quality assurance and policy checks. It also issues link-status checks against third-party sites that customers have linked to, preferring HEAD requests for those. Acquia documents a fallback user agent — a plain Chrome string with no identifying token — used only when the primary one fails, so a share of its traffic is identifiable by address rather than by user agent.

Nano Interactive · SEO tools

Nano Interactive's contextual-advertising crawler. Its published crawler policy lists four desktop and mobile user agents, all carrying the NanoInteractive/1.0 token, and names the two addresses the crawler requests come from so site owners can allow it explicitly.

OpenAI · SEO tools

OpenAI's fetcher for advertising on ChatGPT. When an advertiser submits an ad, OAI-AdsBot visits the submitted landing page to check that it complies with OpenAI's ad policies and to judge when the ad is relevant to show. OpenAI states it only visits pages submitted as ads and that what it collects is not used to train generative AI foundation models. It has its own published IP range list, separate from GPTBot, OAI-SearchBot and ChatGPT-User.

OnCrawl · SEO tools

OnCrawl's SEO crawler, used to analyze a customer's own site structure and content for technical SEO reporting. OnCrawl publishes the fixed set of static addresses its crawler uses — three in the EU (Belgium), which crawls use by default, and three in the US available on request — and tells customers to allow them. The bot's name is configurable per crawl, so the user agent alone does not establish identity.

Outbrain · SEO tools

Outbrain's content-recommendation crawler. It fetches advertiser landing pages so Outbrain's system can pull the correct image and headline for a promoted-content unit, and rejects submitted URLs it cannot reach.

Panscient Inc. · SEO tools

Panscient's large-scale crawler, which traverses public websites so that Panscient can build structured company and professional data feeds licensed to enterprise customers. The operator documents a full-corpus refresh each quarter, a rate limit of at most one request per second to any single domain, and compliance with the Robot Exclusion Standard. A separate "pantest" agent is used for testing. No IP ranges are published.

Peer39 · SEO tools

Contextual-intelligence crawler operated by Peer39, an advertising data company. It visits publisher pages to categorise their content so that advertisers buying through programmatic platforms can judge the context and suitability of the pages their ads appear on. Peer39's crawler notice states that it drops no cookies and does not track or append browsing behaviour to individual users, and that the crawler reads robots.txt so administrators can allow or disallow it.

PowerMapper Software · SEO tools

The crawler behind PowerMapper Software's website testing tools — SortSite and PowerMapper — which scan a site's pages for accessibility, broken links, browser-compatibility and SEO problems and build visual sitemaps. The operator documents one robots.txt user agent shared by all of its products and tells site owners to allow it explicitly to bypass blocks meant for other crawlers. It publishes no IP ranges or reverse-DNS method, so requests cannot be attributed and the entry carries no verification recipe.

Comscore, Inc. · SEO tools

Proximic is Comscore's web crawler. It downloads the static textual content of pages to perform contextual analysis (content language, rating, and IAB categories) so advertising partners can match campaigns to page content. It identifies itself and honors robots.txt.

Moz · SEO tools

Moz's site-audit crawler for Moz Pro Campaigns, distinct from DotBot. Moz's own FAQ states plainly that Rogerbot has no IP range: "we do not use a static IP address or range of IP addresses."

Semrush · SEO tools

The crawler behind the Ryte.com tools, which analyse on-page SEO, technical and usability issues. Ryte was absorbed by Semrush, and RyteBot is now documented as a member of the Semrush bot family with its own robots.txt user agent. No IP ranges are published for it.

Scope3 · SEO tools

Scope3's crawler, which indexes publicly available web content (and paywalled media where the publisher has granted access) to produce content classification and brand-safety assessments for advertising. Scope3 documents adaptive rate limiting of five pages per minute per domain and publishes the single address it crawls from.

SE Ranking · SEO tools

SE Ranking's Website Audit crawler, which crawls a domain on demand when an SE Ranking customer runs the Website Audit tool and reports SEO and technical issues. It is distinct from SERankingBacklinksBot, the operator's backlink crawler. SE Ranking runs the crawler from hosts under its own sr-srv.net domain, so requests can be checked by reverse DNS.

Search Atlas · SEO tools

The crawler behind Search Atlas's SEO platform, which fetches pages for its Site Auditor and monitoring features. The operator publishes the bot's user agent for allowlisting and states that the crawler does not use static IP addresses, so it can only be identified by its user agent.

Semrush · SEO tools

Semrush's site-auditing crawler, distinct from SemrushBot: it crawls a domain on demand when a Semrush customer runs the Site Audit tool, looking for SEO and technical issues. Unlike the backlink crawler, which Semrush says cannot be identified by IP, Site Audit is documented as running from a single dedicated subnet.

Semrush · SEO tools

Semrush's web crawler, feeding the backlink and site-audit data behind the Semrush SEO platform. Semrush's own bot page explicitly states it does not use consecutive IP blocks, so no CIDR list can be sourced.

Semrush · SEO tools

The crawler behind Semrush's Backlink Audit tool, which helps a customer discover and clean up potentially dangerous backlinks in their profile. Semrush documents it as a robots.txt user agent separate from SemrushBot and publishes no IP ranges for it.

Semrush · SEO tools

The crawler for Semrush Enterprise Site Intelligence, which analyses a customer's site for on-page SEO, technical and usability issues. Semrush documents it as its own robots.txt user agent, separate from the SiteAuditBot used by the standard Site Audit tool, and publishes no IP ranges for it.

Semrush · SEO tools

The fetcher behind Semrush's Plagiarism Checker and similar tools, used to verify whether the content being checked also appears on your website. Semrush documents it as its own robots.txt user agent and publishes no IP ranges for it.

Semrush · SEO tools

The crawler that gathers pages for the reports in Semrush's Content Toolkit. Semrush documents it as its own robots.txt user agent, separate from the SemrushBot backlink crawler, and publishes no IP ranges for it.

Semrush · SEO tools

The crawler behind Semrush's On Page SEO Checker and related on-page tools, run against a domain when a Semrush customer sets up a campaign for it. It is a separate robots.txt user agent from SemrushBot, and Semrush documents its own addresses to allowlist for it.

Semrush · SEO tools

The fetcher Semrush's SEO Writing Assistant uses to check whether a URL is accessible. Semrush documents it as its own robots.txt user agent, separate from the SemrushBot backlink crawler, and publishes no IP ranges for it.

Seobility GmbH · SEO tools

Crawler for Seobility's hosted SEO analysis and site-audit tooling, fetching pages of sites its customers analyse. Seobility publishes a machine-readable list of the addresses its bots crawl from.

Jobkicks SLU · SEO tools

The crawler behind SEOkicks, a German backlink database. It follows links across the public web and records the external links it finds, with their anchor text and link attributes, so site owners can query which sites link to a given domain. The operator's crawler page documents the user agent, a minimum one-second pause between two page requests that lengthens when the origin responds slowly, and that the crawler issues GET requests only.

Serpstat · SEO tools

Serpstat's backlink crawler. It continuously crawls the web to add new links and track changes in Serpstat's link database, honouring robots.txt and Crawl-delay directives, and publishes the full list of addresses it crawls from.

Sirdata · SEO tools

Crawler for Sirdata's contextual advertising API. It fetches and categorises page content when Sirdata's on-page script cannot read the document from the parent frame. Sirdata publishes the addresses it crawls from at proxies-list.sirdata.fr, but that pool rotates continuously — it recommends re-fetching every ten minutes — so no snapshot of it stays accurate.

SISTRIX · SEO tools

The crawler behind the SISTRIX Toolbox, a German SEO visibility platform. SISTRIX documents that every crawler IP resolves via reverse DNS to the "sistrix.net" domain rather than publishing a static CIDR list.

SiteGuru · SEO tools

The crawler behind SiteGuru, a hosted SEO auditing tool. It crawls a site either because someone added that site to SiteGuru to monitor its SEO health, or because a site already being audited links to it and SiteGuru is checking that link for breakage. The operator documents that it reads only publicly available content, uses the HTML source to derive meta descriptions, page titles and missing alt text, submits no forms, crawls only pages that are in the sitemap or linked from another page, and limits concurrent requests to bound the load it puts on an origin. For sites it is auditing it also calls the W3 Validator and Google's PageSpeed API.

Siteimprove · SEO tools

Siteimprove's content-suite crawler, which fetches pages of sites its customers have configured in their account to run quality-assurance, accessibility, policy and SEO checks. Companion agents (LinkCheck, Image size, Probe) fetch links and resources for the same checks and crawl from the same published address list.

Smartology · SEO tools

Smartology's SmartMatch contextual advertising crawler. It reads pages on domains where an advertiser is buying advertising space, so that ads can be matched to page context; the operator states the bot is only active on a domain while suitable ad slots and advertiser budget exist. Smartology publishes its current address list and user-agent match pattern as a JSON document.

Semrush · SEO tools

The crawler behind Semrush's SplitSignal tool, which runs SEO A/B tests on a customer's own website. Semrush documents it under its own robots.txt user agent, distinct from the SemrushBot family name, and publishes no IP ranges for it.

StatusNest · SEO tools

StatusNest's link-graph crawler. It visits public web pages to record their outbound and inbound links and maintain a graph of how sites connect to each other, for use by SEO and web analysts. The operator documents a maximum rate of one request per second per domain with automatic backoff, states that it collects only public link data and basic page metadata, and that it re-reads robots.txt before every crawl session. No addresses are published.

Torben Hansen (t3versions) · SEO tools

Private-project crawler that makes single GET requests to sites and looks for TYPO3 fingerprints, collecting statistics on the worldwide usage and development of the open-source TYPO3 CMS. No IP ranges are published, and the operator documents no robots.txt support (exclusion is by email request).

The Trade Desk · SEO tools

The Trade Desk's content scraper. When a page sends an ad request to The Trade Desk, this crawler scans the page to determine the context in which the ads were displayed, and caches that contextual data for ad serving. The operator publishes a plain-text list of the addresses it crawls from at ttd-content.adsrvr.org/ips; that list currently holds 2,640 individual addresses, which is beyond this directory's per-feed range cap, so it is not recorded as a machine-readable recipe here.

Hunter · SEO tools

Hunter's public web crawler, written in Go. It analyses millions of publicly accessible pages every month to build the business datasets and machine learning models behind Hunter's products, and never fetches anything behind a login. The operator documents a deliberate rate limit of one page at a time and one page every two seconds per site.

World Wide Web Consortium (W3C) · SEO tools

W3C's link checker, which follows the links on a user-submitted page to report broken and redirected references. W3C's validation services page marks it as the one validator that, being a crawling service, honors robots.txt directives, and gives the same published source addresses as the other W3C validators.

World Wide Web Consortium (W3C) · SEO tools

W3C's CSS validator, which fetches a user-submitted page and its stylesheets to check them against the CSS specifications. It runs on W3C's Jigsaw server and identifies itself with a Jigsaw prefix followed by its own W3C_CSS_Validator_JFouffa token, from the same published validator addresses.

World Wide Web Consortium (W3C) · SEO tools

The fetcher behind W3C's Internationalization Checker, which retrieves a user-submitted page and reports on its language, encoding and other i18n markup. W3C's validation services page lists its user agent alongside the single IPv4 address and IPv6 prefix all W3C validators fetch from.

World Wide Web Consortium (W3C) · SEO tools

The W3C Markup Validation Service's fetcher, used when a user submits a URL to be checked for HTML conformance. W3C documents a fixed source address for its validation services alongside the user-agent string.

World Wide Web Consortium (W3C) · SEO tools

The fetcher behind the Nu HTML Checker, the W3C validation service that checks a user-submitted URL against current HTML conformance rules. W3C's validation services page lists its user agent and states that all W3C validator traffic comes from one published IPv4 address and one IPv6 prefix.

Xovi GmbH · SEO tools

XoviBot is the web crawler for XOVI, an SEO and online-marketing analytics suite. It crawls sites to gather backlink and ranking data for the platform's SEO tools.

Yandex · SEO tools

The fetcher for Yandex Direct advertising, which downloads the target pages of ads to check their availability and topic before the ads are placed in search results and on partner sites. Yandex states plainly that this robot does not use the robots.txt file and ignores the directives set for it.

Yahoo · SEO tools

Page-fetch client that retrieves the landing pages of URLs listed with Yahoo advertising services. Yahoo documents that it fetches each advertiser-supplied landing page to check policy compliance and to improve the accuracy of the ad listing, and that it uses separate desktop and mobile user agents.

Yandex · SEO tools

Yandex's accessibility checker, which downloads pages to assess how accessible they are to users. Yandex documents that it sends up to three requests per second, ignores the crawl-rate setting in Yandex Webmaster, and does not follow the general robots.txt rules written for arbitrary robots.

Yandex · SEO tools

The Yandex Advertising Network robot. Yandex's robot table lists it separately from the indexing robots and records that it follows the general robots.txt rules.

Yandex · SEO tools

The Yandex Advertising Network robots. YandexDirect downloads information about the content of partner sites to identify their topic categories so that relevant advertising can be matched to them, and YandexDirectDyn generates dynamic banners. Neither follows the general robots.txt rules.

Yandex · SEO tools

The fetcher behind Yandex's structured data validator, which retrieves a page so its markup can be validated. Yandex's robot table records it as taking the general robots.txt rules into account.

Yandex · SEO tools

A Yandex robot that downloads information about the content of Yandex partner sites. Yandex's robot table lists it separately from the indexing robots and records that it does not follow the general robots.txt rules.

Yandex · SEO tools

The robot of Yandex Webmaster, the site-owner console Yandex publishes its crawler documentation in. Yandex's robot table records that it follows the general robots.txt rules.

ZoomInfo Technologies · SEO tools

ZoomInfo's indexing robot, which scans corporate websites, press releases, news services and SEC filings to build ZoomInfo's search index of businesses and business professionals. The operator documents that it obeys robots.txt, spaces out requests on larger sites and never opens more than one connection to a site at a time. No IP ranges are published.

Amazon · Feed fetchers

Amazon's seller-initiated fetcher, documented alongside AmazonProductDiscoverybot. It fetches a website URL that a seller has supplied so that Amazon can build a product page in its store from that page. Amazon's robots.txt statement on the page covers AmazonProductDiscoverybot only, so no robots behaviour is recorded for this agent.

Apple · Feed fetchers

Apple's podcast feed fetcher, which crawls only URLs associated with content registered on Apple Podcasts. Apple documents that iTMS traffic may come from applebot.apple.com hosts and that it does not follow robots.txt because it is not a general search crawler.

Artemis (jamesg.blog) · Feed fetchers

The feed poller behind Artemis, a hosted web reader that follows websites and blogs on behalf of its subscribers. It updates once per day and uses HEAD and conditional GET requests with If-Modified-Since and If-None-Match to reduce bandwidth. Artemis publishes the addresses it polls feeds from as a plain-text list.

BazQux Reader · Feed fetchers

The feed fetcher of the BazQux Reader hosted RSS service. It retrieves and periodically refreshes the RSS/Atom and comment feeds that users have subscribed to, typically no more than once an hour per feed. BazQux documents that the fetcher acts as an agent of those users and therefore ignores robots.txt.

Meta · Feed fetchers

Meta's product-catalog fetcher, identified by the facebookcatalog user-agent. It retrieves merchant product-data feeds used to build and refresh commerce catalogs surfaced across Facebook and Instagram.

Fedica · Feed fetchers

The fetcher used by Fedica's social-media publishing platform. The operator documents that it does not crawl whole sites: it reads only the page or RSS feed a Fedica user has specified, downloading feed content so the user can schedule posts from it, and reading Open Graph metadata so a shared link renders as a card. Fedica publishes the robots.txt token but states no robots.txt policy and no address ranges.

Feedbin · Feed fetchers

Feedbin's feed fetcher, which retrieves RSS/Atom feeds that users have subscribed to. Its user agent includes the internal feed id and current subscriber count, and Feedbin documents forward-confirmed reverse DNS in *.bot.feedbin.com as the way to verify its requests.

Feeder · Feed fetchers

Crawler for Feeder, a hosted RSS reading service. The operator documents that it only fetches feeds users have actively subscribed to, that it acts as an agent on the user's behalf and therefore does not check robots.txt, and that rate limits can be arranged by contacting support. Its user agent is a browser-shaped string in which Feeder deliberately inserts the feeder.co identifier as the honest self-identification. The operator publishes the addresses of its crawler servers, each with a reverse DNS name under feeder.co that resolves back to the same address, and offers the same list as JSON.

Feedly · Feed fetchers

Feedly's fetcher, which retrieves RSS/Atom feed URLs after a user has explicitly added them to their Feedly. Feedly documents that it behaves as a direct agent of the user rather than a robot, and does not publish a fixed IP list because its source IPs change over time.

Feedspot · Feed fetchers

Feedspot is a hosted content reader and feed aggregation service. Its bot fetches RSS and Atom feeds and web content on behalf of Feedspot users.

Mikle KK (FeedWind) · Feed fetchers

The crawler behind FeedWind, an embeddable RSS/Atom widget. It fetches the feed sources that a widget owner has configured, every five minutes to five hours depending on their plan, so the rendered widget stays current. The operator's support page prints the crawler's full user agent and asks feed publishers to unblock it if a firewall is turning it away.

Flipboard, Inc. · Feed fetchers

Flipboard's proxy service, which fetches and prepares elements of a page (e.g. a social feed a user asked Flipboard to scan) for presentation in the Flipboard app. Flipboard's own docs say these requests currently originate from an Amazon EC2 cluster but publish no fixed IP list.

Google · Feed fetchers

Google's feed retrieval agent for RSS and Atom feeds used by Google News and WebSub. It fetches and periodically refreshes feeds that users of an app or service have explicitly subscribed to.

Hatena Co., Ltd. (Hatelabo) · Feed fetchers

The fetcher behind Daichecker, the antenna service run on Hatelabo, Hatena's experimental-services lab. It checks the pages and feeds that users have registered for updates. Hatena documents that it parses only the robots.txt groups that name this user agent directly and does not apply the User-agent: * group, so a wildcard rule will not stop it. Hatena notes the name comes from an internal code name for RSS-reader development and has no connection to the country.

Innguma · Feed fetchers

Innguma's feed fetcher. It retrieves RSS and Atom feeds that users have added to Innguma or to another application built on the Innguma cloud, and refreshes them roughly once an hour. The operator states that because the requests follow explicit action by human users rather than automated crawling, the fetcher does not follow robots.txt, and documents serving an error status to the Innguma/1.0 user agent as the opt-out.

Innologica · Feed fetchers

Inoreader's feed fetcher, which retrieves RSS/Atom feeds that Inoreader users have subscribed to. Its own docs state it does not read robots.txt because it fetches specific, user-requested feed URLs rather than crawling a site, and it publishes a live list of its backend fetcher IPs.

Miniflux · Feed fetchers

Miniflux is a minimalist, open-source, self-hosted feed reader. User-run instances fetch the RSS and Atom feeds their subscribers add, identifying themselves with a Miniflux User-Agent.

Missinglettr · Feed fetchers

The fetcher behind Missinglettr's social-media campaign builder. The operator documents that it crawls a blog post or web page only when a Missinglettr customer submits that content, extracting it so the platform can auto-create social posts, and states explicitly that it does not crawl the web at large like a traditional search engine. Missinglettr documents robots.txt control with a worked MissinglettrBot block and notes that blocking it stops social campaigns being created from the site's content.

NewsBlur · Feed fetchers

NewsBlur's open-source feed fetcher, which polls RSS/Atom feeds on behalf of subscribed users. Its user agent embeds the live subscriber count and the feed's permalink; NewsBlur publishes no fixed IP range for it.

Overcast Radio, LLC · Feed fetchers

The server-side feed crawler of the Overcast podcast app. Overcast fetches each podcast's RSS feed centrally rather than from every listener's device, at intervals from a few minutes to 24 hours depending on the show's popularity, and reports the show's Overcast subscriber count in its User-Agent. Standard HTTP caching headers are obeyed. Note that Overcast's iOS and Apple Watch apps send their own Overcast User-Agents when downloading episode audio, but those requests come from listeners' own devices rather than from Overcast's crawlers, so they are outside the hostnames this entry verifies.

Automattic, Inc. · Feed fetchers

The server-side feed parser of the Pocket Casts podcast app. It periodically downloads podcast RSS feeds to check for new episodes, update show metadata such as titles, descriptions and artwork, and verify that feeds are still reachable, so that listeners' apps do not each fetch the feed themselves. The operator documents reasonable delays between requests and asks publishers not to block the parser's user agent or addresses.

Podscan Inc. · Feed fetchers

The ingestion crawler of Podscan, a podcast monitoring and full-text search platform. It fetches podcast RSS feeds and episode content so the platform can index and alert on what is said in them. Podscan documents conditional GET requests with If-Modified-Since and ETag, gzip support, exponential backoff up to 24 hours for feeds that have not changed, and that it honours Cache-Control max-age, Retry-After, and the RSS skipDays and skipHours tags. It publishes no IP ranges, so requests cannot be attributed to the operator beyond the User-Agent.

Productsup · Feed fetchers

Crawler behind Productsup's website data-import source, which reads pages of a site a Productsup customer has configured and turns them into a product data feed. Productsup documents the default user agent, and notes that customers may append a hash to it for their own filtering.

Rakuten · Feed fetchers

Rakuten's image-extraction bot, which retrieves product images from merchant sites that have partnered with Rakuten. Its bot page names the identifying token that appears in the user-agent string and gives an abuse contact.

Scour (Evan Schwartz) · Feed fetchers

Feed fetcher for Scour, a personalized content feed service in which users subscribe to RSS, Atom and JSON feeds and topics of interest. It polls each subscribed feed once every 900 seconds regardless of subscriber count, uses conditional requests, rate-limits itself to two requests per second per registrable domain, and backs off on error responses.

UpContent · Feed fetchers

UpContent's page requester and web crawler, which the operator describes as identifying and downloading elements of web pages so those pages can be found by UpContent's content-curation users. UpContent documents that Sottopop paces itself to keep requests minimised and sporadic and fetches only when new information is available, and states plainly that it does not currently respect robots.txt while asking site owners to publish a Sottopop rule now so it takes effect when support ships; the documented immediate opt-out is to serve 404 or 410 to the user agent.

Superfeedr · Feed fetchers

Superfeedr's PubSubHubbub feed-polling infrastructure, which fetches feed URLs that publishers or subscribers have registered with the service. Its docs publish a list of current node IPs but warn it changes as they add or remove cloud capacity.

World Wide Web Consortium (W3C) · Feed fetchers

W3C's feed validator, which fetches a user-submitted RSS or Atom feed to check it against the feed specifications. It is a one-off conformance check rather than a subscription fetcher, and W3C publishes its user agent and the shared validator source addresses.

Yandex · Feed fetchers

The Yandex Calendar robot, which downloads calendar files at the request of Yandex Calendar users. Yandex notes that these files are often located in directories that are disallowed for indexing, and that the robot does not follow the general robots.txt rules.

Yandex · Feed fetchers

A Yandex robot that downloads structured data from links website owners have themselves submitted to Yandex services. Yandex's robot table records that it makes no indexing requests and does not follow the general robots.txt rules.

Yandex · Feed fetchers

A Yandex robot that downloads product catalogs in YML files at the request of users. Yandex notes that these files are often placed in directories that are disallowed for indexing, and that the robot does not follow the general robots.txt rules.

Yandex · Feed fetchers

The robot of Yandex Tracker, Yandex's work-management service. Yandex's robot table lists it among the robots that download web documents for purposes other than indexing, and records that it does not follow the general robots.txt rules.

Riga Technical University (Institute of Applied Computer Systems) · Archivers

Crawler run by Riga Technical University that indexes websites and documents to compare against student and researcher works for plagiarism detection. The operator publishes no IP ranges, so requests cannot be verified.

Arquivo.pt (FCCN/FCT) · Archivers

The crawler behind Arquivo.pt, Portugal's public web archive, which captures full page renders (HTML, CSS, JS, images) for long-term preservation. Built on Heritrix. Arquivo.pt publishes the ranges its crawlers operate from as a machine-readable JSON file in the JAFAR format, and recommends combining that IP check with the User-Agent check.

The British Library (UK legal deposit libraries) · Archivers

The web crawler of the UK legal deposit libraries, which harvests UK-published websites so that the British Library and the other five legal deposit libraries can archive online publications as they are entitled to do under legal deposit. It runs on Heritrix and is throttled to avoid taking too much bandwidth or overloading a site. A separate rendering component identifies itself as bl.uk_lddc_renderbot. The libraries state the crawler generally follows the robots.txt exclusion protocol and normally respects robots META tags, but that they may choose to overrule robots.txt in order to harvest content successfully — for example to collect JavaScript or CSS that robots.txt excludes. No IP ranges are published, so requests cannot be verified; the guidance for publishers who want to stop the crawl is to name it in robots.txt.

Bibliothèque nationale de France · Archivers

The web crawler of the Bibliothèque nationale de France, which harvests French websites for the legal deposit of the web to preserve the national documentary heritage. It runs on Heritrix and applies request delays to avoid overloading servers.

CLARIN.SI (CLASSLA Knowledge Centre) · Archivers

The CLASSLA-web crawler collects web corpora for South Slavic and other languages under the CLARIN Knowledge Centre for South Slavic languages, continuing the CEF-funded MaCoCu project's crawling infrastructure. It runs the SpiderLing crawler developed at Masaryk University. The operator documents that it adheres to the robots exclusion standard and reads robots.txt on the first access of each crawl run.

Cloudflare · Archivers

Cloudflare's Always Online crawler, which fetches pages from sites that have the feature enabled so a cached copy can be served to visitors when the origin server is unreachable. Cloudflare's crawler reference documents the CloudFlare-AlwaysOnline user agent for this product.

ROIS-DS Data Lake Research and Development Center · Archivers

A research web crawler run by the Data Lake Research and Development Center of the Joint Support-Center for Data Science Research (ROIS-DS), Research Organization of Information and Systems, Japan. It collects Japanese-language web data for research use. The operator page documents RFC 9309 robots.txt handling with Crawl-delay support, robots meta tag support, and the address range the crawler requests from.

Internet Archive · Archivers

The Internet Archive's Heritrix-based crawler used for its wide crawl of the web, feeding the Wayback Machine. The Archive says it crawls slowly to avoid disrupting sites and publishes no IP ranges. Its help pages note that robots exclusions may prevent archiving, but the operator does not document a commitment to obey robots.txt across its crawls.

Archive Team · Archivers

ArchiveBot is an IRC-controlled archiving bot run by Archive Team that crawls websites on request, writes WARC files, and uploads the captures to the Internet Archive.

PlagAware · Archivers

PlagAware's fetcher, used by the German plagiarism-checking service of the same name. Text sections of a submitted document are passed to search engines, and this bot then reads only the individual pages those searches flagged as possible sources, caching them for 48 hours. The operator documents that it does not crawl whole sites and that it adheres to robots.txt directives.

Turnitin · Archivers

TurnitinBot is the web crawler operated by Turnitin. It collects publicly available web pages to build the content database used by Turnitin's academic-integrity and plagiarism-detection services.

XY Planning Network · Archivers

Website archiver for XY Archive, XYPN's compliance-archiving product for financial advisory firms. It crawls a subscribing firm's site to decide which pages to capture and then screenshots each page for the archive; XYPN documents the single address it archives from.

Censys · Security scanners

Censys' internet-wide scanner, which probes public IP addresses to build the host/service data behind Censys Search. Censys publishes a fixed set of scanner subnets and ASNs for opt-out purposes.

Cloudflare · Security scanners

Cloudflare's custom-hostname ownership checker. Cloudflare documents that requests carrying this user agent are triggered when a customer chooses to validate a custom hostname with an HTTP ownership token, which requires fetching the token from the hostname being claimed.

Cloudflare · Security scanners

The crawler behind Cloudflare Security Insights, which reviews a zone for security problems and surfaces them in the Cloudflare dashboard. Its user agent is a Chrome string with a Cloudflare self-identification appended; Cloudflare's crawler reference documents it but does not tie the traffic to any published address list, so no CIDR recipe is attached.

Cloudflare · Security scanners

Cloudflare's SSL/TLS Recommender crawler, which fetches a customer origin over both HTTP and HTTPS to determine whether the site can be served fully over HTTPS and to recommend TLS settings.

Palo Alto Networks · Security scanners

Palo Alto Networks' attack-surface-management scanner. It continuously scans the global internet from a published set of ranges to map its customers' internet-facing assets and discover emerging threats, and its requests carry a plain-English user agent naming the company and an opt-out contact address.

Detectify · Security scanners

Detectify's external attack-surface and vulnerability scanner, which probes customer-configured assets from a documented set of AWS-hosted source IPs. The operator publishes both the source IPs and the scanner's user-agent strings.

EdgeWatch · Security scanners

Internet-wide scanner for EdgeWatch's attack-surface and threat-intelligence platform. It makes connection attempts to public IPv4 and IPv6 addresses daily and completes protocol handshakes to fingerprint the services it finds; EdgeWatch documents the subnet its scanners use and an e-mail opt-out.

Driftnet · Security scanners

Driftnet's internet-measurement scanner, run from the internet-measurement.com domain, which probes publicly exposed services to give network owners an external view of their infrastructure. The operator states this traffic never attempts to log in to systems and publishes its scanner IP ranges.

JusProg e.V. · Security scanners

Web crawler operated by JusProg e.V., a German non-profit youth-protection association, that fetches and rates web pages to build the age-classification database used by its parental-control filtering software.

LeakIX · Security scanners

LeakIX's internet-wide recon scanner (l9explore), which probes exposed services to populate the LeakIX misconfiguration/vulnerability search engine. Its probe fleet publishes a live, per-host list of source IPs at scan.leakix.net whose hostnames resolve under scan.leakix.org, and its scanning tool sets an identifiable user-agent.

Qualys · Security scanners

Qualys' SSL Labs server test, which runs visitor-initiated and monthly (SSL Pulse) TLS-configuration assessments of public HTTPS sites. Qualys documents the assessments as slow and non-intrusive and publishes the scanner's source IP ranges in its support knowledge base.

Crissy Field GmbH (Repo Lookout) · Security scanners

Repo Lookout is a non-commercial internet-wide security scanner that looks for source-code repositories accidentally exposed to the public web and reports them to the domain's technical contact. The operator documents the scanner's user-agent prefix and offers two opt-outs: an email request, or denying every request whose user agent starts with RepoLookoutBot.

UBT (EU) Ltd · Security scanners

UBT's web crawler, which powers a classification service that categorises public websites by their content. It re-crawls a given site roughly once every three days. UBT publishes no IP list, and instead documents reverse-DNS verification against the ubtsupport.com domain.

Stripe · Security scanners

Stripe's automated web crawler. Collects data from Stripe users' websites so Stripe can provide its services and comply with financial regulations. Distinct from Stripe's webhook delivery traffic.

Sur.ly · Security scanners

Sur.ly's crawler. The operator runs a spam-fighting link-safety service and uses this bot to fetch third-party sites and build a short security profile for each one, querying metadata and favicons and taking a screenshot of the homepage. The operator states it never harvests e-mail addresses or content unrelated to security.

Adyen · Webhooks

Adyen's payment-event webhook delivery service. Not a crawler; POSTs notifications to merchant endpoints. Adyen documents its egress domain for DNS-based allowlisting rather than a static IP list, since its outbound IPs change over time.

EasyCron · Webhooks

Hosted cron service that calls URLs its users schedule, on the interval or cron expression they configure. Requests go only to endpoints the user entered, and EasyCron publishes the IPv4 and IPv6 addresses its execution bots call from as a JSON list.

GitHub · Webhooks

GitHub's webhook delivery service. Not a crawler; sends repository and organization event notifications to configured endpoints from IP ranges published in GitHub's meta API.

Google · Webhooks

Google's agent that delivers push notification messages sent through Google APIs (such as Pub/Sub and WebSub push subscriptions) to subscriber endpoints.

PayPal · Webhooks

PayPal's Instant Payment Notification service. Not a crawler; POSTs payment-event notifications to merchant listener endpoints from a documented, static set of server CIDR ranges (shared with other PayPal server traffic).

Stripe · Webhooks

Stripe's webhook delivery service. Not a crawler; sends event notifications (payments, subscriptions) to merchant endpoints from published IPs.

Svix · Webhooks

Webhook-sending infrastructure used by Svix's customers to deliver events. Not a crawler; Pro/Enterprise plans get a documented, static set of per-region source IPs, and requests include a Svix sender identifier in the User-Agent. Svix publishes those addresses as a JSON file grouped by region, and notes that every webhook from its hosted service also carries the sender identifier sender-9YMgn.

Telegram · Webhooks

Telegram's webhook delivery to bot servers. Not a crawler; POSTs update events to a bot's registered HTTPS endpoint from two documented, static CIDR ranges. Telegram documents no User-Agent for these POSTs; the UA listed here is Telegram's documented fetcher token and is unconfirmed for webhook traffic — verify by source IP, not UA.

Twilio · Webhooks

Twilio's webhook delivery service, which POSTs event callbacks (incoming messages and calls, status updates) to customer-configured endpoints. Not a crawler. Twilio states there is no fixed range of source IPs — requests come from a dynamic pool — so recipients are told to validate the X-Twilio-Signature request signature instead.