https://verifiedbots.dev Bots newly added to the directory, with identity, category, and verification tier. en Wed, 12 Aug 2026 04:57:29 GMT https://verifiedbots.dev/bots/rytebot https://verifiedbots.dev/bots/rytebot Mon, 10 Aug 2026 17:35:41 GMT The crawler behind the Ryte.com tools, which analyse on-page SEO, technical and usability issues. Ryte was absorbed by Semrush, and RyteBot is now documented as a member of the Semrush bot family with its own robots.txt user agent. No IP ranges are published for it. Operated by Semrush. https://verifiedbots.dev/bots/semrush-siteauditbot https://verifiedbots.dev/bots/semrush-siteauditbot Mon, 10 Aug 2026 17:35:41 GMT Semrush's site-auditing crawler, distinct from SemrushBot: it crawls a domain on demand when a Semrush customer runs the Site Audit tool, looking for SEO and technical issues. Unlike the backlink crawler, which Semrush says cannot be identified by IP, Site Audit is documented as running from a single dedicated subnet. Operated by Semrush. https://verifiedbots.dev/bots/semrushbot-si https://verifiedbots.dev/bots/semrushbot-si Mon, 10 Aug 2026 17:35:41 GMT The crawler behind Semrush's On Page SEO Checker and related on-page tools, run against a domain when a Semrush customer sets up a campaign for it. It is a separate robots.txt user agent from SemrushBot, and Semrush documents its own addresses to allowlist for it. Operated by Semrush. https://verifiedbots.dev/bots/zoominfobot https://verifiedbots.dev/bots/zoominfobot Mon, 10 Aug 2026 17:35:41 GMT ZoomInfo's indexing robot, which scans corporate websites, press releases, news services and SEC filings to build ZoomInfo's search index of businesses and business professionals. The operator documents that it obeys robots.txt, spaces out requests on larger sites and never opens more than one connection to a site at a time. No IP ranges are published. Operated by ZoomInfo Technologies. https://verifiedbots.dev/bots/hatena https://verifiedbots.dev/bots/hatena Sat, 08 Aug 2026 14:42:27 GMT The fetchers Hatena's services use to collect information from pages its users link to. Hatena-Favicon identifies a page's favicon, Hatena::Scissors retrieves image thumbnails, HatenaBookmark fetches article information for Hatena Bookmark, Hatena Star associates stars with pages, and Hatena Antenna fetches update differences for the pages a user is monitoring. Operated by Hatena Co., Ltd.. https://verifiedbots.dev/bots/hatena-russia-crawler https://verifiedbots.dev/bots/hatena-russia-crawler Sat, 08 Aug 2026 14:42:27 GMT The fetcher behind Daichecker, the antenna service run on Hatelabo, Hatena's experimental-services lab. It checks the pages and feeds that users have registered for updates. Hatena documents that it parses only the robots.txt groups that name this user agent directly and does not apply the User-agent: * group, so a wildcard rule will not stop it. Hatena notes the name comes from an internal code name for RSS-reader development and has no connection to the country. Operated by Hatena Co., Ltd. (Hatelabo). https://verifiedbots.dev/bots/hatenablog-bot https://verifiedbots.dev/bots/hatenablog-bot Sat, 08 Aug 2026 14:42:27 GMT Hatena Blog's fetcher. It retrieves a linked page when a blog author asks for it: the :title option of Hatena's URL notation pulls the page title, the :embed option collects the title, summary and favicon used to render a blog card, and the blog import feature fetches images so they can be re-uploaded to Hatena Fotolife. Operated by Hatena Co., Ltd.. https://verifiedbots.dev/bots/dragonbot https://verifiedbots.dev/bots/dragonbot Thu, 06 Aug 2026 15:22:58 GMT Dragon Metrics' SEO crawler, which collects data for the platform's Site Audit and Site Explorer features. Its operator documents that it respects robots.txt using Google's open-source parser, and that it crawls from dynamic IP addresses so it can only be identified by user agent. Operated by Dragon Metrics. https://verifiedbots.dev/bots/outbrain https://verifiedbots.dev/bots/outbrain Thu, 06 Aug 2026 15:22:58 GMT Outbrain's content-recommendation crawler. It fetches advertiser landing pages so Outbrain's system can pull the correct image and headline for a promoted-content unit, and rejects submitted URLs it cannot reach. Operated by Outbrain. https://verifiedbots.dev/bots/sentibot https://verifiedbots.dev/bots/sentibot Thu, 06 Aug 2026 15:22:58 GMT SentiOne's social-listening crawler. It indexes user-generated content for the SentiOne Listen platform, which its operator says analyses over 300,000 domains daily. Robots.txt rules written for "sentibot" are honoured, and Yandex-style reverse-DNS plus a published IP list allow verification. Operated by SentiOne. https://verifiedbots.dev/bots/yandex-blogs https://verifiedbots.dev/bots/yandex-blogs Thu, 06 Aug 2026 15:22:58 GMT Yandex's blog-search robot. Yandex's robot table documents it as the blog search robot that indexes post comments, and lists it as following robots.txt directives. Operated by Yandex. https://verifiedbots.dev/bots/yandex-favicons https://verifiedbots.dev/bots/yandex-favicons Thu, 06 Aug 2026 15:22:58 GMT Yandex's favicon fetcher. Yandex's robot table documents it as downloading a site's favicon file for display in search results, and lists it as one of the Yandex robots that do not follow robots.txt directives. Operated by Yandex. https://verifiedbots.dev/bots/yandex-media https://verifiedbots.dev/bots/yandex-media Thu, 06 Aug 2026 15:22:58 GMT Yandex's multimedia-indexing robot. Yandex's robot table documents it as a separate robot from YandexBot, describes it as indexing multimedia data, and lists it as following robots.txt directives. Operated by Yandex. https://verifiedbots.dev/bots/bing-video-preview https://verifiedbots.dev/bots/bing-video-preview Tue, 04 Aug 2026 20:03:13 GMT Microsoft crawler, listed among Bing's crawlers, that fetches pages to generate video previews shown in Bing. It runs desktop and mobile variants and is verifiable through the same reverse-DNS check as Bing's other crawlers. Operated by Microsoft. https://verifiedbots.dev/bots/cortex-xpanse https://verifiedbots.dev/bots/cortex-xpanse Tue, 04 Aug 2026 20:03:13 GMT Palo Alto Networks' attack-surface-management scanner. It continuously scans the global internet from a published set of ranges to map its customers' internet-facing assets and discover emerging threats, and its requests carry a plain-English user agent naming the company and an opt-out contact address. Operated by Palo Alto Networks. https://verifiedbots.dev/bots/dynatrace-synthetic https://verifiedbots.dev/bots/dynatrace-synthetic Tue, 04 Aug 2026 20:03:13 GMT Dynatrace's synthetic monitoring runs browser and HTTP checks against sites its customers configure. Dynatrace always appends a RuxitSynthetic token to the user agent — even when the customer sets a custom one — so that synthetic traffic can be identified in server logs. Checks are user-configured, not crawling, so robots.txt is not part of the documented behaviour. Operated by Dynatrace. https://verifiedbots.dev/bots/google-agent https://verifiedbots.dev/bots/google-agent Tue, 04 Aug 2026 20:03:13 GMT The fetcher used by AI agents hosted on Google infrastructure to navigate the web and perform actions on behalf of a user who asked for them. Google publishes a dedicated IP range list for it and signs a subset of its requests with Web Bot Auth under the agent.bot.goog identity. As a user-triggered fetcher it generally ignores robots.txt rules. Operated by Google. https://verifiedbots.dev/bots/dataprovider https://verifiedbots.dev/bots/dataprovider Sun, 02 Aug 2026 14:52:18 GMT Dataprovider.com's in-house crawler. It indexes more than 400 million domains each month and structures what it finds into the company's web dataset (business information, technology detection, classifications and risk signals). The operator documents that it follows the robot exclusion protocol and that its crawlers can be identified by a reverse DNS lookup. Operated by Dataprovider.com. https://verifiedbots.dev/bots/dubbot https://verifiedbots.dev/bots/dubbot Sun, 02 Aug 2026 14:52:18 GMT The crawler behind DubBot's web-governance platform. It inventories a customer's own website by following every link from a supplied URL and checks the pages for accessibility, broken links, spelling and content-policy problems. DubBot runs it from AWS on static IP addresses that the operator publishes for allowlisting. Operated by DubBot. https://verifiedbots.dev/bots/searchatlas https://verifiedbots.dev/bots/searchatlas Sun, 02 Aug 2026 14:52:18 GMT The crawler behind Search Atlas's SEO platform, which fetches pages for its Site Auditor and monitoring features. The operator publishes the bot's user agent for allowlisting and states that the crawler does not use static IP addresses, so it can only be identified by its user agent. Operated by Search Atlas. https://verifiedbots.dev/bots/startmebot https://verifiedbots.dev/bots/startmebot Sun, 02 Aug 2026 14:52:18 GMT Start.me's fetcher. When a user adds a link or an RSS widget to their Start.me start page, this bot retrieves three things from the target site: the page title, the site's favicon, and the contents of any RSS feed the site offers. Operated by Start.me. https://verifiedbots.dev/bots/surdotlybot https://verifiedbots.dev/bots/surdotlybot Sun, 02 Aug 2026 14:52:18 GMT Sur.ly's crawler. The operator runs a spam-fighting link-safety service and uses this bot to fetch third-party sites and build a short security profile for each one, querying metadata and favicons and taking a screenshot of the homepage. The operator states it never harvests e-mail addresses or content unrelated to security. Operated by Sur.ly. https://verifiedbots.dev/bots/velen https://verifiedbots.dev/bots/velen Sun, 02 Aug 2026 14:52:18 GMT Hunter's public web crawler, written in Go. It analyses millions of publicly accessible pages every month to build the business datasets and machine learning models behind Hunter's products, and never fetches anything behind a login. The operator documents a deliberate rate limit of one page at a time and one page every two seconds per site. Operated by Hunter. https://verifiedbots.dev/bots/cincraw https://verifiedbots.dev/bots/cincraw Thu, 30 Jul 2026 16:13:28 GMT The web crawler operated by CINC, a Japanese data-solutions company, to collect the page data behind its marketing and SEO analytics products. Its documented policy is to fetch page body content, header and HTTP status information and the JS/CSS needed to render a page, then store a rendered screen capture. CINC states that it does not follow advertising links, deletes all cookies between requests, and does not load analytics or ad-measurement tags. No robots.txt policy and no IP ranges are published. Operated by CINC Corp. (株式会社CINC). https://verifiedbots.dev/bots/cloudflare-ai-search https://verifiedbots.dev/bots/cloudflare-ai-search Thu, 30 Jul 2026 16:13:28 GMT The crawler behind Cloudflare AI Search, which indexes website content so it can be searched. Cloudflare documents that it only crawls a website the customer owns — the domain must exist in the same Cloudflare account and be selected as an AI Search data source. Operated by Cloudflare. https://verifiedbots.dev/bots/cloudflare-alwaysonline https://verifiedbots.dev/bots/cloudflare-alwaysonline Thu, 30 Jul 2026 16:13:28 GMT Cloudflare's Always Online crawler, which fetches pages from sites that have the feature enabled so a cached copy can be served to visitors when the origin server is unreachable. Cloudflare's crawler reference documents the CloudFlare-AlwaysOnline user agent for this product. Operated by Cloudflare. https://verifiedbots.dev/bots/cloudflare-browser-run-crawler https://verifiedbots.dev/bots/cloudflare-browser-run-crawler Thu, 30 Jul 2026 16:13:28 GMT The crawler behind the /crawl endpoint of Cloudflare's Browser Run (Browser Rendering) developer product, which crawls third-party websites on behalf of Cloudflare customers building applications on the platform. Its user agent is not configurable, and every request is signed with Web Bot Auth HTTP message signatures that site owners can verify against Cloudflare's published key directory. Operated by Cloudflare. https://verifiedbots.dev/bots/cloudflare-custom-hostname-verification https://verifiedbots.dev/bots/cloudflare-custom-hostname-verification Thu, 30 Jul 2026 16:13:28 GMT Cloudflare's custom-hostname ownership checker. Cloudflare documents that requests carrying this user agent are triggered when a customer chooses to validate a custom hostname with an HTTP ownership token, which requires fetching the token from the hostname being claimed. Operated by Cloudflare. https://verifiedbots.dev/bots/cloudflare-diagnostics https://verifiedbots.dev/bots/cloudflare-diagnostics Thu, 30 Jul 2026 16:13:28 GMT Cloudflare's support-diagnostics fetcher. Cloudflare documents that requests with this user agent are triggered when Cloudflare Support Engineers perform error checks, and by the continuous monitoring that raises alerts in the Cloudflare dashboard. Operated by Cloudflare. https://verifiedbots.dev/bots/jooblebot https://verifiedbots.dev/bots/jooblebot Thu, 30 Jul 2026 16:13:28 GMT The crawler for Jooble, a job-search aggregator that indexes job listings published across the web. Jooble's bot page states that it uses a web crawler which identifies itself as JoobleBot. No IP ranges are published. Operated by Jooble. https://verifiedbots.dev/bots/panscient https://verifiedbots.dev/bots/panscient Thu, 30 Jul 2026 16:13:28 GMT Panscient's large-scale crawler, which traverses public websites so that Panscient can build structured company and professional data feeds licensed to enterprise customers. The operator documents a full-corpus refresh each quarter, a rate limit of at most one request per second to any single domain, and compliance with the Robot Exclusion Standard. A separate "pantest" agent is used for testing. No IP ranges are published. Operated by Panscient Inc.. https://verifiedbots.dev/bots/bazqux https://verifiedbots.dev/bots/bazqux Tue, 28 Jul 2026 19:51:24 GMT The feed fetcher of the BazQux Reader hosted RSS service. It retrieves and periodically refreshes the RSS/Atom and comment feeds that users have subscribed to, typically no more than once an hour per feed. BazQux documents that the fetcher acts as an agent of those users and therefore ignores robots.txt. Operated by BazQux Reader. https://verifiedbots.dev/bots/bublupbot https://verifiedbots.dev/bots/bublupbot Tue, 28 Jul 2026 19:51:24 GMT Bublup's content-discovery bot. It fetches pages to build the database behind Bublup's suggestion engine, reading page title, description and related images. Bublup states its crawling IPs are not fixed and documents reverse-DNS verification against the bublup.com domain instead. Operated by Bublup. https://verifiedbots.dev/bots/freewebmonitoring https://verifiedbots.dev/bots/freewebmonitoring Tue, 28 Jul 2026 19:51:24 GMT The website-monitoring robot of the FreeWebMonitoring service. It only checks URLs that registered members have submitted, and the operator documents that all checks originate from a single server address. GreenWave Online also warns that an older 0.1 agent name is forged by an unrelated scanner. Operated by GreenWave Online Inc.. https://verifiedbots.dev/bots/streamline3bot https://verifiedbots.dev/bots/streamline3bot Tue, 28 Jul 2026 19:51:24 GMT UBT's web crawler, which powers a classification service that categorises public websites by their content. It re-crawls a given site roughly once every three days. UBT publishes no IP list, and instead documents reverse-DNS verification against the ubtsupport.com domain. Operated by UBT (EU) Ltd. https://verifiedbots.dev/bots/trovitbot https://verifiedbots.dev/bots/trovitbot Tue, 28 Jul 2026 19:51:24 GMT Trovit's web crawler, which discovers new and updated pages to add to the Trovit classifieds search index. Trovit documents that it does not fetch most sites more than once per second and that it can be blocked with the trovitBot robots.txt token. Operated by Trovit. https://verifiedbots.dev/bots/yandex-images https://verifiedbots.dev/bots/yandex-images Tue, 28 Jul 2026 19:51:24 GMT Yandex's image-indexing robot, which fetches images so they can be displayed in Yandex Images. Yandex documents it as a separate robot from YandexBot and lists it as following robots.txt directives. Operated by Yandex. https://verifiedbots.dev/bots/yandex-video https://verifiedbots.dev/bots/yandex-video Tue, 28 Jul 2026 19:51:24 GMT Yandex's video-indexing robot, which fetches pages and video content for display in Yandex video search. Yandex documents it as a separate robot from YandexBot and lists it as following robots.txt directives. Operated by Yandex. https://verifiedbots.dev/bots/amzn-searchbot https://verifiedbots.dev/bots/amzn-searchbot Sun, 26 Jul 2026 13:26:40 GMT Amazon's crawler used to improve search experiences in Amazon products and services; Amazon states it does not crawl content for generative AI model training. Amazon publishes a human-readable IP list but no machine-parseable feed in a format this project's schema supports. Operated by Amazon. https://verifiedbots.dev/bots/amzn-user https://verifiedbots.dev/bots/amzn-user Sun, 26 Jul 2026 13:26:40 GMT Amazon's on-demand fetcher supporting user actions, such as responding to Alexa queries that need up-to-date information; Amazon states it does not crawl content for generative AI model training. Amazon publishes a human-readable live-crawl IP list but no machine-parseable feed. Operated by Amazon. https://verifiedbots.dev/bots/cocolyzebot https://verifiedbots.dev/bots/cocolyzebot Sun, 26 Jul 2026 13:26:40 GMT Crawler for Cocolyze's SEO analysis platform, fetching pages of sites its users analyse. Cocolyze publishes no IP ranges, so requests cannot be verified beyond the user agent. Operated by Cocolyze. https://verifiedbots.dev/bots/deepnoc https://verifiedbots.dev/bots/deepnoc Sun, 26 Jul 2026 13:26:40 GMT Research crawler operated by deepnoc GmbH that parses and stores public web page content so that new search engines can query it without running their own crawling infrastructure. No IP ranges are published. Operated by deepnoc GmbH. https://verifiedbots.dev/bots/duckassistbot https://verifiedbots.dev/bots/duckassistbot Sun, 26 Jul 2026 13:26:40 GMT DuckDuckGo's real-time fetcher for DuckDuckGo Search's AI-assisted answers, which cite their sources. DuckDuckGo states the data is not used to train AI models, and publishes a machine-readable list of the bot's IP addresses. Operated by DuckDuckGo. https://verifiedbots.dev/bots/meta-externalads https://verifiedbots.dev/bots/meta-externalads Sun, 26 Jul 2026 13:26:40 GMT Meta's crawler that fetches pages for advertising and other business-related products and services, separate from the AI-training and link-preview crawlers. Verified by ASN lookup (AS32934); Meta publishes no IP feed. Operated by Meta. https://verifiedbots.dev/bots/meta-webindexer https://verifiedbots.dev/bots/meta-webindexer Sun, 26 Jul 2026 13:26:40 GMT Meta's crawler that navigates the web to improve the quality of Meta AI search results, analysing page content for relevance and accuracy in Meta AI responses. Verified by ASN lookup (AS32934); Meta publishes no IP feed. Operated by Meta. https://verifiedbots.dev/bots/metrics-tools https://verifiedbots.dev/bots/metrics-tools Sun, 26 Jul 2026 13:26:40 GMT Crawler for Metrics Tools, a German SEO analytics service, collecting page data for its visibility and ranking analyses. The operator publishes no IP ranges, so requests cannot be verified beyond the user agent. Operated by Metrics Tools (Andreas Knatz). https://verifiedbots.dev/bots/t3versions https://verifiedbots.dev/bots/t3versions Sun, 26 Jul 2026 13:26:40 GMT Private-project crawler that makes single GET requests to sites and looks for TYPO3 fingerprints, collecting statistics on the worldwide usage and development of the open-source TYPO3 CMS. No IP ranges are published, and the operator documents no robots.txt support (exclusion is by email request). Operated by Torben Hansen (t3versions). https://verifiedbots.dev/bots/webzio-extended https://verifiedbots.dev/bots/webzio-extended Sun, 26 Jul 2026 13:26:40 GMT Second crawler in Webz.io's crawler pair, which performs ethical validation on the data collected by Webzio and tags it as usable or not usable for AI and machine-learning training. Webz.io publishes no IP ranges. Operated by Webz.io Ltd.. https://verifiedbots.dev/bots/caliperbot https://verifiedbots.dev/bots/caliperbot Fri, 24 Jul 2026 13:19:52 GMT Conductor's single web crawler. It reads the HTML of pages on sites its customers track, recording on-page elements such as title tags, header tags and other metadata for Conductor's SEO and search-visibility reporting. Conductor publishes the address range it crawls from and will lower the crawl rate on request. Operated by Conductor. https://verifiedbots.dev/bots/cookiebot-scanner https://verifiedbots.dev/bots/cookiebot-scanner Fri, 24 Jul 2026 13:19:52 GMT Cookiebot's cookie-consent compliance scanner. It crawls the domains its customers have registered, on a roughly monthly schedule, to detect the cookies and tracking technologies in use and generate a cookie declaration. Cookiebot documents that scans run only from a fixed pool of addresses. Operated by Cookiebot (Usercentrics).