Feed fetchers 32

Amazon · Feed fetchers

Amazon's seller-initiated fetcher, documented alongside AmazonProductDiscoverybot. It fetches a website URL that a seller has supplied so that Amazon can build a product page in its store from that page. Amazon's robots.txt statement on the page covers AmazonProductDiscoverybot only, so no robots behaviour is recorded for this agent.

Apple · Feed fetchers

Apple's podcast feed fetcher, which crawls only URLs associated with content registered on Apple Podcasts. Apple documents that iTMS traffic may come from applebot.apple.com hosts and that it does not follow robots.txt because it is not a general search crawler.

Artemis (jamesg.blog) · Feed fetchers

The feed poller behind Artemis, a hosted web reader that follows websites and blogs on behalf of its subscribers. It updates once per day and uses HEAD and conditional GET requests with If-Modified-Since and If-None-Match to reduce bandwidth. Artemis publishes the addresses it polls feeds from as a plain-text list.

BazQux Reader · Feed fetchers

The feed fetcher of the BazQux Reader hosted RSS service. It retrieves and periodically refreshes the RSS/Atom and comment feeds that users have subscribed to, typically no more than once an hour per feed. BazQux documents that the fetcher acts as an agent of those users and therefore ignores robots.txt.

Meta · Feed fetchers

Meta's product-catalog fetcher, identified by the facebookcatalog user-agent. It retrieves merchant product-data feeds used to build and refresh commerce catalogs surfaced across Facebook and Instagram.

Fedica · Feed fetchers

The fetcher used by Fedica's social-media publishing platform. The operator documents that it does not crawl whole sites: it reads only the page or RSS feed a Fedica user has specified, downloading feed content so the user can schedule posts from it, and reading Open Graph metadata so a shared link renders as a card. Fedica publishes the robots.txt token but states no robots.txt policy and no address ranges.

Feedbin · Feed fetchers

Feedbin's feed fetcher, which retrieves RSS/Atom feeds that users have subscribed to. Its user agent includes the internal feed id and current subscriber count, and Feedbin documents forward-confirmed reverse DNS in *.bot.feedbin.com as the way to verify its requests.

Feeder · Feed fetchers

Crawler for Feeder, a hosted RSS reading service. The operator documents that it only fetches feeds users have actively subscribed to, that it acts as an agent on the user's behalf and therefore does not check robots.txt, and that rate limits can be arranged by contacting support. Its user agent is a browser-shaped string in which Feeder deliberately inserts the feeder.co identifier as the honest self-identification. The operator publishes the addresses of its crawler servers, each with a reverse DNS name under feeder.co that resolves back to the same address, and offers the same list as JSON.

Feedly · Feed fetchers

Feedly's fetcher, which retrieves RSS/Atom feed URLs after a user has explicitly added them to their Feedly. Feedly documents that it behaves as a direct agent of the user rather than a robot, and does not publish a fixed IP list because its source IPs change over time.

Feedspot · Feed fetchers

Feedspot is a hosted content reader and feed aggregation service. Its bot fetches RSS and Atom feeds and web content on behalf of Feedspot users.

Mikle KK (FeedWind) · Feed fetchers

The crawler behind FeedWind, an embeddable RSS/Atom widget. It fetches the feed sources that a widget owner has configured, every five minutes to five hours depending on their plan, so the rendered widget stays current. The operator's support page prints the crawler's full user agent and asks feed publishers to unblock it if a firewall is turning it away.

Flipboard, Inc. · Feed fetchers

Flipboard's proxy service, which fetches and prepares elements of a page (e.g. a social feed a user asked Flipboard to scan) for presentation in the Flipboard app. Flipboard's own docs say these requests currently originate from an Amazon EC2 cluster but publish no fixed IP list.

Google · Feed fetchers

Google's feed retrieval agent for RSS and Atom feeds used by Google News and WebSub. It fetches and periodically refreshes feeds that users of an app or service have explicitly subscribed to.

Hatena Co., Ltd. (Hatelabo) · Feed fetchers

The fetcher behind Daichecker, the antenna service run on Hatelabo, Hatena's experimental-services lab. It checks the pages and feeds that users have registered for updates. Hatena documents that it parses only the robots.txt groups that name this user agent directly and does not apply the User-agent: * group, so a wildcard rule will not stop it. Hatena notes the name comes from an internal code name for RSS-reader development and has no connection to the country.

Innguma · Feed fetchers

Innguma's feed fetcher. It retrieves RSS and Atom feeds that users have added to Innguma or to another application built on the Innguma cloud, and refreshes them roughly once an hour. The operator states that because the requests follow explicit action by human users rather than automated crawling, the fetcher does not follow robots.txt, and documents serving an error status to the Innguma/1.0 user agent as the opt-out.

Innologica · Feed fetchers

Inoreader's feed fetcher, which retrieves RSS/Atom feeds that Inoreader users have subscribed to. Its own docs state it does not read robots.txt because it fetches specific, user-requested feed URLs rather than crawling a site, and it publishes a live list of its backend fetcher IPs.

Miniflux · Feed fetchers

Miniflux is a minimalist, open-source, self-hosted feed reader. User-run instances fetch the RSS and Atom feeds their subscribers add, identifying themselves with a Miniflux User-Agent.

Missinglettr · Feed fetchers

The fetcher behind Missinglettr's social-media campaign builder. The operator documents that it crawls a blog post or web page only when a Missinglettr customer submits that content, extracting it so the platform can auto-create social posts, and states explicitly that it does not crawl the web at large like a traditional search engine. Missinglettr documents robots.txt control with a worked MissinglettrBot block and notes that blocking it stops social campaigns being created from the site's content.

NewsBlur · Feed fetchers

NewsBlur's open-source feed fetcher, which polls RSS/Atom feeds on behalf of subscribed users. Its user agent embeds the live subscriber count and the feed's permalink; NewsBlur publishes no fixed IP range for it.

Overcast Radio, LLC · Feed fetchers

The server-side feed crawler of the Overcast podcast app. Overcast fetches each podcast's RSS feed centrally rather than from every listener's device, at intervals from a few minutes to 24 hours depending on the show's popularity, and reports the show's Overcast subscriber count in its User-Agent. Standard HTTP caching headers are obeyed. Note that Overcast's iOS and Apple Watch apps send their own Overcast User-Agents when downloading episode audio, but those requests come from listeners' own devices rather than from Overcast's crawlers, so they are outside the hostnames this entry verifies.

Automattic, Inc. · Feed fetchers

The server-side feed parser of the Pocket Casts podcast app. It periodically downloads podcast RSS feeds to check for new episodes, update show metadata such as titles, descriptions and artwork, and verify that feeds are still reachable, so that listeners' apps do not each fetch the feed themselves. The operator documents reasonable delays between requests and asks publishers not to block the parser's user agent or addresses.

Podscan Inc. · Feed fetchers

The ingestion crawler of Podscan, a podcast monitoring and full-text search platform. It fetches podcast RSS feeds and episode content so the platform can index and alert on what is said in them. Podscan documents conditional GET requests with If-Modified-Since and ETag, gzip support, exponential backoff up to 24 hours for feeds that have not changed, and that it honours Cache-Control max-age, Retry-After, and the RSS skipDays and skipHours tags. It publishes no IP ranges, so requests cannot be attributed to the operator beyond the User-Agent.

Productsup · Feed fetchers

Crawler behind Productsup's website data-import source, which reads pages of a site a Productsup customer has configured and turns them into a product data feed. Productsup documents the default user agent, and notes that customers may append a hash to it for their own filtering.

Rakuten · Feed fetchers

Rakuten's image-extraction bot, which retrieves product images from merchant sites that have partnered with Rakuten. Its bot page names the identifying token that appears in the user-agent string and gives an abuse contact.

Scour (Evan Schwartz) · Feed fetchers

Feed fetcher for Scour, a personalized content feed service in which users subscribe to RSS, Atom and JSON feeds and topics of interest. It polls each subscribed feed once every 900 seconds regardless of subscriber count, uses conditional requests, rate-limits itself to two requests per second per registrable domain, and backs off on error responses.

UpContent · Feed fetchers

UpContent's page requester and web crawler, which the operator describes as identifying and downloading elements of web pages so those pages can be found by UpContent's content-curation users. UpContent documents that Sottopop paces itself to keep requests minimised and sporadic and fetches only when new information is available, and states plainly that it does not currently respect robots.txt while asking site owners to publish a Sottopop rule now so it takes effect when support ships; the documented immediate opt-out is to serve 404 or 410 to the user agent.

Superfeedr · Feed fetchers

Superfeedr's PubSubHubbub feed-polling infrastructure, which fetches feed URLs that publishers or subscribers have registered with the service. Its docs publish a list of current node IPs but warn it changes as they add or remove cloud capacity.

World Wide Web Consortium (W3C) · Feed fetchers

W3C's feed validator, which fetches a user-submitted RSS or Atom feed to check it against the feed specifications. It is a one-off conformance check rather than a subscription fetcher, and W3C publishes its user agent and the shared validator source addresses.

Yandex · Feed fetchers

The Yandex Calendar robot, which downloads calendar files at the request of Yandex Calendar users. Yandex notes that these files are often located in directories that are disallowed for indexing, and that the robot does not follow the general robots.txt rules.

Yandex · Feed fetchers

A Yandex robot that downloads structured data from links website owners have themselves submitted to Yandex services. Yandex's robot table records that it makes no indexing requests and does not follow the general robots.txt rules.

Yandex · Feed fetchers

A Yandex robot that downloads product catalogs in YML files at the request of users. Yandex notes that these files are often placed in directories that are disallowed for indexing, and that the robot does not follow the general robots.txt rules.

Yandex · Feed fetchers

The robot of Yandex Tracker, Yandex's work-management service. Yandex's robot table lists it among the robots that download web documents for purposes other than indexing, and records that it does not follow the general robots.txt rules.