Feed fetchers 14

Amazon · Feed fetchers

Amazon's seller-initiated fetcher, documented alongside AmazonProductDiscoverybot. It fetches a website URL that a seller has supplied so that Amazon can build a product page in its store from that page. Amazon's robots.txt statement on the page covers AmazonProductDiscoverybot only, so no robots behaviour is recorded for this agent.

Apple · Feed fetchers

Apple's podcast feed fetcher, which crawls only URLs associated with content registered on Apple Podcasts. Apple documents that iTMS traffic may come from applebot.apple.com hosts and that it does not follow robots.txt because it is not a general search crawler.

BazQux Reader · Feed fetchers

The feed fetcher of the BazQux Reader hosted RSS service. It retrieves and periodically refreshes the RSS/Atom and comment feeds that users have subscribed to, typically no more than once an hour per feed. BazQux documents that the fetcher acts as an agent of those users and therefore ignores robots.txt.

Meta · Feed fetchers

Meta's product-catalog fetcher, identified by the facebookcatalog user-agent. It retrieves merchant product-data feeds used to build and refresh commerce catalogs surfaced across Facebook and Instagram.

Feedbin · Feed fetchers

Feedbin's feed fetcher, which retrieves RSS/Atom feeds that users have subscribed to. Its user agent includes the internal feed id and current subscriber count, and Feedbin documents forward-confirmed reverse DNS in *.bot.feedbin.com as the way to verify its requests.

Feedly · Feed fetchers

Feedly's fetcher, which retrieves RSS/Atom feed URLs after a user has explicitly added them to their Feedly. Feedly documents that it behaves as a direct agent of the user rather than a robot, and does not publish a fixed IP list because its source IPs change over time.

Feedspot · Feed fetchers

Feedspot is a hosted content reader and feed aggregation service. Its bot fetches RSS and Atom feeds and web content on behalf of Feedspot users.

Flipboard, Inc. · Feed fetchers

Flipboard's proxy service, which fetches and prepares elements of a page (e.g. a social feed a user asked Flipboard to scan) for presentation in the Flipboard app. Flipboard's own docs say these requests currently originate from an Amazon EC2 cluster but publish no fixed IP list.

Google · Feed fetchers

Google's feed retrieval agent for RSS and Atom feeds used by Google News and WebSub. It fetches and periodically refreshes feeds that users of an app or service have explicitly subscribed to.

Hatena Co., Ltd. (Hatelabo) · Feed fetchers

The fetcher behind Daichecker, the antenna service run on Hatelabo, Hatena's experimental-services lab. It checks the pages and feeds that users have registered for updates. Hatena documents that it parses only the robots.txt groups that name this user agent directly and does not apply the User-agent: * group, so a wildcard rule will not stop it. Hatena notes the name comes from an internal code name for RSS-reader development and has no connection to the country.

Innologica · Feed fetchers

Inoreader's feed fetcher, which retrieves RSS/Atom feeds that Inoreader users have subscribed to. Its own docs state it does not read robots.txt because it fetches specific, user-requested feed URLs rather than crawling a site, and it publishes a live list of its backend fetcher IPs.

Miniflux · Feed fetchers

Miniflux is a minimalist, open-source, self-hosted feed reader. User-run instances fetch the RSS and Atom feeds their subscribers add, identifying themselves with a Miniflux User-Agent.

NewsBlur · Feed fetchers

NewsBlur's open-source feed fetcher, which polls RSS/Atom feeds on behalf of subscribed users. Its user agent embeds the live subscriber count and the feed's permalink; NewsBlur publishes no fixed IP range for it.

Superfeedr · Feed fetchers

Superfeedr's PubSubHubbub feed-polling infrastructure, which fetches feed URLs that publishers or subscribers have registered with the service. Its docs publish a list of current node IPs but warn it changes as they add or remove cloud capacity.