Social previews 25

Buffer · Social previews

The crawler behind link previews in Buffer's social-media management product. When someone shares a link in Buffer it fetches the page's public metadata — title, description, thumbnail and Open Graph or Twitter Card data — so the poster can see how the post will look. Buffer publishes its egress addresses both on that page and as a machine-readable JSON document, and sends X-BufferLinkPreviewBot headers alongside the user agent.

Discord · Social previews

Discord's crawler that fetches shared links to render embeds in chat. Discord's own docs only specify the User-Agent format required of API clients, not this embed fetcher; no IP ranges, ASN, or rDNS are published.

Embedly · Social previews

Embedly's fetcher, used to generate rich embeds and previews of shared URLs for its customers' apps and sites. Embedly's FAQ publishes a fixed, small set of source IPs rather than a live CIDR feed.

Meta · Social previews

Meta's crawler that fetches shared links from Facebook, Instagram, and Messenger to generate rich link previews. Meta documents one exception to its robots.txt compliance: the crawler might bypass robots.txt when performing security or integrity checks, such as looking for malware or malicious content. Verified by ASN lookup (AS32934); Meta's own docs describe IP-based allowlisting but publish no IP feed.

Fastmail · Social previews

Fastmail's server-side fetcher, used on behalf of its mail users. It proxies images embedded in messages so that no browser cookies, browser user agent or user IP address reach the remote server, and it fetches subscribed ICS and remote CalDAV calendars. Fastmail documents that every user agent it sends contains the string FastmailUA.

Hatena Co., Ltd. · Social previews

The fetchers Hatena's services use to collect information from pages its users link to. Hatena-Favicon identifies a page's favicon, Hatena::Scissors retrieves image thumbnails, HatenaBookmark fetches article information for Hatena Bookmark, Hatena Star associates stars with pages, and Hatena Antenna fetches update differences for the pages a user is monitoring.

Hatena Co., Ltd. · Social previews

Hatena Blog's fetcher. It retrieves a linked page when a blog author asks for it: the :title option of Hatena's URL notation pulls the page title, the :embed option collects the title, summary and favicon used to render a blog card, and the blog import feature fetches images so they can be re-uploaded to Hatena Fotolife.

Iframely · Social previews

Iframely's fetcher, used to generate embeds and link previews of shared URLs for its customers' apps and sites. Iframely publishes live IPv4/IPv6 address lists and documents reverse-DNS resolution to its own domain.

LinkedIn · Social previews

LinkedIn's crawler that fetches shared links to build post link previews. LinkedIn's own robots.txt names the "LinkedInBot" token, but LinkedIn publishes no dedicated bot-documentation page, IP list, ASN, or reverse-DNS pattern for this crawler.

ListGem · Social previews

ListGem's link unfurler. When someone adds a URL to a ListGem list it fetches that single page to read its public metadata; the operator documents that it does no bulk crawling or link spidering. Every request carries an HTTP Message Signature (RFC 9421, Web Bot Auth profile, Ed25519) and the public key directory is published so requests can be verified.

Livelap · Social previews

LivelapBot is the crawler for Livelap, a content discovery app. It fetches pages shared on social media and crawls RSS feeds on a schedule, indexing HTML and media meta tags to build link previews shown in the Livelap app.

Mastodon gGmbH · Social previews

The generic link-preview fetcher built into Mastodon server software (FetchLinkCardService, using the http.rb gem), run independently by every self-hosted instance. There is no single operator, IP range, or ASN to verify against — traffic can originate from any of thousands of instances.

Microsoft · Social previews

Microsoft fetcher, listed among Bing's crawlers, that retrieves pages to generate link previews for Microsoft products and services.

PagePeeker SRL · Social previews

PagePeeker's thumbnailing robot. It captures a screenshot of a page when one of PagePeeker's customers requests a thumbnail for it, refetching a given site at most once every five to seven days.

Pinterest · Social previews

Pinterest's crawler, used both for general web indexing and for validating Pin metadata/broken links. Pinterest documents a reverse-DNS verification method rather than a fixed IP list, since only its US-based traffic uses a stable range.

Reddit · Social previews

Reddit's on-demand fetcher that retrieves a shared URL's metadata to build the link preview shown on a post. Reddit's robots.txt now disallows all crawling and documents no user agent, IP list, or verification method for this fetcher.

Slack · Social previews

Slack's image-proxying fetcher, which retrieves and caches images posted in channels while stripping referrer data. Slack documents the User-Agent string but publishes no IP ranges or other verification method.

Slack · Social previews

Slack's crawler that expands links posted in channels into rich previews by reading oEmbed, Twitter Card, and Open Graph metadata. Slack documents the User-Agent strings but publishes no IP ranges or other verification method.

Snap Inc. · Social previews

Snapchat's link-preview fetcher, which scans the HTML of URLs shared in Snapchat chats to build preview cards from Open Graph or Twitter Card tags. Snap documents that responses are cached for 30 minutes to reduce traffic; no IP ranges or robots.txt behavior are documented.

Start.me · Social previews

Start.me's fetcher. When a user adds a link or an RSS widget to their Start.me start page, this bot retrieves three things from the target site: the page title, the site's favicon, and the contents of any RSS feed the site offers.

Telegram · Social previews

Telegram's fetcher for generating link previews in chats. Telegram's docs document static CIDR ranges only for inbound webhook delivery to bot servers, not for this outbound preview fetcher, so no IP verification is attached here.

X · Social previews

X (formerly Twitter)'s crawler that fetches shared links to render Card previews. X's own developer docs give no IP ranges, ASN, or reverse-DNS verification method; the commonly cited "*.twttr.com" rDNS check and aggregate IP ranges trace only to community write-ups, not official docs.

VK · Social previews

vkShare is VK's link-preview fetcher. When a user shares a URL on the VK social network, it retrieves the target page to build a preview (title, description, and image) for the shared post.

Meta · Social previews

WhatsApp's fetcher that requests a shared URL to build the link preview shown in chat. Meta documents the request's User-Agent format but no IP feed; verified by ASN lookup (AS32934), same as Meta's other crawlers.

Yahoo · Social previews

Yahoo Link Preview fetches a page when a Yahoo Mail user includes its URL in an email message, generating a thumbnail, title and description preview. It fetches only user-referenced pages rather than crawling the web.