Understanding
bots

A field guide to bot identity — how legitimate bots announce themselves, how their claims get verified, and how this directory decides who gets listed, who gets "listed only," and who gets left out.

What counts as a bot?

Roughly a third of web traffic is automated. The legitimate kinds fall into a few families, and they behave differently enough that lumping them together leads to bad blocking decisions:

Crawlers discover and fetch pages at their own initiative — search engines indexing the web, AI operators collecting training data, archives preserving it. These are the bots robots.txt was invented for.
Fetchers retrieve a specific URL because a person did something — pasting a link into a chat (preview bots), asking an AI assistant a question, subscribing to a feed. They typically don't obey robots.txt by design: a human asked for that exact page.
Service traffic visits you because you (or your vendor) configured it — uptime monitors probing your endpoints, webhook deliveries POSTing events to your API, security scanners you hired. Blocking these usually breaks something you set up yourself.

The directory's ten categories map onto these families, so you can set policy per family instead of guessing per bot.

Identity: the User-Agent

An honest bot announces itself on every request with a stable User-Agent token. The conventional anatomy:

Mozilla/5.0 (compatible; ExampleBot/2.1; +https://example.com/bot)
                        └ name          └ version  └ info URL for site owners

The token (ExampleBot) is also what you address in robots.txt. But here's the problem that motivates everything else on this page: a User-Agent is just a string, and anyone can send any string. Scrapers routinely claim to be Googlebot to borrow its welcome. So identity claims need proof.

Verification: proving the claim

Four methods exist in the wild, in rough order of adoption:

Published IP ranges (CIDR feeds)

The operator publishes the exact IP addresses its bot crawls from, as a machine-readable list at a stable URL. A CIDR like 66.249.64.0/27 is shorthand for a block of addresses (the /27 means the first 27 bits are fixed — 32 addresses). If a request claims to be Googlebot but its IP is not in Google's published list, it isn't Googlebot.

Google publishes common-crawlers.json; this directory re-serves every verified range as plain-text allowlists.

Reverse DNS (forward-confirmed)

The operator points the DNS records of its crawl IPs at hostnames on a domain it controls. Verification is a loop: look up the IP's PTR record → the hostname must end in the operator's domain → look up that hostname's A/AAAA record → it must return the same IP you started with. Both halves matter: anyone can fake a PTR record, but only the domain owner can make the forward lookup match.

Feedbin requests reverse-resolve to *.bot.feedbin.com; Bing's crawlers to *.search.msn.com.

ASN (network ownership)

Every network on the internet routes traffic under an Autonomous System Number. When an operator owns its network outright, checking which AS announces the source IP attributes the traffic — coarser than an IP list (a whole network, not specific crawl ranges), but hard to fake, because routing is public.

Meta's crawlers originate from AS32934, Meta's own network.

Web Bot Auth (signed requests)

The newest method, being standardized at the IETF: the bot cryptographically signs each request (HTTP Message Signatures) and hosts its public keys at a well-known URL. Verification doesn't depend on IPs at all — the signature either checks out against the operator's published key or it doesn't.

Key directories live at /.well-known/http-message-signatures-directory.

The directory's tiers describe which proof is available: Tier 1 — fully verifiable (machine-readable IP feed or Web Bot Auth: you can automate the check), Tier 2 — verifiable (rDNS, ASN, or documented static ranges: checkable, with a bit more work), Tier 3 — listed only (identity established, no technical proof published yet).

Who gets listed — and who doesn't

Every candidate faces the same gate: an identifiable operator (a real organization with working documentation), a distinctive, honest User-Agent (impersonating a browser or another bot disqualifies you), and a legitimate purpose. Pass those three and you're listed — verification then decides your tier, not your existence. Fail any of them and you're not listed at all, because the directory could not say anything true and useful about you.

The distinction matters: Tier 3 is an unverified claim by a known party. An unlisted bot is a claim with no party behind it — or no claim at all. Real cases from this directory's curation:

Listed — Tier 1

ClaudeBot (Anthropic)

Identifiable operator, honest UA, and — since Anthropic began publishing a machine-readable feed of its crawler IPs — full verifiability. ClaudeBot spent time in Tier 3 before that feed existed: same bot, same operator, better proof. Tiers move when operators publish.

Listed — Tier 3

archive.org_bot (Internet Archive)

One of the most identifiable operators on the web, with an honest UA and documentation — but no published IP ranges and no rDNS scheme. You can recognize its traffic and know who claims it; you can't yet prove a given request is really them. That's exactly what "listed only" means.

Listed — Tier 3

Twilio webhooks

Twilio documents that its webhooks come from a dynamic IP pool with no stable ranges — and tells you to verify authenticity a different way (request signatures). A fully accountable operator whose proof mechanism just doesn't fit IP-based recipes yet.

Not listed

SendGrid event webhooks

SendGrid's delivery traffic sends no distinctive User-Agent at all, and the operator documents neither stable IPs nor a verification method. There is nothing recognizable to list: a directory entry that can't match any traffic in your logs would be an empty promise.

Not listed

Screaming Frog SEO Spider

The opposite failure: a perfectly recognizable UA that doesn't identify anyone accountable. It's desktop software run from arbitrary machines — the UA names the software, not an operator responsible for the traffic — and it ships presets for impersonating other bots, making even the software claim advisory. No accountability chain, nothing to verify, no path to a tier.

One thing the gate deliberately does not judge: crawl volume or popularity. The directory verifies identity and accountability, not behavior — whether a heavy-but-honest crawler is welcome on your site is your policy call, and verified identity is what makes that call enforceable.

Put it to work

Read the Good Bot Practices standard entries are measured against, pull the machine-readable data (every verified IP range, UA pattern, and per-tier slice), or submit a bot — operators who publish verification move up tiers, as ClaudeBot, Datadog, and Grafana have all done.