Verified Bots Directory

Directory / AI crawlers

CCBot

Fully verifiable

Common Crawl's crawler that builds the freely available Common Crawl web archive, widely reused as AI training data by third parties.

Operated by Common Crawl Foundation · Official documentation

User agents

Patterns this directory matches, with real observed strings.

  • regexCCBot/\d
  • observedCCBot/2.0 (https://commoncrawl.org/faq/)

How to verify

Fetch the official IP range feed

curl -s https://index.commoncrawl.org/ccbot.json

Reverse-DNS check a client IP

dig -x <client-ip> +short   # expect a hostname matching *.crawl.commoncrawl.org

Reverse-DNS suffixes: *.crawl.commoncrawl.org

IP ranges (6)

Refreshed daily from the operator's feed. Also available as /data/ips/ccbot.ips and /data/bots/ccbot.json.

18.97.14.80/29
18.97.14.88/30
18.97.9.168/29
2600:1f28:365:8000::/56
3.41.188.32/29
98.85.178.216/32

Good Bot Practices scorecard

  • Identifies honestly

    Stable UA token documented (1 pattern)

  • Verifiable

    Fully verifiable

  • Respects robots.txt

    Honors robots.txt (token: CCBot)

  • Behaves

    Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset

  • Reachable operator

    Operator and documentation published

Measured against the Good Bot Practices.