Directory / AI crawlers

Meta External Agent

Verifiable

Meta's crawler for AI training data and content indexing across Meta products. Verified by ASN lookup (AS32934); Meta publishes no IP feed.

Operated by Meta · Official documentation

User agents

Patterns this directory matches, with real observed strings.

  • meta-externalagent/\d
  • meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

How to verify

Look up the announced ranges by ASN

whois -h whois.radb.net -- '-i origin AS32934' | grep route

Good Bot Practices scorecard

  • Identifies honestly

    Stable UA token documented (1 pattern)

  • Verifiable

    Verifiable

  • Respects robots.txt

    Honors robots.txt (token: meta-externalagent)

  • Behaves

    Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset

  • Reachable operator

    Operator and documentation published

Measured against the Good Bot Practices.