Verified Bots Directory

Directory / AI crawlers

Meta Web Indexer

Verifiable

Meta's crawler that navigates the web to improve the quality of Meta AI search results, analysing page content for relevance and accuracy in Meta AI responses. Verified by ASN lookup (AS32934); Meta publishes no IP feed.

Operated by Meta · Official documentation

User agents

Patterns this directory matches, with real observed strings.

  • regexmeta-webindexer/\d
  • observedmeta-webindexer/1.1

How to verify

Look up the announced ranges by ASN

whois -h whois.radb.net -- '-i origin AS32934' | grep route

Good Bot Practices scorecard

  • Identifies honestly

    Stable UA token documented (1 pattern)

  • Verifiable

    Verifiable

  • Respects robots.txt

    Honors robots.txt (token: meta-webindexer)

  • Behaves

    Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset

  • Reachable operator

    Operator and documentation published

Measured against the Good Bot Practices.