AI2Bot

Listed only

The Allen Institute for AI's crawler, used to build open training datasets such as Dolma. It crawls selected domains for web content that is then used to train open language models. Ai2 runs it under two user agents, a plain AI2Bot and an Ai2Bot-Dolma variant that names the dataset, both pointing at the same crawling notice. Ai2 publishes no IP ranges, ASN, or reverse-DNS pattern, and its crawling notice offers user-agent filtering as the only documented control.

Operated by Allen Institute for AI · Official documentation

User agents

Patterns this directory matches, with real observed strings.

regexAI2Bot
regexAi2Bot-Dolma
observedMozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)
observedMozilla/5.0 (compatible) Ai2Bot-Dolma (+https://www.allenai.org/crawler)

How to verify

No verification recipe published by the operator.

Good Bot Practices scorecard

Identifies honestly
Stable UA token documented (2 patterns)
Verifiable
Operator publishes no verification path
Respects robots.txt
Does not honor robots.txt
Behaves
Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset
Reachable operator
Operator and documentation published

Measured against the Good Bot Practices.