Diffbot

Listed only

Diffbot's general-purpose web crawler, used to build its Knowledge Graph and power its structured-extraction APIs. Diffbot documents robots.txt behavior but publishes no IP list, ASN, or reverse-DNS pattern.

Operated by Diffbot · Official documentation

User agents

Patterns this directory matches, with real observed strings.

regexDiffbot/\d
observedMozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)

How to verify

No verification recipe published by the operator.

Good Bot Practices scorecard

Identifies honestly
Stable UA token documented (1 pattern)
Verifiable
Operator publishes no verification path
Respects robots.txt
Honors robots.txt (token: Diffbot)
Behaves
Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset
Reachable operator
Operator and documentation published

Measured against the Good Bot Practices.