gwf-carbon-txt-crawler

Listed only

The Green Web Foundation's annual crawl measuring adoption of carbon.txt, an open standard for publishing sustainability data about a website and its hosting. Once a year it works through the top 100 million hostnames in the Common Crawl web graph, making a small number of requests per host: robots.txt, the root page, the two carbon.txt locations, and any URL named in a carbon.txt DNS record or HTTP header. The operator states it crawls no other pages and stores nothing but the carbon.txt files it finds.

Operated by Green Web Foundation · Official documentation

User agents

Patterns this directory matches, with real observed strings.

regexgwf-carbon-txt-crawler
observedgwf-carbon-txt-crawler

How to verify

No verification recipe published by the operator.

Good Bot Practices scorecard

Identifies honestly
Stable UA token documented (1 pattern)
Verifiable
Operator publishes no verification path
Respects robots.txt
Service traffic you configured — robots.txt does not apply
Behaves
Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset
Reachable operator
Operator and documentation published

Measured against the Good Bot Practices.