VelenPublicWebCrawler

Listed only

Hunter's public web crawler, written in Go. It analyses millions of publicly accessible pages every month to build the business datasets and machine learning models behind Hunter's products, and never fetches anything behind a login. The operator documents a deliberate rate limit of one page at a time and one page every two seconds per site.

Operated by Hunter · Official documentation

User agents

Patterns this directory matches, with real observed strings.

regexVelenPublicWebCrawler
observedMozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io)
observedVelenPublicWebCrawler (velen.io)

How to verify

No verification recipe published by the operator.

Good Bot Practices scorecard

Identifies honestly
Stable UA token documented (1 pattern)
Verifiable
Operator publishes no verification path
Respects robots.txt
Honors robots.txt (token: VelenPublicWebCrawler)
Behaves
Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset
Reachable operator
Operator and documentation published

Measured against the Good Bot Practices.