Panscient Crawler
Listed onlyPanscient's large-scale crawler, which traverses public websites so that Panscient can build structured company and professional data feeds licensed to enterprise customers. The operator documents a full-corpus refresh each quarter, a rate limit of at most one request per second to any single domain, and compliance with the Robot Exclusion Standard. A separate "pantest" agent is used for testing. No IP ranges are published.
Operated by Panscient Inc. · Official documentation
User agents
Patterns this directory matches, with real observed strings.
- regexpanscient\.com
- regexpantest
- observedpanscient.com
- observedpantest
How to verify
No verification recipe published by the operator.
Good Bot Practices scorecard
- Identifies honestly
Stable UA token documented (2 patterns)
- Verifiable
Operator publishes no verification path
- Respects robots.txt
Honors robots.txt (token: panscient.com)
- Behaves
Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset
- Reachable operator
Operator and documentation published
Measured against the Good Bot Practices.