DigitalOcean Gradient AI Website Crawler
Listed onlyThe crawler DigitalOcean runs when a customer adds a website URL as a data source for a DigitalOcean Knowledge Base, which indexes the content so it can power retrieval-augmented generation for that customer's own agents. The documentation states the crawler indexes up to 5,500 pages per job and skips inaccessible or disallowed links to keep indexing jobs bounded, follows HTML links depending on the behaviour the customer selects, indexes text and certain image types, and ignores videos and navigation links.
Operated by DigitalOcean, LLC · Official documentation
User agents
Patterns this directory matches, with real observed strings.
- regexDigitalOceanGradientAICrawler
- observedDigitalOceanGradientAICrawler/1.0
How to verify
No verification recipe published by the operator.
Good Bot Practices scorecard
- Identifies honestly
Stable UA token documented (1 pattern)
- Verifiable
Operator publishes no verification path
- Respects robots.txt
Honors robots.txt
- Behaves
Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset
- Reachable operator
Operator and documentation published
Measured against the Good Bot Practices.