bl.uk_lddc_bot
Listed onlyThe web crawler of the UK legal deposit libraries, which harvests UK-published websites so that the British Library and the other five legal deposit libraries can archive online publications as they are entitled to do under legal deposit. It runs on Heritrix and is throttled to avoid taking too much bandwidth or overloading a site. A separate rendering component identifies itself as bl.uk_lddc_renderbot. The libraries state the crawler generally follows the robots.txt exclusion protocol and normally respects robots META tags, but that they may choose to overrule robots.txt in order to harvest content successfully — for example to collect JavaScript or CSS that robots.txt excludes. No IP ranges are published, so requests cannot be verified; the guidance for publishers who want to stop the crawl is to name it in robots.txt.
Operated by The British Library (UK legal deposit libraries) · Official documentation
User agents
Patterns this directory matches, with real observed strings.
- regexbl\.uk_lddc_bot
- regexbl\.uk_lddc_renderbot
- observedbl.uk_lddc_bot
- observedMozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 bl.uk_lddc_renderbot/2.0.0 (+ http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)
How to verify
No verification recipe published by the operator.
Good Bot Practices scorecard
- Identifies honestly
Stable UA token documented (2 patterns)
- Verifiable
Operator publishes no verification path
- Respects robots.txt
Honors robots.txt (token: bl.uk_lddc_bot)
- Behaves
Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset
- Reachable operator
Operator and documentation published
Measured against the Good Bot Practices.