Verified Bots Directory

Directory / Search engines

SemanticScholarBot

Listed only

SemanticScholarBot is the crawler for Semantic Scholar, an academic search engine built by the Allen Institute for AI. It crawls the web to find scholarly PDFs and metadata for indexing.

Operated by Allen Institute for AI (Ai2) · Official documentation

User agents

Patterns this directory matches, with real observed strings.

  • regexSemanticScholarBot
  • observedSemanticScholarBot/1.0 (+http://s2.allenai.org/bot.html)
  • observedMozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)

How to verify

No verification recipe published by the operator.

Good Bot Practices scorecard

  • Identifies honestly

    Stable UA token documented (1 pattern)

  • Verifiable

    Operator publishes no verification path

  • Respects robots.txt

    Does not honor robots.txt

  • Behaves

    Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset

  • Reachable operator

    Operator and documentation published

Measured against the Good Bot Practices.