AIWebIndex

Fully verifiable

The crawler behind Lyrenth, an index of the public web that serves each page to software agents as a single structured "AIDocument". The operator states the crawler is an indexer and not a model trainer, that it implements RFC 9309 robots.txt including Crawl-delay, Sitemap and HTTP 429/503 backoff, that it fails closed when robots.txt cannot be read, and that it fetches at most once every two seconds per domain across the whole fleet. The bot page offers three independent ways to confirm a request: a published address feed, Web Bot Auth request signatures, and forward-confirmed reverse DNS under lyrenth.com.

Operated by Aleksma Ai, Inc. (Lyrenth) · Official documentation

User agents

Patterns this directory matches, with real observed strings.

regexAIWebIndex/\d
observedAIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)
observedAIWebIndex/2.0 verification (+https://lyrenth.com/bot)

How to verify

Fetch the official IP range feed

curl -s https://lyrenth.com/bot/ip-ranges.json

Fetch the Web Bot Auth key directory

curl -s https://api.lyrenth.com/.well-known/http-message-signatures-directory

Reverse-DNS check a client IP

dig -x <client-ip> +short   # expect a hostname matching *.lyrenth.com

Reverse-DNS suffixes: *.lyrenth.com

IP ranges (22)

Refreshed daily from the operator's feed. Also available as /data/ips/aiwebindex.ips and /data/bots/aiwebindex.json.

138.201.157.96/32
142.132.225.162/32
152.53.93.229/32
159.69.116.231/32
159.69.189.54/32
168.119.98.242/32
178.104.212.30/32
178.105.121.172/32
178.105.146.59/32
178.105.195.233/32
178.105.197.214/32
178.105.200.108/32
178.105.207.129/32
178.105.210.26/32
178.105.220.145/32
178.105.223.90/32
178.105.66.244/32
178.105.70.116/32
178.105.9.229/32
46.224.73.201/32
46.225.147.220/32
49.13.94.158/32

Good Bot Practices scorecard

Identifies honestly
Stable UA token documented (1 pattern)
Verifiable
Fully verifiable
Respects robots.txt
Honors robots.txt (token: AIWebIndex)
Behaves
Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset
Reachable operator
Operator and documentation published

Measured against the Good Bot Practices.