Verified Bots Directory

Directory / Search engines

Google special-case crawlers

Fully verifiable

A group of Google crawlers that operate under separate agreements between the crawled site and a specific Google product (AdsBot, AdSense, Google Safety scanning, and similar). Several members of this group ignore the wildcard robots.txt rule and must be targeted explicitly.

Operated by Google · Official documentation

User agents

Patterns this directory matches, with real observed strings.

  • regexAdsBot-Google
  • regexMediapartners-Google
  • regexGoogle-Safety
  • observedAdsBot-Google (+http://www.google.com/adsbot.html)
  • observedMediapartners-Google
  • observedGoogle-Safety

How to verify

Fetch the official IP range feed

curl -s https://developers.google.com/static/search/apis/ipranges/special-crawlers.json

IP ranges (270)

Refreshed daily from the operator's feed. Also available as /data/ips/google-special-crawlers.ips and /data/bots/google-special-crawlers.json.

108.177.2.0/27
108.177.2.128/27
108.177.2.160/27
108.177.2.192/27
108.177.2.224/27
108.177.2.32/27
108.177.2.64/27
108.177.2.96/27
192.178.16.0/27
192.178.16.128/27
192.178.16.160/27
192.178.16.192/27
192.178.16.224/27
192.178.16.32/27
192.178.16.64/27
192.178.16.96/27
192.178.17.0/27
2001:4860:4801:2008::/64
2001:4860:4801:200c::/64
2001:4860:4801:200d::/64
2001:4860:4801:2010::/64
2001:4860:4801:2011::/64
2001:4860:4801:2012::/64
2001:4860:4801:2013::/64
2001:4860:4801:2014::/64
2001:4860:4801:2015::/64
2001:4860:4801:2016::/64
2001:4860:4801:2017::/64
2001:4860:4801:2018::/64
2001:4860:4801:2019::/64
2001:4860:4801:201a::/64
2001:4860:4801:201b::/64
2001:4860:4801:201c::/64
2001:4860:4801:201d::/64
2001:4860:4801:201e::/64
2001:4860:4801:201f::/64
2001:4860:4801:2020::/64
2001:4860:4801:2021::/64
2001:4860:4801:2022::/64
2001:4860:4801:2023::/64
2001:4860:4801:2024::/64
2001:4860:4801:2025::/64
2001:4860:4801:2026::/64
2001:4860:4801:2027::/64
2001:4860:4801:2028::/64
2001:4860:4801:2029::/64
2001:4860:4801:202a::/64
2001:4860:4801:202b::/64
2001:4860:4801:202c::/64
2001:4860:4801:202d::/64
2001:4860:4801:202e::/64
2001:4860:4801:202f::/64
2001:4860:4801:2030::/64
2001:4860:4801:2031::/64
2001:4860:4801:2032::/64
2001:4860:4801:2033::/64
2001:4860:4801:2034::/64
2001:4860:4801:2035::/64
2001:4860:4801:2036::/64
2001:4860:4801:2037::/64
2001:4860:4801:2038::/64
2001:4860:4801:2039::/64
2001:4860:4801:203a::/64
2001:4860:4801:203b::/64
2001:4860:4801:203c::/64
2001:4860:4801:203d::/64
2001:4860:4801:203e::/64
2001:4860:4801:203f::/64
2001:4860:4801:2040::/64
2001:4860:4801:2041::/64
2001:4860:4801:2042::/64
2001:4860:4801:2044::/64
2001:4860:4801:2045::/64
2001:4860:4801:2046::/64
2001:4860:4801:2047::/64
2001:4860:4801:2048::/64
2001:4860:4801:2049::/64
2001:4860:4801:204a::/64
2001:4860:4801:204b::/64
2001:4860:4801:204c::/64
2001:4860:4801:204d::/64
2001:4860:4801:204e::/64
2001:4860:4801:2050::/64
2001:4860:4801:2051::/64
2001:4860:4801:2052::/64
2001:4860:4801:2053::/64
2001:4860:4801:2054::/64
2001:4860:4801:2055::/64
2001:4860:4801:2056::/64
2001:4860:4801:2060::/64
2001:4860:4801:2061::/64
2001:4860:4801:2062::/64
2001:4860:4801:2063::/64
2001:4860:4801:2064::/64
2001:4860:4801:2065::/64
2001:4860:4801:2066::/64
2001:4860:4801:2067::/64
2001:4860:4801:2068::/64
2001:4860:4801:2069::/64
2001:4860:4801:206a::/64
2001:4860:4801:206b::/64
2001:4860:4801:206c::/64
2001:4860:4801:206d::/64
2001:4860:4801:206e::/64
2001:4860:4801:206f::/64
2001:4860:4801:2070::/64
2001:4860:4801:2071::/64
2001:4860:4801:2072::/64
2001:4860:4801:2073::/64
2001:4860:4801:2074::/64
2001:4860:4801:2075::/64
2001:4860:4801:2076::/64
2001:4860:4801:2077::/64
2001:4860:4801:2078::/64
2001:4860:4801:2079::/64
2001:4860:4801:207a::/64
2001:4860:4801:207b::/64
2001:4860:4801:207c::/64
2001:4860:4801:207d::/64
2001:4860:4801:207e::/64
2001:4860:4801:207f::/64
2001:4860:4801:2080::/64
2001:4860:4801:2081::/64
2001:4860:4801:2082::/64
2001:4860:4801:2083::/64
2001:4860:4801:2084::/64
2001:4860:4801:2085::/64
2001:4860:4801:2086::/64
2001:4860:4801:2087::/64
2001:4860:4801:2088::/64
2001:4860:4801:2090::/64
2001:4860:4801:2091::/64
2001:4860:4801:2092::/64
2001:4860:4801:2093::/64
2001:4860:4801:2094::/64
2001:4860:4801:2095::/64
2001:4860:4801:2096::/64
2001:4860:4801:2097::/64
2001:4860:4801:20a0::/64
2001:4860:4801:20a1::/64
2001:4860:4801:20a2::/64
2001:4860:4801:20a3::/64
2001:4860:4801:20a4::/64
2001:4860:4801:20a5::/64
2001:4860:4801:20a6::/64
2001:4860:4801:20b0::/64
2001:4860:4801:20b1::/64
2001:4860:4801:20b2::/64
2001:4860:4801:20b3::/64
2001:4860:4801:20b4::/64
2001:4860:4801:20b5::/64
2001:4860:4801:20b6::/64
209.85.238.0/27
209.85.238.128/27
209.85.238.160/27
209.85.238.192/27
209.85.238.224/27
209.85.238.32/27
209.85.238.64/27
209.85.238.96/27
66.249.87.0/27
66.249.87.128/27
66.249.87.160/27
66.249.87.192/27
66.249.87.224/27
66.249.87.32/27
66.249.87.64/27
66.249.87.96/27
66.249.89.0/27
66.249.89.128/27
66.249.89.160/27
66.249.89.192/27
66.249.89.224/27
66.249.89.32/27
66.249.89.64/27
66.249.89.96/27
66.249.90.0/27
66.249.90.128/27
66.249.90.160/27
66.249.90.192/27
66.249.90.224/27
66.249.90.32/27
66.249.90.64/27
66.249.90.96/27
66.249.91.0/27
66.249.91.128/27
66.249.91.160/27
66.249.91.192/27
66.249.91.224/27
66.249.91.32/27
66.249.91.64/27
66.249.91.96/27
66.249.92.0/27
66.249.92.128/27
66.249.92.160/27
66.249.92.192/27
66.249.92.224/27
66.249.92.32/27
66.249.92.64/27
66.249.92.96/27
72.14.199.128/27
72.14.199.160/27
72.14.199.192/27
72.14.199.224/27
72.14.199.32/27
72.14.199.64/27
72.14.199.96/27
74.125.148.0/27
74.125.148.128/27
74.125.148.160/27
74.125.148.192/27
74.125.148.224/27
74.125.148.32/27
74.125.148.64/27
74.125.148.96/27
74.125.149.0/27
74.125.149.128/27
74.125.149.160/27
74.125.149.192/27
74.125.149.224/27
74.125.149.32/27
74.125.149.64/27
74.125.149.96/27
74.125.150.0/27
74.125.150.128/27
74.125.150.160/27
74.125.150.192/27
74.125.150.32/27
74.125.150.64/27
74.125.150.96/27
74.125.151.0/27
74.125.151.128/27
74.125.151.160/27
74.125.151.192/27
74.125.151.224/27
74.125.151.32/27
74.125.151.64/27
74.125.151.96/27
74.125.216.0/27
74.125.216.128/27
74.125.216.160/27
74.125.216.192/27
74.125.216.224/27
74.125.216.32/27
74.125.216.64/27
74.125.216.96/27
74.125.217.0/27
74.125.217.128/27
74.125.217.160/27
74.125.217.192/27
74.125.217.224/27
74.125.217.32/27
74.125.217.64/27
74.125.217.96/27
74.125.218.0/27
74.125.218.128/27
74.125.218.160/27
74.125.218.192/27
74.125.218.224/27
74.125.218.32/27
74.125.218.64/27
74.125.218.96/27
74.125.219.0/27
74.125.219.128/27
74.125.219.160/27
74.125.219.192/27
74.125.219.224/27
74.125.219.32/27
74.125.219.64/27
74.125.219.96/27

Good Bot Practices scorecard

  • Identifies honestly

    Stable UA token documented (3 patterns)

  • Verifiable

    Fully verifiable

  • Respects robots.txt

    Does not honor robots.txt

  • Behaves

    Crawl-rate behavior is operator-declared; not machine-verifiable from this dataset

  • Reachable operator

    Operator and documentation published

Measured against the Good Bot Practices.