Minelead web statisticsUpdated noviembre 2026
Which websites block AI bots?
We read the robots.txt of 24 660 993 live business websites and recorded which of 1,050 known bots each one shuts out: AI crawlers, SEO tools, search engines and scrapers. Here is who blocks what, by sector, country, site popularity and CMS.
14 503 084 of the 24 660 993 sites have a robots.txt; shares are of those sites.
How we measuredDownload the data (CSV)Free to use with a link to this page
The most blocked bots
Share of sites with a robots.txt that block each bot: solid for the whole site, striped for some pages only.
-
1
PetalBotSearch engine 5,88%+0,23% some pages
-
2
GPTBotAI 3,83%+6,01% some pages
-
3
BytespiderAI 3,82%+5,45% some pages
-
4
CCBotAI 3,8%+5,66% some pages
-
5
AmazonbotAI 3,57%+5,53% some pages
-
6
ClaudeBotAI 3,5%+5,95% some pages
-
7
Meta-ExternalAgentAI 3,47%+5,58% some pages
-
8
BaiduspiderSearch engine 3,45%+0,14% some pages
-
9
Google-ExtendedAI 3,34%+5,86% some pages
-
10
anthropic-aiAI 3,25%+5,5% some pages
-
11
Applebot-ExtendedAI 3,24%+5,67% some pages
-
12
cohere-aiAI 3,14%+5,41% some pages
-
13
omgiliAI 3,1%+5,17% some pages
-
14
DiffbotAI 3,01%+0,21% some pages
-
15
Claude-WebAI 2,99%+0,32% some pages
-
16
GoogleOtherAI 2,91%+5,27% some pages
-
17
ICC-CrawlerDownloader 2,84%+0,11% some pages
-
18
magpie-crawlerDownloader 2,82%+0,01% some pages
-
19
AwarioSmartBotSEO 2,79%+0,01% some pages
-
20
AwarioRssBotSEO 2,77%+0,01% some pages
-
21
Google-CloudVertexBotAI 2,73%+0,04% some pages
-
22
MJ12botSEO 2,49%+0,15% some pages
-
23
AhrefsBotSEO 2,16%+0,3% some pages
-
24
SemrushBotSEO 1,57%+0,19% some pages
-
25
dotbotSEO 1,56%+0,14% some pages
-
26
BLEXBotSEO 1,1%+0,07% some pages
-
27
NerdyBotOther 0,68%+0,0% some pages
-
28
FacebookBotAI 0,54%+5,27% some pages
-
29
NutchDownloader 0,52%+0,01% some pages
-
30
WebCopierDownloader 0,5%+0,0% some pages
-
31
rogerbotSEO 0,49%+0,03% some pages
-
32
ia_archiverArchive 0,49%+0,04% some pages
-
33
ExabotSearch engine 0,46%+0,04% some pages
-
34
turnitinbotAI 0,46%+0,01% some pages
-
35
YandexbotSearch engine 0,46%+0,1% some pages
-
36
barkrowlerSEO 0,45%+0,03% some pages
-
37
HTTrackDownloader 0,44%+0,0% some pages
-
38
WebZIPDownloader 0,44%+0,0% some pages
-
39
YandexSearch engine 0,43%+0,35% some pages
-
40
WebStripperDownloader 0,42%+0,0% some pages
-
41
DataForSeoBotSEO 0,42%+0,05% some pages
-
42
SiteSnaggerDownloader 0,41%+0,0% some pages
-
43
Offline ExplorerDownloader 0,38%+0,0% some pages
-
44
omgilibotAI 0,38%+5,18% some pages
-
45
serpstatbotSEO 0,37%+0,04% some pages
-
46
MSIECrawlerDownloader 0,37%+0,0% some pages
-
47
larbinDownloader 0,37%+0,0% some pages
-
48
WgetDownloader 0,37%+0,0% some pages
-
49
ChatGPT-UserAI 0,35%+0,86% some pages
-
50
Ai2BotAI 0,35%+5,22% some pages
-
51
MegaIndex.ruSEO 0,34%+0,0% some pages
-
52
SpbotDownloader 0,34%+0,0% some pages
-
53
TimpibotAI 0,33%+0,13% some pages
-
54
ImagesiftBotAI 0,33%+0,12% some pages
-
55
cohere-training-data-crawlerAI 0,31%+5,08% some pages
-
56
ZeusDownloader 0,3%+0,0% some pages
-
57
GigabotSearch engine 0,3%+0,0% some pages
-
58
ExtractorProEmail harvester 0,3%+0,0% some pages
-
59
PerplexityBotAI 0,3%+0,94% some pages
-
60
Ai2Bot-DolmaAI 0,29%+5,16% some pages
AI training vs. AI answers
AI companies run one bot to collect pages for training and others that fetch a page when someone asks their assistant or search. Of the sites that block a company's training bot, this many also block its answer bots.
OpenAI
of the 555 250 sites blocking GPTBot also block OAI-SearchBot or ChatGPT-User
Anthropic
of the 507 723 sites blocking ClaudeBot also block Claude-SearchBot or Claude-User
Perplexity
of the 43 617 sites blocking PerplexityBot also block Perplexity-User
Meta
of the 503 854 sites blocking Meta-ExternalAgent also block Meta-ExternalFetcher
Which sectors, CMS and countries block AI bots
Share of sites with a robots.txt that block at least one AI bot. A site counts in the sector of its best fitting business category.
By sector
The site's best fitting business category
By CMS
The platform the site runs on
By site popularity
Sites ranked by links from other sites
By country
The countries with the most sites
Every top bot, sector by sector
Share of each sector's sites blocking each bot from the whole site. Click a bot to sort the sectors by it.
| Sector | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 6,1 | 4,1 | 4,8 | 4,5 | 4,5 | 3,7 | 4,6 | 5,0 | 3,3 | 3,4 | 4,1 | 4,1 | 4,1 | 4,0 | 3,1 | 2,9 | 3,0 | 3,0 | 3,0 | 2,9 | 2,8 | 4,1 | 3,4 | 2,5 | 2,1 | |
| 6,4 | 5,6 | 5,2 | 5,9 | 4,9 | 5,0 | 4,9 | 4,8 | 4,8 | 5,1 | 4,7 | 4,7 | 4,6 | 4,5 | 4,7 | 4,3 | 4,1 | 4,5 | 4,2 | 4,2 | 4,2 | 2,4 | 1,9 | 1,5 | 1,9 | |
| 6,9 | 5,5 | 5,6 | 5,6 | 5,1 | 5,1 | 5,1 | 4,9 | 5,1 | 5,0 | 5,0 | 4,9 | 4,9 | 4,9 | 4,8 | 4,7 | 4,7 | 4,7 | 4,7 | 4,7 | 4,6 | 1,4 | 1,1 | 0,9 | 1,1 | |
| 7,0 | 4,9 | 4,6 | 4,8 | 4,3 | 4,3 | 4,1 | 3,4 | 4,3 | 4,1 | 4,0 | 3,8 | 3,8 | 3,4 | 3,3 | 3,4 | 3,0 | 2,9 | 2,9 | 2,9 | 2,9 | 2,3 | 2,0 | 1,5 | 1,6 | |
| 7,8 | 5,0 | 4,7 | 4,9 | 4,6 | 4,6 | 4,5 | 4,0 | 4,4 | 4,2 | 4,3 | 4,0 | 4,1 | 3,8 | 3,6 | 3,7 | 3,5 | 3,5 | 3,5 | 3,4 | 3,5 | 3,2 | 2,9 | 2,0 | 2,1 | |
| 7,9 | 3,4 | 4,9 | 4,6 | 4,9 | 2,9 | 2,9 | 2,9 | 2,8 | 2,8 | 2,7 | 2,7 | 2,6 | 2,4 | 2,5 | 2,5 | 2,4 | 2,3 | 2,3 | 2,3 | 2,3 | 2,2 | 1,9 | 1,7 | 1,1 | |
| 6,9 | 3,6 | 4,2 | 3,3 | 3,3 | 3,3 | 3,1 | 4,0 | 2,9 | 2,9 | 2,8 | 2,7 | 2,8 | 2,7 | 2,7 | 2,4 | 2,5 | 2,5 | 2,5 | 2,5 | 2,4 | 3,6 | 2,8 | 2,2 | 2,3 | |
| 5,7 | 3,9 | 4,0 | 3,7 | 3,7 | 3,5 | 3,5 | 3,8 | 3,1 | 3,1 | 3,0 | 2,9 | 2,9 | 2,9 | 2,8 | 2,7 | 2,6 | 2,7 | 2,8 | 2,8 | 2,6 | 3,1 | 2,6 | 2,2 | 2,2 | |
| 6,2 | 4,0 | 3,9 | 3,9 | 3,7 | 3,6 | 3,5 | 3,5 | 3,6 | 3,5 | 3,5 | 3,3 | 3,3 | 3,3 | 3,3 | 3,1 | 3,1 | 3,1 | 3,1 | 3,1 | 3,0 | 1,7 | 1,2 | 1,1 | 1,0 | |
| 6,5 | 3,7 | 3,6 | 3,7 | 3,3 | 3,4 | 3,3 | 4,9 | 3,2 | 3,0 | 3,1 | 2,9 | 2,9 | 2,8 | 2,8 | 2,7 | 2,7 | 2,7 | 2,7 | 2,7 | 2,6 | 3,2 | 3,0 | 2,0 | 2,3 | |
| 5,5 | 3,1 | 3,2 | 3,2 | 3,0 | 3,0 | 2,9 | 3,1 | 2,7 | 2,7 | 2,6 | 2,5 | 2,5 | 2,4 | 2,5 | 2,4 | 2,4 | 2,5 | 2,4 | 2,3 | 2,2 | 3,2 | 2,4 | 1,7 | 1,5 | |
| 6,6 | 3,8 | 3,6 | 3,5 | 3,4 | 3,3 | 3,3 | 3,3 | 3,2 | 2,9 | 3,0 | 2,8 | 2,8 | 2,6 | 2,6 | 2,6 | 2,5 | 2,5 | 2,4 | 2,4 | 2,4 | 2,7 | 2,5 | 1,9 | 1,6 | |
| 5,7 | 3,5 | 3,5 | 3,6 | 3,3 | 3,3 | 3,2 | 3,4 | 3,2 | 3,1 | 3,1 | 3,0 | 3,0 | 2,9 | 3,0 | 2,9 | 2,9 | 2,8 | 2,8 | 2,8 | 2,7 | 1,7 | 1,5 | 1,3 | 1,3 | |
| 5,3 | 3,2 | 3,1 | 3,1 | 3,0 | 2,9 | 2,9 | 3,0 | 2,8 | 2,7 | 2,7 | 2,6 | 2,6 | 2,6 | 2,5 | 2,5 | 2,5 | 2,5 | 2,4 | 2,4 | 2,4 | 2,1 | 1,9 | 1,6 | 1,5 | |
| 5,4 | 3,4 | 3,4 | 3,5 | 3,2 | 3,2 | 3,1 | 3,0 | 3,2 | 3,0 | 3,1 | 3,0 | 3,0 | 2,8 | 2,8 | 2,9 | 2,7 | 2,7 | 2,7 | 2,7 | 2,7 | 1,3 | 1,0 | 0,8 | 1,0 | |
| 5,9 | 3,1 | 3,1 | 3,1 | 2,9 | 2,9 | 2,9 | 3,2 | 2,8 | 2,7 | 2,7 | 2,6 | 2,6 | 2,5 | 2,6 | 2,5 | 2,5 | 2,5 | 2,4 | 2,4 | 2,4 | 2,0 | 1,8 | 1,3 | 1,2 | |
| 6,3 | 3,2 | 3,2 | 3,2 | 3,0 | 3,0 | 2,9 | 3,1 | 2,9 | 2,8 | 2,8 | 2,7 | 2,7 | 2,5 | 2,6 | 2,6 | 2,5 | 2,5 | 2,4 | 2,4 | 2,4 | 1,9 | 1,5 | 1,0 | 1,2 | |
| 5,6 | 2,7 | 2,9 | 2,9 | 2,5 | 2,6 | 2,5 | 2,5 | 2,5 | 2,4 | 2,4 | 2,3 | 2,3 | 2,2 | 2,2 | 2,2 | 2,1 | 2,1 | 2,0 | 2,0 | 2,0 | 2,0 | 1,7 | 1,1 | 1,1 | |
| 5,2 | 2,6 | 2,7 | 2,6 | 2,3 | 2,4 | 2,3 | 2,5 | 2,3 | 2,2 | 2,2 | 2,1 | 2,1 | 2,0 | 2,0 | 2,1 | 2,0 | 2,1 | 2,0 | 2,0 | 1,9 | 1,7 | 1,3 | 1,0 | 1,0 | |
| 5,9 | 2,8 | 2,7 | 2,8 | 2,6 | 2,6 | 2,6 | 2,6 | 2,5 | 2,5 | 2,5 | 2,4 | 2,4 | 2,2 | 2,3 | 2,3 | 2,2 | 2,2 | 2,1 | 2,1 | 2,1 | 1,8 | 1,4 | 1,1 | 1,2 | |
| 5,7 | 2,7 | 2,7 | 2,7 | 2,5 | 2,5 | 2,5 | 2,6 | 2,4 | 2,4 | 2,3 | 2,3 | 2,3 | 2,1 | 2,2 | 2,1 | 2,1 | 2,1 | 2,0 | 2,0 | 2,0 | 2,0 | 1,7 | 1,0 | 1,0 | |
| 6,3 | 2,8 | 2,6 | 2,7 | 2,5 | 2,5 | 2,4 | 2,4 | 2,4 | 2,3 | 2,3 | 2,2 | 2,2 | 2,0 | 2,1 | 2,1 | 2,1 | 1,9 | 1,9 | 1,9 | 1,9 | 2,4 | 2,1 | 1,0 | 0,9 |
More from the same robots.txt files
Find the sites behind these numbers
Lead Finder filters companies by the bots they block, their sector, country, CMS and technologies. See who blocks GPTBot in your market, then find their contacts.
Use these numbers
You can use any number from this page in articles, reports, presentations and research. The one condition: credit Minelead with a link to this page.
Source: <a href="https://minelead.io/stats/ai-bot-blocking/">Minelead web statistics, noviembre 2026</a>
How we measured
- Sites
- 24 660 993 live business websites from Minelead's crawl, the latest read of each site as of noviembre 17, 2026. Parked, spam and moved domains are left out. Sites whose robots.txt shut our own crawler out could not be read, so sites that block every bot are undercounted.
- Blocked
- A bot is blocked when a robots.txt group naming it disallows the whole site (
Disallow: /) and noAllowline opens anything again. Sites that block a bot from some paths only are partial blocks, the striped part of each bar, and are left out of every other number on this page: most of them run a website platform whose default robots.txt lists the AI bots beside every other bot over its admin pages (Squarespace does this on every site). A bot named only to be allowed counts as neither, and rules for*(every bot) don't make a named bot blocked. - Parsing
- User-agent lines in a row share the rules after them; an agent named in several groups gets all their rules; comments, byte-order marks and any line ending are handled. Version suffixes are folded:
GPTBot/1.1is GPTBot. - Bots
- 1,050 bots banned on 100 or more of the top million sites, each with a type (AI, SEO, search engine, downloader...). AI training bots: GPTBot, ClaudeBot, anthropic-ai, Claude-Web, CCBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, FacebookBot, Bytespider, Amazonbot, cohere-ai, cohere-training-data-crawler, Ai2Bot, Ai2Bot-Dolma, PanguBot, Diffbot, omgili, omgilibot, Timpibot, img2dataset, ImagesiftBot, Webzio-Extended, VelenPublicWebCrawler, FriendlyCrawler, Kangaroo Bot, TikTokSpider, Google-CloudVertexBot, iaskspider, Panscient, MyCentralAIScraperBot. AI answer bots (fetch a page for a person's question): OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User, Meta-ExternalFetcher, Gemini-Deep-Research, YouBot.
- Sectors
- Each of 2,487 business categories belongs to one of 22 sectors; a site counts in the sector of its best fitting category. Sectors, countries and platforms with fewer than 1,000 sites that have a robots.txt are not shown.
- Updates
- The numbers are recomputed every month from the latest crawl; earlier months stay for comparison. Our earlier study (July 2024, 223k top sites, full blocks only) is here.
- Data
- The CSV has every number on this page. Using them needs a link to this page (credit line).
- Icons
- Sector icons are from Lucide. Company logos and CMS logos are from Simple Icons and belong to their owners; the other bots' icons are Bottts by Pablo Stanley, made with DiceBear; flags are from flag-icons.