ClickCease Which websites block AI bots? Robots.txt blocking by sector | Minelead Minelead - Free Email Finder & Verifier

Minelead web statisticsUpdated November 2026

Which websites block AI bots?

We read the robots.txt of 24.660.993 live business websites and recorded which of 1,050 known bots each one shuts out: AI crawlers, SEO tools, search engines and scrapers. Here is who blocks what, by sector, country, site popularity and CMS.

5,11%of sites with a robots.txt block at least one AI bot
4,75%block a bot that collects pages to train AI models
0,62%block a bot that fetches pages for AI answers and assistants
11,13%block at least one bot of any kind

14.503.084 of the 24.660.993 sites have a robots.txt; shares are of those sites.
How we measuredDownload the data (CSV)Free to use with a link to this page

The most blocked bots

Share of sites with a robots.txt that block each bot: solid for the whole site, striped for some pages only.

  1. 1 PetalBotSearch engine 5,88%+0,23% some pages
  2. 2 GPTBotAI 3,83%+6,01% some pages
  3. 3 BytespiderAI 3,82%+5,45% some pages
  4. 4 CCBotAI 3,8%+5,66% some pages
  5. 5 AmazonbotAI 3,57%+5,53% some pages
  6. 6 ClaudeBotAI 3,5%+5,95% some pages
  7. 7 Meta-ExternalAgentAI 3,47%+5,58% some pages
  8. 8 BaiduspiderSearch engine 3,45%+0,14% some pages
  9. 9 Google-ExtendedAI 3,34%+5,86% some pages
  10. 10 anthropic-aiAI 3,25%+5,5% some pages
  11. 11 Applebot-ExtendedAI 3,24%+5,67% some pages
  12. 12 cohere-aiAI 3,14%+5,41% some pages
  13. 13 omgiliAI 3,1%+5,17% some pages
  14. 14 DiffbotAI 3,01%+0,21% some pages
  15. 15 Claude-WebAI 2,99%+0,32% some pages
  16. 16 GoogleOtherAI 2,91%+5,27% some pages
  17. 17 ICC-CrawlerDownloader 2,84%+0,11% some pages
  18. 18 magpie-crawlerDownloader 2,82%+0,01% some pages
  19. 19 AwarioSmartBotSEO 2,79%+0,01% some pages
  20. 20 AwarioRssBotSEO 2,77%+0,01% some pages
  21. 21 Google-CloudVertexBotAI 2,73%+0,04% some pages
  22. 22 MJ12botSEO 2,49%+0,15% some pages
  23. 23 AhrefsBotSEO 2,16%+0,3% some pages
  24. 24 SemrushBotSEO 1,57%+0,19% some pages
  25. 25 dotbotSEO 1,56%+0,14% some pages
  26. 26 BLEXBotSEO 1,1%+0,07% some pages
  27. 27 NerdyBotOther 0,68%+0,0% some pages
  28. 28 FacebookBotAI 0,54%+5,27% some pages
  29. 29 NutchDownloader 0,52%+0,01% some pages
  30. 30 WebCopierDownloader 0,5%+0,0% some pages
  31. 31 rogerbotSEO 0,49%+0,03% some pages
  32. 32 ia_archiverArchive 0,49%+0,04% some pages
  33. 33 ExabotSearch engine 0,46%+0,04% some pages
  34. 34 turnitinbotAI 0,46%+0,01% some pages
  35. 35 YandexbotSearch engine 0,46%+0,1% some pages
  36. 36 barkrowlerSEO 0,45%+0,03% some pages
  37. 37 HTTrackDownloader 0,44%+0,0% some pages
  38. 38 WebZIPDownloader 0,44%+0,0% some pages
  39. 39 YandexSearch engine 0,43%+0,35% some pages
  40. 40 WebStripperDownloader 0,42%+0,0% some pages
  41. 41 DataForSeoBotSEO 0,42%+0,05% some pages
  42. 42 SiteSnaggerDownloader 0,41%+0,0% some pages
  43. 43 Offline ExplorerDownloader 0,38%+0,0% some pages
  44. 44 omgilibotAI 0,38%+5,18% some pages
  45. 45 serpstatbotSEO 0,37%+0,04% some pages
  46. 46 MSIECrawlerDownloader 0,37%+0,0% some pages
  47. 47 larbinDownloader 0,37%+0,0% some pages
  48. 48 WgetDownloader 0,37%+0,0% some pages
  49. 49 ChatGPT-UserAI 0,35%+0,86% some pages
  50. 50 Ai2BotAI 0,35%+5,22% some pages
  51. 51 MegaIndex.ruSEO 0,34%+0,0% some pages
  52. 52 SpbotDownloader 0,34%+0,0% some pages
  53. 53 TimpibotAI 0,33%+0,13% some pages
  54. 54 ImagesiftBotAI 0,33%+0,12% some pages
  55. 55 cohere-training-data-crawlerAI 0,31%+5,08% some pages
  56. 56 ZeusDownloader 0,3%+0,0% some pages
  57. 57 GigabotSearch engine 0,3%+0,0% some pages
  58. 58 ExtractorProEmail harvester 0,3%+0,0% some pages
  59. 59 PerplexityBotAI 0,3%+0,94% some pages
  60. 60 Ai2Bot-DolmaAI 0,29%+5,16% some pages

AI training vs. AI answers

AI companies run one bot to collect pages for training and others that fetch a page when someone asks their assistant or search. Of the sites that block a company's training bot, this many also block its answer bots.

OpenAI

9,4%

of the 555.250 sites blocking GPTBot also block OAI-SearchBot or ChatGPT-User

Anthropic

3,25%

of the 507.723 sites blocking ClaudeBot also block Claude-SearchBot or Claude-User

Perplexity

23,67%

of the 43.617 sites blocking PerplexityBot also block Perplexity-User

Meta

4,48%

of the 503.854 sites blocking Meta-ExternalAgent also block Meta-ExternalFetcher

Which sectors, CMS and countries block AI bots

Share of sites with a robots.txt that block at least one AI bot. A site counts in the sector of its best fitting business category.

By sector

The site's best fitting business category

  1. Automotive7,04%
  2. Blogs & Communities7,04%
  3. Technology & Software6,53%
  4. News & Media6,13%
  5. Arts, Entertainment & Recreation6,05%
  6. Nonprofits & Religion5,92%
  7. Real Estate5,81%
  8. Ecommerce & Retail5,76%
  9. Finance & Insurance5,44%
  10. Government & Public Services4,86%
  11. Travel & Hospitality4,73%
  12. Education & Research4,66%
  13. Transportation & Logistics4,43%
  14. Manufacturing & Industrial4,3%
  15. Marketing & Advertising4,26%
  16. Energy, Utilities & Agriculture4,08%
  17. Professional & Business Services4,05%
  18. Healthcare & Medical3,72%
  19. Legal Services3,67%
  20. Construction & Home Services3,66%
  21. Food & Drink3,61%
  22. Beauty, Wellness & Fitness3,56%

By CMS

The platform the site runs on

  1. WordPress3,31%
  2. Squarespace1,64%
  3. Shopify0,85%
  4. Wix0,85%
  5. Duda0,37%
  6. Next.js13,97%
  7. Joomla5,41%
  8. Webflow3,72%
  9. Astro15,8%
  10. Drupal5,25%
  11. Weebly0,26%
  12. Jimdo0,02%
  13. GoDaddy Website Builder0,5%
  14. TYPO3 CMS5,09%
  15. Framer Sites1,82%
  16. Nuxt.js9,47%
  17. PrestaShop7,48%
  18. Contao0,88%
  19. HubSpot CMS Hub1,26%
  20. Hostinger Website Builder0,54%

By site popularity

Sites ranked by links from other sites

  1. Top 10k18,36%
  2. 10k-100k12,59%
  3. 100k-1M7,19%
  4. Beyond 1M5,42%
  5. Unranked4,8%

By country

The countries with the most sites

  1. United States4,84%
  2. Germany3,4%
  3. United Kingdom4,88%
  4. France4,05%
  5. Netherlands3,84%
  6. Italy3,27%
  7. Brazil7,56%
  8. Australia4,75%
  9. Spain4,25%
  10. India5,65%
  11. Canada5,0%
  12. Switzerland3,91%
  13. Japan2,71%
  14. Poland5,6%
  15. Austria2,58%
  16. Belgium3,5%
  17. Sweden4,01%
  18. Czechia4,24%
  19. Russian Federation5,15%
  20. Türkiye9,45%
  21. China4,79%
  22. Mexico4,7%
  23. South Africa3,43%
  24. Denmark3,84%
  25. United Arab Emirates5,92%
  26. New Zealand5,19%
  27. Portugal4,17%
  28. Argentina15,12%
  29. Finland3,59%
  30. Indonesia13,73%

Every top bot, sector by sector

Share of each sector's sites blocking each bot from the whole site. Click a bot to sort the sectors by it.

Sector
Automotive6,14,14,84,54,53,74,65,03,33,44,14,14,14,03,12,93,03,03,02,92,84,13,42,52,1
Blogs & Communities6,45,65,25,94,95,04,94,84,85,14,74,74,64,54,74,34,14,54,24,24,22,41,91,51,9
Technology & Software6,95,55,65,65,15,15,14,95,15,05,04,94,94,94,84,74,74,74,74,74,61,41,10,91,1
News & Media7,04,94,64,84,34,34,13,44,34,14,03,83,83,43,33,43,02,92,92,92,92,32,01,51,6
Arts, Entertainment & Recreation7,85,04,74,94,64,64,54,04,44,24,34,04,13,83,63,73,53,53,53,43,53,22,92,02,1
Nonprofits & Religion7,93,44,94,64,92,92,92,92,82,82,72,72,62,42,52,52,42,32,32,32,32,21,91,71,1
Real Estate6,93,64,23,33,33,33,14,02,92,92,82,72,82,72,72,42,52,52,52,52,43,62,82,22,3
Ecommerce & Retail5,73,94,03,73,73,53,53,83,13,13,02,92,92,92,82,72,62,72,82,82,63,12,62,22,2
Finance & Insurance6,24,03,93,93,73,63,53,53,63,53,53,33,33,33,33,13,13,13,13,13,01,71,21,11,0
Government & Public Services6,53,73,63,73,33,43,34,93,23,03,12,92,92,82,82,72,72,72,72,72,63,23,02,02,3
Travel & Hospitality5,53,13,23,23,03,02,93,12,72,72,62,52,52,42,52,42,42,52,42,32,23,22,41,71,5
Education & Research6,63,83,63,53,43,33,33,33,22,93,02,82,82,62,62,62,52,52,42,42,42,72,51,91,6
Transportation & Logistics5,73,53,53,63,33,33,23,43,23,13,13,03,02,93,02,92,92,82,82,82,71,71,51,31,3
Manufacturing & Industrial5,33,23,13,13,02,92,93,02,82,72,72,62,62,62,52,52,52,52,42,42,42,11,91,61,5
Marketing & Advertising5,43,43,43,53,23,23,13,03,23,03,13,03,02,82,82,92,72,72,72,72,71,31,00,81,0
Energy, Utilities & Agriculture5,93,13,13,12,92,92,93,22,82,72,72,62,62,52,62,52,52,52,42,42,42,01,81,31,2
Professional & Business Services6,33,23,23,23,03,02,93,12,92,82,82,72,72,52,62,62,52,52,42,42,41,91,51,01,2
Healthcare & Medical5,62,72,92,92,52,62,52,52,52,42,42,32,32,22,22,22,12,12,02,02,02,01,71,11,1
Legal Services5,22,62,72,62,32,42,32,52,32,22,22,12,12,02,02,12,02,12,02,01,91,71,31,01,0
Construction & Home Services5,92,82,72,82,62,62,62,62,52,52,52,42,42,22,32,32,22,22,12,12,11,81,41,11,2
Food & Drink5,72,72,72,72,52,52,52,62,42,42,32,32,32,12,22,12,12,12,02,02,02,01,71,01,0
Beauty, Wellness & Fitness6,32,82,62,72,52,52,42,42,42,32,32,22,22,02,12,12,11,91,91,91,92,42,11,00,9

More from the same robots.txt files

6,8%of sites blocking AI bots use a copied block list of 10+ downloader bots: a template, not a site-by-site choice
5,22%ask every bot to slow down with a Crawl-delay
25,24%name at least one bot in their robots.txt, often in a list their website platform adds
9,94%of all sites have an llms.txt for AI tools, often added by their website platform

Find the sites behind these numbers

Lead Finder filters companies by the bots they block, their sector, country, CMS and technologies. See who blocks GPTBot in your market, then find their contacts.

Open Lead Finder

Use these numbers

You can use any number from this page in articles, reports, presentations and research. The one condition: credit Minelead with a link to this page.

Source: <a href="https://minelead.io/stats/ai-bot-blocking/">Minelead web statistics, November 2026</a>

How we measured

Sites
24.660.993 live business websites from Minelead's crawl, the latest read of each site as of November 17, 2026. Parked, spam and moved domains are left out. Sites whose robots.txt shut our own crawler out could not be read, so sites that block every bot are undercounted.
Blocked
A bot is blocked when a robots.txt group naming it disallows the whole site (Disallow: /) and no Allow line opens anything again. Sites that block a bot from some paths only are partial blocks, the striped part of each bar, and are left out of every other number on this page: most of them run a website platform whose default robots.txt lists the AI bots beside every other bot over its admin pages (Squarespace does this on every site). A bot named only to be allowed counts as neither, and rules for * (every bot) don't make a named bot blocked.
Parsing
User-agent lines in a row share the rules after them; an agent named in several groups gets all their rules; comments, byte-order marks and any line ending are handled. Version suffixes are folded: GPTBot/1.1 is GPTBot.
Bots
1,050 bots banned on 100 or more of the top million sites, each with a type (AI, SEO, search engine, downloader...). AI training bots: GPTBot, ClaudeBot, anthropic-ai, Claude-Web, CCBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, FacebookBot, Bytespider, Amazonbot, cohere-ai, cohere-training-data-crawler, Ai2Bot, Ai2Bot-Dolma, PanguBot, Diffbot, omgili, omgilibot, Timpibot, img2dataset, ImagesiftBot, Webzio-Extended, VelenPublicWebCrawler, FriendlyCrawler, Kangaroo Bot, TikTokSpider, Google-CloudVertexBot, iaskspider, Panscient, MyCentralAIScraperBot. AI answer bots (fetch a page for a person's question): OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User, Meta-ExternalFetcher, Gemini-Deep-Research, YouBot.
Sectors
Each of 2,487 business categories belongs to one of 22 sectors; a site counts in the sector of its best fitting category. Sectors, countries and platforms with fewer than 1,000 sites that have a robots.txt are not shown.
Updates
The numbers are recomputed every month from the latest crawl; earlier months stay for comparison. Our earlier study (July 2024, 223k top sites, full blocks only) is here.
Data
The CSV has every number on this page. Using them needs a link to this page (credit line).
Icons
Sector icons are from Lucide. Company logos and CMS logos are from Simple Icons and belong to their owners; the other bots' icons are Bottts by Pablo Stanley, made with DiceBear; flags are from flag-icons.
Suche...
×
Passwort festlegen
Sie haben Ihr Konto mit Google SSO erstellt. Sie müssen ein Passwort festlegen.

Telefonische Bestätigung

ANGEBOT NUR FÜR BEGRENZTE ZEIT

Verpasse es nicht

B2B-Leads in deiner Reichweite

Für neue Benutzer

Starte kostenlos & erhalte 15% Rabatt auf deinen ersten Plan

Melde dich in Sekunden an — keine Kreditkarte erforderlich. Wende den Code beim Bezahlen an, um deinen Rabatt freizuschalten.

MNLD15
Kostenlos registrieren →