User-Agent Token
Claude-WebRespects robots.txt
UnknownImpact Level
Notable
10M+ users - Regional/specialized AI companies
Estimated Reach
10M+ users - Regional/specialized AI companies
🎯What is Claude-Web?
Claude-Web is a web crawler operated by Anthropic. Purpose not documented.
🚫 What Happens If You Block
Unknown - purpose not documented.
💡 Good to Know
May be deprecated or internal. Block if concerned about unknown crawling.
🏢About Anthropic
🛡️Claude-Web robots.txt Configuration
Control Claude-Web access to your website using robots.txt directives.
Block Claude-Web
To completely block Claude-Web from crawling your site:
User-agent: Claude-Web
Disallow: /Allow Claude-Web Full Access
To explicitly allow Claude-Web to crawl your entire site:
User-agent: Claude-Web
Allow: /Selective Access for Claude-Web
To allow Claude-Web but restrict certain directories:
User-agent: Claude-Web
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /Claude-Web User-Agent String
The user-agent token for Claude-Web is:
Claude-Web🌐Who Blocks Claude-Web?
Blocking (49 sites)
05bet.me0kkbd.com100layercake.com104nk88.com10best.com112bdt222.com112phsky.vip113bd333.vip116bdt222.com116phsky.vip11freunde.de121bet15.com121bet.cc123rf.com12bajeevip.com12g555.com12no777.com13abc.com13g555.com13wham.com13x555.com157jilipark.com166bet.com1777cb.com1777sx.com1777sz.com17bdbajee.com1861.ca1881.no1xjl.cfd20minutes.fr21viptaka.com222bd2.com228aceph.cc22hero789.com24bdbajee.com24heures.ch24phcash.co24phdream.com24phpark.com2521673.com25newsnow.com26superph.com2777cb.com2777cx.com2822news.com28phjoy.com299bet.com29news.com
Explicitly Allowing (1 sites)
🔗Other Anthropic Bots
Check Your Site's AI Policy
See if you're blocking or allowing Claude-Web and other AI crawlers.