User-Agent Token
Claude-WebRespects robots.txt
UnknownImpact Level
Notable
10M+ users - Regional/specialized AI companies
Estimated Reach
10M+ users - Regional/specialized AI companies
🎯What is Claude-Web?
Claude-Web is a web crawler operated by Anthropic. Purpose not documented.
🚫 What Happens If You Block
Unknown - purpose not documented.
💡 Good to Know
May be deprecated or internal. Block if concerned about unknown crawling.
🏢About Anthropic
🛡️Claude-Web robots.txt Configuration
Control Claude-Web access to your website using robots.txt directives.
Block Claude-Web
To completely block Claude-Web from crawling your site:
User-agent: Claude-Web
Disallow: /Allow Claude-Web Full Access
To explicitly allow Claude-Web to crawl your entire site:
User-agent: Claude-Web
Allow: /Selective Access for Claude-Web
To allow Claude-Web but restrict certain directories:
User-agent: Claude-Web
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /Claude-Web User-Agent String
The user-agent token for Claude-Web is:
Claude-Web🌐Who Blocks Claude-Web?
Blocking (49 sites)
0kkbd.com10best.com11freunde.de123rf.com12bajeevip.com12g555.com13abc.com13g555.com13wham.com13x555.com166bet.com1777sx.com17bdbajee.com1881.no20minutes.fr21viptaka.com24bdbajee.com24heures.ch24phcash.co2521673.com26superph.com299bet.com29news.com30seconds.com33bd.org355jilipark.vip3dvf.com3rr.com4029tv.com47news.jp499bet.bingo4p.de4players.de4plebs.org4vid.top50states.com52216452.com55brlbet.com57bet777.win588ku.com5gbet51.com5gbet.com6523145.com666p999.com678bd44.com72bet.me777jaya.net777tiger888.com7788bet.com
Explicitly Allowing (1 sites)
🔗Other Anthropic Bots
Check Your Site's AI Policy
See if you're blocking or allowing Claude-Web and other AI crawlers.