User-Agent Token
Claude-WebRespects robots.txt
UnknownImpact Level
Notable
10M+ users - Regional/specialized AI companies
Estimated Reach
10M+ users - Regional/specialized AI companies
🎯What is Claude-Web?
Claude-Web is a web crawler operated by Anthropic. Purpose not documented.
🚫 What Happens If You Block
Unknown - purpose not documented.
💡 Good to Know
May be deprecated or internal. Block if concerned about unknown crawling.
🏢About Anthropic
🛡️Claude-Web robots.txt Configuration
Control Claude-Web access to your website using robots.txt directives.
Block Claude-Web
To completely block Claude-Web from crawling your site:
User-agent: Claude-Web
Disallow: /Allow Claude-Web Full Access
To explicitly allow Claude-Web to crawl your entire site:
User-agent: Claude-Web
Allow: /Selective Access for Claude-Web
To allow Claude-Web but restrict certain directories:
User-agent: Claude-Web
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /Claude-Web User-Agent String
The user-agent token for Claude-Web is:
Claude-Web🌐Who Blocks Claude-Web?
Blocking (46 sites)
123rf.com20minutes.fr21viptaka.com24bdbajee.com24heures.ch299bet.com47news.jp499bet.bingo678bd44.com777tiger888.com7sur7.be888pg-ox.vip88betpk.com8newsnow.com9news.com.auaachener-zeitung.deabc27.comabc4.comabc6onyourside.comabc.net.auabendblatt.deabout.comactu.frad.nladressa.noadvrider.comadweek.comaerzteblatt.deafr.comaftenposten.noaftonbladet.seagrarheute.comagriculture.comahrq.goval.comaljazeera.comaljazeera.netallmusic.comallrecipes.comalltheweb.comaltavista.comamarujala.comandroidpolice.comanime-planet.comansa.itaol.com
Explicitly Allowing (4 sites)
🔗Other Anthropic Bots
Check Your Site's AI Policy
See if you're blocking or allowing Claude-Web and other AI crawlers.