ClaudeBot🎯What is ClaudeBot?
ClaudeBot is an AI training crawler operated by Anthropic. Collects data to train AI models.
Pre-training for Claude models. Anthropic emphasizes ethical data sourcing and AI safety.
Your content won't train future Claude models. Current Claude knowledge unaffected.
Anthropic is known for AI safety research. Founded by ex-OpenAI researchers. Major enterprise adoption.
🏢About Anthropic
Anthropic operates 5 known bots for AI model training. Their service reaches 100M+ users.
🛡️ClaudeBot robots.txt Configuration
Control ClaudeBot access to your website using robots.txt directives.
Block ClaudeBot
To completely block ClaudeBot from crawling your site:
User-agent: ClaudeBot
Disallow: /Allow ClaudeBot Full Access
To explicitly allow ClaudeBot to crawl your entire site:
User-agent: ClaudeBot
Allow: /Selective Access for ClaudeBot
To allow ClaudeBot but restrict certain directories:
User-agent: ClaudeBot
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /✓ ClaudeBot respects robots.txt directives.
ClaudeBot User-Agent String
The user-agent token for ClaudeBot is:
ClaudeBot🌐Who Blocks ClaudeBot?
🔗Other Anthropic Bots
Check Your Site's AI Policy
See if you're blocking or allowing ClaudeBot and other AI crawlers.