Claude-SearchBot
Tier 1Indexes content for Claude's web search capabilities.
Claude-SearchBot🎯What is Claude-SearchBot?
Claude-SearchBot is an AI search crawler operated by Anthropic. Powers AI-enhanced search features.
Real-time retrieval for search. Content attributed, not used for training.
Your site won't appear in Claude's search results or web citations.
Claude search launched 2024. Blocking reduces visibility to Claude users researching topics.
🏢About Anthropic
Anthropic operates 5 known bots for AI-powered search. Their service reaches 100M+ users.
🛡️Claude-SearchBot robots.txt Configuration
Control Claude-SearchBot access to your website using robots.txt directives.
Block Claude-SearchBot
To completely block Claude-SearchBot from crawling your site:
User-agent: Claude-SearchBot
Disallow: /Allow Claude-SearchBot Full Access
To explicitly allow Claude-SearchBot to crawl your entire site:
User-agent: Claude-SearchBot
Allow: /Selective Access for Claude-SearchBot
To allow Claude-SearchBot but restrict certain directories:
User-agent: Claude-SearchBot
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /✓ Claude-SearchBot respects robots.txt directives.
Claude-SearchBot User-Agent String
The user-agent token for Claude-SearchBot is:
Claude-SearchBot🌐Who Blocks Claude-SearchBot?
🔗Other Anthropic Bots
Check Your Site's AI Policy
See if you're blocking or allowing Claude-SearchBot and other AI crawlers.