User-Agent Token
Grok-DeepSearchRespects robots.txt
UnknownImpact Level
Major
100M+ users - Meta, Apple, Microsoft, Perplexity, xAI
Estimated Reach
X Premium subscribers
🎯What is Grok DeepSearch?
Grok DeepSearch is an AI search crawler operated by xAI. Powers AI-enhanced search features.
📊 How Your Data is Used
Real-time web search for Grok responses.
🚫 What Happens If You Block
Grok's search won't include your content in research answers.
💡 Good to Know
Competes with Perplexity and ChatGPT search. Integrated into X.
🏢About xAI
xAI
xAI operates 3 known bots for AI-powered search. Their service reaches X Premium subscribers.
🛡️Grok-DeepSearch robots.txt Configuration
Control Grok-DeepSearch access to your website using robots.txt directives.
Block Grok-DeepSearch
To completely block Grok DeepSearch from crawling your site:
User-agent: Grok-DeepSearch
Disallow: /Allow Grok-DeepSearch Full Access
To explicitly allow Grok DeepSearch to crawl your entire site:
User-agent: Grok-DeepSearch
Allow: /Selective Access for Grok-DeepSearch
To allow Grok DeepSearch but restrict certain directories:
User-agent: Grok-DeepSearch
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /Grok-DeepSearch User-Agent String
The user-agent token for Grok DeepSearch is:
Grok-DeepSearch🌐Who Blocks Grok DeepSearch?
Blocking (22 sites)
Explicitly Allowing (3 sites)
🔗Other xAI Bots
Check Your Site's AI Policy
See if you're blocking or allowing Grok DeepSearch and other AI crawlers.