User-Agent Token
GrokBotRespects robots.txt
UnknownImpact Level
Major
100M+ users - Meta, Apple, Microsoft, Perplexity, xAI
Estimated Reach
Available to X Premium subscribers
🎯What is GrokBot?
GrokBot is an AI training crawler operated by xAI. Collects data to train AI models.
📊 How Your Data is Used
Pre-training for Grok AI models.
🚫 What Happens If You Block
Content won't train Grok models.
💡 Good to Know
xAI founded by Elon Musk. Grok integrated into X (Twitter). Has real-time access to X posts.
🏢About xAI
xAI
xAI operates 3 known bots for AI model training. Their service reaches Available to X Premium subscribers.
🛡️GrokBot robots.txt Configuration
Control GrokBot access to your website using robots.txt directives.
Block GrokBot
To completely block GrokBot from crawling your site:
User-agent: GrokBot
Disallow: /Allow GrokBot Full Access
To explicitly allow GrokBot to crawl your entire site:
User-agent: GrokBot
Allow: /Selective Access for GrokBot
To allow GrokBot but restrict certain directories:
User-agent: GrokBot
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /GrokBot User-Agent String
The user-agent token for GrokBot is:
GrokBot🌐Who Blocks GrokBot?
Blocking (42 sites)
abendblatt.deaid.noautomap.itautotrader.combbv-net.debgland24.deblog.euroka.moeblv.nobonus.cabraunschweiger-zeitung.dechiemgau24.declinisciences.comcome-on.decostanachrichten.comcyclingtips.comdenniksport.skdorstenerzeitung.deecho24.deeverybodywiki.comfederalnewsnetwork.comfnp.defranceculture.frfranceinter.frfrancemusique.frfr.defritz.defuldaerzeitung.degiessener-allgemeine.degiessener-anzeiger.deha-halden.nohellwegeranzeiger.dehendrickcars.comhertzcarsales.comhna.dehr3.deikz-online.deingame.dekbb.comkreiszeitung.delaagendalsposten.nol-camera-forum.comliftedtrucks.com
Explicitly Allowing (8 sites)
🔗Other xAI Bots
Check Your Site's AI Policy
See if you're blocking or allowing GrokBot and other AI crawlers.