Cloudflare AutoRAG
Tier 3Indexes sites for Cloudflare's Vertex AI Search and RAG services.
Cloudflare-AutoRAG🎯What is Cloudflare AutoRAG?
Cloudflare AutoRAG is an AI search crawler operated by Cloudflare. Powers AI-enhanced search features.
Powers Cloudflare's AI Gateway and RAG features for enterprises.
Content won't be searchable via Cloudflare's AI search services.
Used by businesses to build AI-powered search over their content.
🏢About Cloudflare
🛡️Cloudflare-AutoRAG robots.txt Configuration
Control Cloudflare-AutoRAG access to your website using robots.txt directives.
Block Cloudflare-AutoRAG
To completely block Cloudflare AutoRAG from crawling your site:
User-agent: Cloudflare-AutoRAG
Disallow: /Allow Cloudflare-AutoRAG Full Access
To explicitly allow Cloudflare AutoRAG to crawl your entire site:
User-agent: Cloudflare-AutoRAG
Allow: /Selective Access for Cloudflare-AutoRAG
To allow Cloudflare AutoRAG but restrict certain directories:
User-agent: Cloudflare-AutoRAG
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /✓ Cloudflare AutoRAG respects robots.txt directives.
Cloudflare-AutoRAG User-Agent String
The user-agent token for Cloudflare AutoRAG is:
Cloudflare-AutoRAG🌐Who Blocks Cloudflare AutoRAG?
Check Your Site's AI Policy
See if you're blocking or allowing Cloudflare AutoRAG and other AI crawlers.