🤖
User-Agent Token
TavilyBotRespects robots.txt
UnknownImpact Level
Niche
Smaller players and developer tools
Estimated Reach
Smaller players and developer tools
🎯What is TavilyBot?
TavilyBot is an AI search crawler operated by Tavily. Powers AI-enhanced search features.
📊 How Your Data is Used
Search API used by developers building AI apps.
🚫 What Happens If You Block
Content won't appear in Tavily-powered AI applications.
💡 Good to Know
Developer tool. Powers search in many AI applications and agents.
🏢About Tavily
🛡️TavilyBot robots.txt Configuration
Control TavilyBot access to your website using robots.txt directives.
Block TavilyBot
To completely block TavilyBot from crawling your site:
User-agent: TavilyBot
Disallow: /Allow TavilyBot Full Access
To explicitly allow TavilyBot to crawl your entire site:
User-agent: TavilyBot
Allow: /Selective Access for TavilyBot
To allow TavilyBot but restrict certain directories:
User-agent: TavilyBot
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /TavilyBot User-Agent String
The user-agent token for TavilyBot is:
TavilyBot🌐Who Blocks TavilyBot?
Blocking (50 sites)
arkansasonline.comaskmen.combabycenter.combabycentre.co.ukbarcelona.catbrainerddispatch.comchrispederick.comdiariodejerez.esdiariodepontevedra.esemmasdiary.co.ukeurogamer.neteurogamer.pteuropasur.esextremetech.comgamesindustry.bizgeek.comgld.nlgranadahoy.comhaiku-os.orghealthecareers.comheavengames.comhoustonpublicmedia.orghowlongtobeat.comhuelvainformacion.esign.cominsidehalton.coml1nieuws.nllifehacker.commapgenie.iomashable.commaxroll.ggmother.lynationen.noomim.orgomroepwest.nlomroepzeeland.nlpathfinderwiki.compcmag.compolarsteps.compostcrossing.compzwiki.netrecyclingtoday.comrijnmond.nlrockpapershotgun.comrtvdrenthe.nlrtvnoord.nlrtvoost.nlrtvutrecht.nlsemissourian.comtf1.fr
Check Your Site's AI Policy
See if you're blocking or allowing TavilyBot and other AI crawlers.