🤖
Datenbank Crawler
Tier 4📚 AI Trainingby NetEstate ↗
German company's international data gathering bot.
User-Agent Token
datenbank-crawlerRespects robots.txt
UnknownImpact Level
Niche
Smaller players and developer tools
Estimated Reach
Smaller players and developer tools
🎯What is Datenbank Crawler?
Datenbank Crawler is an AI training crawler operated by NetEstate. Collects data to train AI models.
📊 How Your Data is Used
Commercial data aggregation and analysis.
🚫 What Happens If You Block
NetEstate can't index your content.
🏢About NetEstate
🛡️datenbank-crawler robots.txt Configuration
Control datenbank-crawler access to your website using robots.txt directives.
Block datenbank-crawler
To completely block Datenbank Crawler from crawling your site:
User-agent: datenbank-crawler
Disallow: /Allow datenbank-crawler Full Access
To explicitly allow Datenbank Crawler to crawl your entire site:
User-agent: datenbank-crawler
Allow: /Selective Access for datenbank-crawler
To allow Datenbank Crawler but restrict certain directories:
User-agent: datenbank-crawler
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /datenbank-crawler User-Agent String
The user-agent token for Datenbank Crawler is:
datenbank-crawler🔗Other NetEstate Bots
Check Your Site's AI Policy
See if you're blocking or allowing Datenbank Crawler and other AI crawlers.