Velen Public Web Crawler
Tier 4Analyzes pages for business intelligence and AI datasets.
VelenPublicWebCrawler🎯What is Velen Public Web Crawler?
Velen Public Web Crawler is an AI training crawler operated by Velen/Hunter. Collects data to train AI models.
Builds B2B intelligence datasets.
Content excluded from Velen's business datasets.
🏢About Velen/Hunter
Velen/Hunter operates 1 known bot for AI model training.
🛡️VelenPublicWebCrawler robots.txt Configuration
Control VelenPublicWebCrawler access to your website using robots.txt directives.
Block VelenPublicWebCrawler
To completely block Velen Public Web Crawler from crawling your site:
User-agent: VelenPublicWebCrawler
Disallow: /Allow VelenPublicWebCrawler Full Access
To explicitly allow Velen Public Web Crawler to crawl your entire site:
User-agent: VelenPublicWebCrawler
Allow: /Selective Access for VelenPublicWebCrawler
To allow Velen Public Web Crawler but restrict certain directories:
User-agent: VelenPublicWebCrawler
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /✓ Velen Public Web Crawler respects robots.txt directives.
VelenPublicWebCrawler User-Agent String
The user-agent token for Velen Public Web Crawler is:
VelenPublicWebCrawler🌐Who Blocks Velen Public Web Crawler?
Check Your Site's AI Policy
See if you're blocking or allowing Velen Public Web Crawler and other AI crawlers.