Site History
Track how businessinsider.de's technology stack and AI policy have changed over time
📍
Feb 9, 2026, 03:10 PMPolicy SnapshotLatest
CLOSED52 AI bots blocked (32 training)
robots.txt
Policy changes:
OPEN→CLOSED
Started blocking:
anthropic-aiAI2BotAi2Bot-DolmaAmazonbotApplebot-ExtendedBedrock BotBytespiderCCBotClaude-SearchBotClaude-UserClaude-WebClaudeBotCohere AICohere Training CrawlerCotoyogiDevinDiffbotDuckAssistBotFacebookBotFirecrawlGemini Deep ResearchGoogle-CloudVertexBotGoogle-ExtendedProject MarinerGoogleOtherGoogleOther-ImageGoogleOther-Videoimg2dataset (LAION)Kangaroo BotMeta External AgentMeta External FetcherMistral AI UserNovaActOmgiliOmgilibotOperatorPanguBotPerplexity-UserPerplexityBotPetalBotPhindBotQualifiedBotQuillBot CrawlerSBIntuitionsBotScrapyTikTokSpiderTimpibotVelen Public Web CrawlerWebz.io ExtendedYandexAdditionalYandexAdditionalBotYouBot
📍
Feb 9, 2026, 03:10 PMTech ScanLatest
View Details18 technologies detected
Changes from previous scan:
+ Added:
core-jsEcovium
📦
🛡️
Jan 29, 2026, 11:02 AMPolicy Snapshot
OPEN0 AI bots blocked
robots.txt
First recorded policy snapshot
Summary
2
Tech Scans
2
Policy Snapshots
18
Current Techs
2
Techs Added (All Time)