Site History
Track how welt.de's technology stack and AI policy have changed over time
📍
Feb 9, 2026, 11:43 PMTech ScanLatest
View Details6 technologies detected
📍
Feb 9, 2026, 05:10 AMPolicy SnapshotLatest
CLOSED52 AI bots blocked (32 training)
robots.txt
Policy changes:
Started blocking:
Bedrock BotCohere AICohere Training CrawlerFirecrawlGemini Deep ResearchProject Marinerimg2dataset (LAION)Meta External AgentMeta External FetcherMistral AI UserOmgiliOmgilibotQuillBot CrawlerVelen Public Web CrawlerWebz.io Extended
Stopped blocking:
bedrockbotcohere-aicohere-training-data-crawlerFirecrawlAgentGemini-Deep-ResearchGoogleAgent-Marinerimg2datasetmeta-externalagentMeta-ExternalFetcherMistralAI-UseromgiliomgilibotQuillBotVelenPublicWebCrawlerwebzio-extended
Removed llms.txt
📦
Feb 9, 2026, 05:10 AMTech Scan
View Details6 technologies detected
📦
🛡️
Jan 29, 2026, 07:06 AMPolicy Snapshot
CLOSED52 AI bots blocked (32 training)
robots.txtllms.txt
Policy changes:
UNKNOWN→CLOSED
Started blocking:
anthropic-aiAI2BotAi2Bot-DolmaAmazonbotApplebot-ExtendedbedrockbotBytespiderCCBotClaude-SearchBotClaude-UserClaude-WebClaudeBotcohere-aicohere-training-data-crawlerCotoyogiDevinDiffbotDuckAssistBotFacebookBotFirecrawlAgentGemini-Deep-ResearchGoogle-CloudVertexBotGoogle-ExtendedGoogleAgent-MarinerGoogleOtherGoogleOther-ImageGoogleOther-Videoimg2datasetKangaroo Botmeta-externalagentMeta-ExternalFetcherMistralAI-UserNovaActomgiliomgilibotOperatorPanguBotPerplexity-UserPerplexityBotPetalBotPhindBotQualifiedBotQuillBotSBIntuitionsBotScrapyTikTokSpiderTimpibotVelenPublicWebCrawlerwebzio-extendedYandexAdditionalYandexAdditionalBotYouBot
🛡️
Jan 29, 2026, 06:35 AMPolicy Snapshot
UNKNOWN0 AI bots blocked
llms.txt
Policy changes:
CLOSED→UNKNOWN
Stopped blocking:
anthropic-aiAI2BotAi2Bot-DolmaAmazonbotApplebot-ExtendedbedrockbotBytespiderCCBotClaude-SearchBotClaude-UserClaude-WebClaudeBotcohere-aicohere-training-data-crawlerCotoyogiDevinDiffbotDuckAssistBotFacebookBotFirecrawlAgentGemini-Deep-ResearchGoogle-CloudVertexBotGoogle-ExtendedGoogleAgent-MarinerGoogleOtherGoogleOther-ImageGoogleOther-Videoimg2datasetKangaroo Botmeta-externalagentMeta-ExternalFetcherMistralAI-UserNovaActomgiliomgilibotOperatorPanguBotPerplexity-UserPerplexityBotPetalBotPhindBotQualifiedBotQuillBotSBIntuitionsBotScrapyTikTokSpiderTimpibotVelenPublicWebCrawlerwebzio-extendedYandexAdditionalYandexAdditionalBotYouBot
🛡️
Jan 29, 2026, 06:20 AMPolicy Snapshot
CLOSED52 AI bots blocked (32 training)
robots.txtllms.txt
First recorded policy snapshot
Summary
3
Tech Scans
4
Policy Snapshots
6
Current Techs
0
Techs Added (All Time)