Site History
Track how fox21online.com's technology stack and AI policy have changed over time
📍
May 24, 2026, 10:08 AMPolicy SnapshotLatest
CLOSED26 AI bots blocked (18 training)
robots.txt
Policy changes:
SELECTIVE→CLOSED
Started blocking:
AI2BotAi2Bot-DolmaApplebot-ExtendedBytespiderClaude-WebClaudeBotDiffbotFacebookBotMeta External AgentOAI-SearchBotPerplexityBotPetalBotScrapyTimpibotVelen Public Web CrawlerWebz.io ExtendedYouBotanthropic-aiCohere AIimg2dataset (LAION)OmgiliOmgilibot
📍
🛡️
Feb 2, 2026, 01:50 PMPolicy Snapshot
SELECTIVE4 AI bots blocked (3 training)
robots.txt
First recorded policy snapshot
Summary
1
Tech Scans
2
Policy Snapshots
18
Current Techs
0
Techs Added (All Time)