Site History
Track how htxt.co.za's technology stack and AI policy have changed over time
📍
May 14, 2026, 04:31 PMPolicy SnapshotLatest
CLOSED33 AI bots blocked (22 training)
robots.txt
Policy changes:
Started blocking:
Meta External AgentMeta External FetcherVelen Public Web CrawlerWebz.io ExtendedCohere AIimg2dataset (LAION)OmgiliOmgilibot
Stopped blocking:
meta-externalagentMeta-ExternalFetcherVelenPublicWebCrawlerwebzio-extendedcohere-aiimg2datasetomgiliomgilibot
📍
🛡️
Feb 2, 2026, 10:23 AMPolicy Snapshot
CLOSED33 AI bots blocked (22 training)
robots.txt
First recorded policy snapshot
Summary
1
Tech Scans
2
Policy Snapshots
19
Current Techs
0
Techs Added (All Time)