Lars Jansen(@oldstackjournal):robots.txt has been around since the 90's, but it took until 2022 for it to become an official internet standard. Nearly 30 years of websites politely asking robots not to wander into certain places. And now some of us are using the same system to try to keep AI crawlers out. 😂

Lars Jansen

@oldstackjournal

🛠️ AI tools & simple web workflows for builders! 🔥 🔥 🔥 🔥

参加 May 2026

2.2K フォロー中 4K ファン

Lars Jansen@oldstackjournal

2026.10.29 18:31

robots.txt has been around since the 90's, but it took until 2022 for it to become an official internet standard. Nearly 30 years of websites politely asking robots not to wander into certain places. And now some of us are using the same system to try to keep AI crawlers out. 😂