We made Claude Code and Codex surf docs and counted every 404 they hit before finding the answer.
We served the same content 4 ways: raw HTML, plain markdown, markdown + an llms.txt index, and markdown with the llms.txt inlined on every page.
We found that agents reach for markdown and llms.txt on their own, and HTML-only finished last by a wide margin (15-30x more 404s than markdown + llms.txt).