llms.txt is served as text/plain
Agents identify llms.txt by both location AND content type. Serving it as text/html or application/octet-stream causes some ingesters to skip the file or treat it as a binary, which defeats the point of having it.
How the check decides
After locating the llms.txt file, the check reads its Content-Type response header. It passes if the value contains text/plain (case-insensitive). It returns a warning otherwise. If no llms.txt is published it returns N/A so it doesn’t double-count against the score.
How to implement it
Configure your web server or static host to serve .txt files with Content-Type: text/plain; charset=utf-8. Most static hosts (Vercel, Netlify, GitHub Pages, S3) do this by default for .txt files; check your custom redirects or rewrites if you’ve put llms.txt behind one.
Pass
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
# Example Docs
- [Install](https://example.com/docs/install.md)
Fail
HTTP/1.1 200 OK
Content-Type: application/octet-stream
# Example Docs
- [Install](https://example.com/docs/install.md)
What we measured
Our benchmark served llms.txt as text/plain throughout and never varied it, so the study below does not measure what a wrong content type costs. Treat the guidance above as convention, not as something we have put a number on.
What it does measure is worth knowing anyway: when the agent read the file it used 33% fewer tokens on the same page (177,735 against 266,591), but across four arms, a <link rel="llms-txt"> tag, a <link rel="alternate" type="text/markdown"> tag, a visible “For agents” footer link, and no signal at all, it fetched llms.txt 0 times out of 5. Only a line in the prompt got it to look.