llms.txt explained: what it is, and does it actually work?
A plain-English breakdown of llms.txt and llms-full.txt, who reads them today, and how to write one that is genuinely useful to AI crawlers.
llms.txt explained: what it is, and does it actually work?
The proposal is simple: a file at /llms.txt, written in markdown, listing your key URLs with a one-line description each. A companion /llms-full.txt inlines the actual content so an agent can read your docs in a single fetch instead of crawling fifty pages.
Who actually reads it
Agentic fetchers do — coding assistants, research agents and anything that resolves a URL on demand. Large-scale training crawlers mostly ignore it. So the honest framing is: llms.txt improves the experience of an agent that already decided to visit you. It does not make one decide to visit.
What a good llms.txt looks like
- An H1 with your brand name and a one-sentence description of what you do.
- A blockquote summary an agent can quote verbatim.
- Sections grouped by intent: docs, products, guides, pricing, contact.
- Absolute HTTPS URLs with a short description after each link.
- No marketing adjectives — agents discard them and they dilute the signal.
Common mistakes
- Dumping every URL in the sitemap into it — the whole point is curation.
- Letting it drift out of date, so agents quote pages you deleted.
- Treating it as a replacement for schema markup or server-rendered HTML.
- Serving it as text/html instead of text/plain or text/markdown.
Our verdict
Add it. It takes twenty minutes and it makes your site legible to agents. Then spend the rest of your effort on the things that actually decide citation: crawlable HTML, direct answers, and structured data.
Frequently asked questions
What is llms.txt?
llms.txt is a markdown file at the root of a site that gives large language models a curated, plain-text map of your most important pages and what each one covers.
Is llms.txt required for AI search?
No. It is optional and not an official standard. It helps AI agents that fetch it, but robots.txt permissions, server-rendered HTML and schema markup matter far more today.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a short index of links with one-line descriptions. llms-full.txt inlines the actual content so an agent can read your documentation without crawling every page.
Cite this page
Referencing this in a piece of writing? Copy a formatted citation attribution helps other builders find the source.
Usman Jatoi (2026). llms.txt explained: what it is, and does it actually work?. Build on Vibe. Retrieved from https://buildonvibe.site/blog/llms-txt-explained
@misc{bov-blog-llms-txt-explained,
title = {llms.txt explained: what it is, and does it actually work?},
author = {Usman Jatoi},
year = {2026},
url = {https://buildonvibe.site/blog/llms-txt-explained},
note = {Build on Vibe}
}