example.com/llms.txt) that provides a structured summary of the site’s most important content for AI crawlers and LLMs. It follows the llms.txt specification (llmstxt.org), serving a similar purpose to robots.txt but focused on content discovery rather than access control.
Why it matters
AI crawlers and LLMs use llms.txt to understand which pages on a site are most relevant for answering user queries. Without it, an LLM must discover content through its own crawling and indexing — which may miss important pages or over-index low-value ones. A well-structured llms.txt file explicitly signals “these are the pages worth reading” to every AI crawler that supports the standard.How Cited uses it
Cited’s docs site uses Mintlify, which auto-generates both/llms.txt (page listings with descriptions) and /llms-full.txt (full markdown content of every page). Both files are referenced via HTTP link headers on every page response.
Related concepts
- Fix your llms.txt and robots.txt — the practical optimization playbook
- Crawl budget — llms.txt helps crawlers allocate budget efficiently
- What sources LLMs cite — llms.txt improves discoverability