llms.txt is a plain-text file placed at the root of a website (e.g.,Documentation Index
Fetch the complete documentation index at: https://docs.getcited.in/llms.txt
Use this file to discover all available pages before exploring further.
example.com/llms.txt) that provides a structured summary of the site’s most important content for AI crawlers and LLMs. It follows the llms.txt specification (llmstxt.org), serving a similar purpose to robots.txt but focused on content discovery rather than access control.
Why it matters
AI crawlers and LLMs use llms.txt to understand which pages on a site are most relevant for answering user queries. Without it, an LLM must discover content through its own crawling and indexing — which may miss important pages or over-index low-value ones. A well-structured llms.txt file explicitly signals “these are the pages worth reading” to every AI crawler that supports the standard.How Cited uses it
Cited’s docs site uses Mintlify, which auto-generates both/llms.txt (page listings with descriptions) and /llms-full.txt (full markdown content of every page). Both files are referenced via HTTP link headers on every page response. This auto-generation is one of the mechanisms that compensates for the custom schema limitation on the Mintlify platform.
Related concepts
- Fix your llms.txt and robots.txt — the practical optimization playbook
- Crawl budget — llms.txt helps crawlers allocate budget efficiently
- What sources LLMs cite — llms.txt improves discoverability