llms-full.txt Explained: Deep Ingestion for AI Agents
While llms.txt acts as a concise directory index, llms-full.txt is a companion file designed to serve complete page contents for deep crawling by AI models. To learn the basic standard first, read our guide on what is the llms.txt standard.
Key Takeaways
llms-full.txtacts as a full textual database rather than a simple link index.- Inlining complete markdown pages reduces round-trip times for AI crawlers.
- Demarcate clean page content using H2 headers (
## Content: Page Title). - Providing both index and full-text files optimizes ingestion for coding assistants.
1. The Index vs the Database
Think of llms.txt as the table of contents. It lists what is available on your site. llms-full.txt acts as the actual content library, inlining the full text of your resources to allow AI agents to digest the material without making multiple HTTP requests. Read more about general validation rules in how to validate compliance guide.
2. Inlining Strategies
To compile llms-full.txt, you inline the cleaned text of your pages:
- Separate with Headings: Use H2 headers (
## Content: Page Title) to demarcate pages. - List Source Paths: Include the origin URL immediately beneath each header.
- Format Cleanly: Strip navigation menus, footers, and tracking scripts, leaving only raw Markdown content.
LLMs.txt vs. LLMs-full.txt Comparison
| Specification Aspect | llms.txt (Index File) | llms-full.txt (Full Database) |
|---|---|---|
| Primary Purpose | High-level directory index | Complete text corpus ingestion |
| Structure Format | Markdown lists and links | Inlined page Markdown separated by H2 headings |
| Typical Size | Small (usually < 10KB) | Medium to Large (100KB to 2MB) |
| Recommended Links | Links to pages and sub-sections | Origin URLs mapped to header content |
3. Improving Developer UX
Coding assistants and search bots use this file to index developer documentations quickly. Instead of crawling 50 different URLs, the bot downloads the single llms-full.txt file to query the full documentation, reducing server load and API latency. See our directory of the best llms.txt generator tools.
Frequently Asked Questions
It serves as a comprehensive companion file to llms.txt, containing the complete, inlined textual content of your pages for deep indexing.
llms.txt lists links and resource summaries (an index), while llms-full.txt contains the actual full-text content of those resources.
Yes. Coding assistants (like Cursor and Windsurf) use it to ingest library documentations quickly in a single crawl pass.
Yes. It should be hosted at the root directory level (e.g. domain.com/llms-full.txt) alongside the main index file.
Use level 2 Markdown headings (## Content: Title) to separate the inlined content of different pages.
It can reduce overall server load by allowing crawlers to fetch all site content in a single request, rather than hitting multiple pages.
While there is no strict limit, keep the file concise and focused on high-priority resources to avoid exceeding LLM context windows.
Yes. Many static site generators and documentation frameworks compile it automatically during the build phase.
Yes. The standard practice is to link to the full version at the bottom of your primary llms.txt file.
Yes, AI agents crawl llms-full.txt to extract detailed contextual data for answers.