How to Generate llms.txt and llms-full.txt in Next.js (App Router Guide)

Serving statically hardcoded metadata is no longer sufficient. Modern dynamic web frameworks need to render LLM-ready directories on the fly. Let's look at how to build and cache dynamic llms.txt files in Next.js.

1. The Architecture of Next.js dynamic routing

Next.js App Router relies on Route Handlers to serve custom formats like plain text, XML, or JSON. Placing an llms.txt file in your public directory works only for static content. When content updates, a static asset requires a full redeploy of the application.

Using a dynamic Route Handler lets you query CMS platforms or local files on the fly. This ensures crawling tools immediately locate newly added reference material. For high performance, we recommend deploying on robust instances like DigitalOcean, which scales easily under crawler spikes.

2. Implementation of app/llms.txt/route.ts

Create a directory path at app/llms.txt and add a route.ts file. This script queries your documentation pages and compiles the main directory index.

import { NextResponse } from 'next/server';

export const revalidate = 86400; // Cache for 24 hours

export async function GET() {
  const siteUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://llms-txt.xyz';
  
  // Example fetching dynamic resources
  const docs = [
    { title: "Validator Guide", url: `${siteUrl}/how-to-validate-llms-txt` },
    { title: "SEO Benefits", url: `${siteUrl}/llms-txt-seo-benefit` },
    { title: "Shopify Integration", url: `${siteUrl}/llms-txt-shopify` }
  ];

  let output = `# llms-txt.xyz Docs\n`;
  output += `> Dynamic documentation index generated for AI agents.\n\n`;
  output += `## Core Guides\n`;
  
  docs.forEach(doc => {
    output += `- [${doc.title}](${doc.url})\n`;
  });
  
  output += `\n## Full Indexes\n`;
  output += `- [llms-full.txt](${siteUrl}/llms-full.txt): Complete full-text documentation database.\n`;

  return new NextResponse(output, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, s-maxage=86400, stale-while-revalidate=3600'
    }
  });
}

3. Dynamic llms-full.txt Compilation

The standard recommends linking to a full-text representation at llms-full.txt. This contains inlined markdown descriptions of every linked page. To learn more about this format, check out our guide on llms-full.txt explained.

Let's look at the differences between the two endpoints:

To automate compilation, you can write script controllers that read local MDX assets, strip YAML frontmatter, and output clean content buffers. You can explore how to set this up in our guide on generating llms-full.txt programmatically.

4. Testing Your Implementation

After deploying your Next.js route, verify that the served format is correct. Relative URL formats or lack of headers can cause crawler parsers to fail. You can debug your routes using our free llms.txt validator tool to check for formatting compliant with standard parameters.