Skip to content

mkdocs-llmstxt¤

ci documentation pypi version gitter

MkDocs plugin to generate an /llms.txt file.

/llms.txt - A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.

See our own dynamically generated /llms.txt as a demonstration.

Installation¤

pip install mkdocs-llmstxt

Usage¤

Enable the plugin in mkdocs.yml:

mkdocs.yml
site_name: My project
site_description: Description of my project.
site_url: https://myproject.com/  # Required for the llmstxt plugin to work.

plugins:
- llmstxt:
    markdown_description: Long description of my project.
    sections:
      Usage documentation:
      - file1.md
      - file2.md

The resulting /llms.txt file will be available at the root of your documentation. With the previous example, it will be accessible at https://myproject.com/llms.txt and will contain the following:

# My project

> Description of my project.

Long description of my project.

## Usage documentation

- [File1 title](https://myproject.com/file1.md)
- [File2 title](https://myproject.com/file2.md)

Each source file included in sections will have its own Markdown file available at the specified URL in the /llms.txt. See Markdown generation for more details.

File globbing is supported:

mkdocs.yml
plugins:
- llmstxt:
    sections:
      Usage documentation:
      - index.md
      - usage/*.md

Full output¤

Although not explicitly written out in the https://llmstxt.org/ guidelines, it is common to output a llms-full.txt file with every page content expanded. This file can be generated by setting the full_output configuration value:

plugins:
- llmstxt:
    full_output: llms-full.txt
    sections:
      Usage documentation:
      - index.md
      - usage/*.md

Markdown generation¤

To generate a Markdown page from a source file, the plugin will:

Doing so is necessary to ensure that dynamically generated contents (API documentation, executed code blocks, snippets from other files, Jinja macros, etc.) are part of the generated text files.

Credits to Petyo Ivanov for the original idea ✨.

You can disable auto-cleaning of the HTML:

mkdocs.yml
plugins:
- llmstxt:
    autoclean: false

You can also pre-process the HTML before it is converted back to Markdown:

mkdocs.yml
plugins:
- llmstxt:
    preprocess: path/to/script.py

The specified script.py must expose a preprocess function that accepts the soup and output arguments:

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from bs4 import BeautifulSoup

def preprocess(soup: BeautifulSoup, output: str) -> None:
    ...  # modify the soup

The output argument lets you modify the soup depending on which file is being generated.

Have a look at our own cleaning function to get inspiration.