Pages for agents

A Markdown copy of each page, and an index of them, for language models and coding agents.

A language model reads Markdown more easily than a web page. The navbar, the sidebar and the scripts of a page are noise to it. So smalldocs can write a Markdown copy of each page, and an index of the copies, the file llms.txt. A coding agent, such as Claude Code or Codex, can then read the site’s documentation as plain text.

smalldocs does this as Quarto does it, with the same option and the same file names. Quarto added the option in version 1.9.

Turn it on

Set llms-txt: true under website. The key is the same in _smalldocs.yml and in _quarto.yml:

website:
  title: My project
  description: What my project does
  llms-txt: true

The option is off unless you set it. A site without it builds as before, and the option changes no HTML page.

What a build writes

With the option on, a build writes two more kinds of file into the output folder:

A Markdown copy of each page. It goes beside the page, with .llms.md in place of .html. So guide/config.html gets guide/config.llms.md.
llms.txt, at the top of the site. It has the title of the site, its description, and a list of the pages, each with a link to its copy. With site-url set, the links are full addresses.

This site turns the option on. Here are its files:

The index: llms.txt.
This page, llms.qmd, and its copy, llms.llms.md.
The page on writing, markdown.qmd, and its copy, markdown.llms.md. It shows callouts, tabs, tables, maths and footnotes.
A notebook, notebooks.ipynb, and its copy, notebooks.llms.md. The copy holds the code and the saved outputs.

What a copy holds

A copy is the page’s own Markdown, after includes and shortcodes have run. It starts with the title of the page and its description. The front matter is left out.

A link to another page of the site goes to that page’s copy. A link to another site stays as it is.
A fenced div, such as a callout or a tabset, becomes an HTML <div> with the same classes and attributes. Markdown allows HTML, and the Markdown inside the div stays Markdown:
```
<div class="callout-note">

A note.

</div>
```
A span, such as [text]{.smallcaps}, becomes an HTML <span> in the same way.
Code blocks, maths, footnotes, cross-references and the attributes of headings stay as they are written.

A notebook’s copy

A notebook’s copy is made as nbconvert’s Markdown export makes it. nbconvert is Jupyter’s own tool for turning a notebook into other formats.

A code cell is a fenced code block, in the notebook’s language.
Printed text, a plain result and an error are indented by four spaces, Markdown’s other form of code. The colour codes of the terminal are removed.
An HTML output, such as a table from pandas, and a Markdown output stay as they are.
A picture is saved as a file beside the copy, and the copy links to it. So the picture of the ninth cell of notebooks.ipynb is notebooks_files/notebooks_8_0.png.
A picture’s caption, from fig-cap, follows it in a <div class="figure-caption">. Its fig-alt is the picture’s alt text.

The copy shows what the page shows. A cell that echo, output, include or a tag hides on the page is hidden in the copy too.

Content for agents only

The format llms-txt lets a block be only in the copy, or only on the page:

::: {.content-visible when-format="llms-txt"}
Only the Markdown copy has this paragraph.
:::

::: {.content-hidden when-format="llms-txt"}
Only the HTML page has this paragraph.
:::

In the copy, a condition on llms-txt decides alone. Any other condition decides as it does on the page. So a block for html is in the copy, and a block for pdf is not. On the HTML page, llms-txt is a format that does not hold, as pdf does not.

Drafts and the 404 page

A draft that a build leaves out has no copy. A draft with draft-mode: unlinked is built, but it has no copy and no line in llms.txt. The 404 page has a copy, but no line in llms.txt, as in Quarto.

The copies are not pages

The copies are written only into the output folder. smalldocs never reads the output folder as pages, and smalldocs preview does not watch it. So the next build does not read a copy as a page.

A page may link to llms.txt or to a copy, as this page does. smalldocs keeps such a link as it is, and does not report it as broken.

How it differs from Quarto

Quarto makes each copy from the built HTML page, with Pandoc. smalldocs makes it from the page’s own Markdown, which is simpler and keeps what the author wrote. The option, the file names and llms.txt are Quarto’s, so a Quarto site that uses them ports without change. The copies themselves differ from Quarto’s. For example, a callout is a <div> here and a quote in Quarto.