llms.txt: Standardized Site Information Documentation for Large Language Models

General Introduction

llms.txt is a standardized document format designed specifically for Large Language Models (LLMs) to help websites provide concise, structured information that can be easily and efficiently used by LLMs in the reasoning process. This specification is supported by well-known companies such as Cloudflare and Anthropic. By using the llms.txt file, websites can provide AI programming tools with accurate contextual information to quickly implement targeted features.

Function List

• Provide a standardized document format for easy LLM parsing
• Supports a variety of application scenarios, such as API documentation, developer guides, etc.
• Compatible with existing web standards such as sitemaps and robots.txt
• Helping websites improve accessibility and utilization of information
• Support for integration of automation tools and AI programming tools

Using Help

Installation process

The llms.txt file does not need to be installed, just create a Markdown file named llms.txt in the root directory of your website and fill in the content according to the standard format.

Usage Process

1. Creating the llms.txt file : Create a Markdown file called llms.txt in the root directory of your website.
2. Fill in the background information : Provide concise background information in the document to help LLM understand the main content and purpose of the site.
3. Adding guidance content : Provide instructional content on how to use the site to help LLMs better parse and utilize the information on the site.
4. Link Details : Add links to detailed Markdown files in the llms.txt file that can contain more specific information such as API documentation, product descriptions, etc.
5. Keeping documents up to date : Update the llms.txt file regularly to ensure that the information is accurate and current.

typical example

# llms.txt
## Background Information
This site provides detailed documentation and guidance content on the use of the Large Language Model (LLM) and is intended to help developers and users better understand and utilize LLM technology.
## Guidance Content
- How to use the API documentation on this site
- Frequently Asked Questions
- Developer's Guide
## Links
- API Documentation
- Developer's Guide
- Frequently Asked Questions

consultation

https://developers.cloudflare.com/llms-full.txt

https://docs.anthropic.com/llms-full.txt

https://docs.play.ai/llms-full.txt

By following these steps, you can easily create a llms.txt file that will help LLM better utilize your website information.

Related news: Simplifying AI Documentation with /llms.txt

Our team recently released support for /llms.txt and /llms-full.txt, which means that Mintlify now automatically generates and hosts all documents in plain text format for easier ingestion by Large Language Models (LLMs).

In September 2024, Jeremy Howard, co-founder of Answer.AI, proposed the /llms.txt file format as a standardized way for websites to provide key information to LLMs. This Markdown-based file distills a website's content into a clean, structured format, eliminating the need for LLMs to parse complex HTML, advertisements, or JavaScript.

By adopting this standard, we are moving toward a future in which each company will provide two versions of the document: one for human use and another for LLM use.

Documentation is about delivering answers

Until AI comes along, finding answers from documents can be like a guessing game.

Searching in the documentation depends on using the right keywords. If you're searching for "invoice editing" but the document refers to it as "billing adjustments", you're out of luck. Without an effective search, customers often have to struggle to decipher the navigational hierarchy and navigate endless pages.

One might try to turn to a third-party search engine to find the right document or relevant community forum. But this relies on strong SEO and also chance to use the right keywords - another roll of the dice.

At Mintlify, we specialize in addressing these challenges through the following:

• SEO optimized: for example, we offer hosting in subdirectories instead of subdomains.
• AI Chat: Our chat widgets use your documents as a knowledge base to intelligently answer questions without the need for exact keywords.

Help AI help you

Now, we're giving LLM a better way to work with documents and help people use AI tools to find and apply information about your products:

/llms.txt: Massively Friendly Navigation for AI

This auto-generated Markdown file provides a simplified view of your entire document navigation, allowing for features like the ChatGPT maybe Perplexity Such LLMs make it easier to index your content. Think of it as SEO for AI - users can now find product-specific information directly through generic LLMs.

/llms-full.txt: all documents in one place

This automatically generated file compiles all of your document text into a single Markdown file.

Some AI tools allow you to paste links to load information directly into their context windows. For example, you can make your document available to a tool like the Codeium Such an AI coding assistant makes it possible to provide context-aware suggestions based on specific details of your product, such as your authentication process.

Instead of pasting dozens of links to enrich the helper, you can use /llms-full.txt to provide a comprehensive file.

the next step

By adopting the /llms.txt standard, we are pleased to help lay the groundwork for its widespread use throughout the industry.

These releases are just the beginning. We will continue to invest in new ways that AI can make documents more useful and accessible - stay tuned for major updates coming soon.