LLMs.txt: Setting Up High-Quality Site Interaction with AI
Any site should be as clear and user-friendly as possible for the people who use it. This is one of the main requirements for web developers. Today, when websites are accessed not only by people but also by AI, web developers have a new challenge: to make obtaining information from them equally comfortable for a human user and a large language model (LLM). To address this challenge, the LLMs.txt tool has been developed. In this article, we will tell you what it is and why it is important to implement it.
What is LLMs.txt
LLMs.txt is a tool for improving LLM interaction with the website. It is a text file, designed for AI systems that can understand and analyze information. It acts as a guide, showing them where the most important things are on the site.
The purpose of the LLMs.txt file is almost the same as that of the sitemap.xml and robots.txt documents, which help search engines. It offers for large language models structured information about a specific website in an easily readable and analyzable format. Why is another file needed when sitemap.xml already lists all available pages? It turns out that such a list does not give AI any understanding of their content. The task of robots.txt is to control search engine access to the site. This file specifies which pages they are allowed to index. However, it does not help at all in analyzing the content placed on them. The LLMs.txt document excludes everything unnecessary (HTML fragments, advertising inclusions, JavaScript syntax) and provides AI with data in a machine-readable structured form.
LLMs.txt exists in 2 varieties:
- /llms.txt – provides a simplified site structure.
- /llms-full.txt – provides comprehensive details on all site content.
Although this file is intended for websites, it can also be used to enhance other digital assets that need to be made more accessible to AI. It allows companies to make it easier for AI tools to process their internal documentation, API instructions, and entire cloud storage.
Why LLMs.txt was Developed
In September 2024, Jeremy Howard, one of the founders of the Answer.AI research lab, publicly proposed the idea of creating the LLMs.txt document. According to the developer, it should solve an important technical problem that every LLM faces today.
- Automate the work of an online store or landing
- Empower through integration
- Don't spend money on programmers and integrators
- Save time by automating routine tasks
As you know, the context window of any AI has limitations, so it is difficult for it to process very large volumes of content and documentation. When a language model analyzes a web page, the limited resource of the context window is spent, in particular, on navigation elements, styles, scripts, etc. These components are unnecessary and useless to it. The LLMs.txt file solves this problem by directing the AI system exclusively to the information it needs, cutting off everything unnecessary. The data in it is formatted in a way convenient for machine analysis.
Jeremy Howard also suggested creating AI-friendly versions of web pages that contain information useful for LLMs. The AI-optimized page should have the same URL as the original, but with the mandatory .md extension at the end. This extension stands for “Markdown documentation.”
How to Create LLMs.txt
For the LLMs.txt file to work correctly, it must be created according to the rules. The most important of them is that you can only use the Markdown language. In addition, you must adhere to a single style of document design. It should not contain messy lists and randomly structured paragraphs that are in no way connected to each other.
Step-by-step guide:
- The first line is the H1 (#) heading. This is where you write the name of your site or project.
- The second line is an information block in quote format (>). Here, briefly describe what your site is about and what it contains. 1-3 sentences are enough.
- Next, create logical blocks (for example, “Main Documents”, “Products”). The name of each block is formatted as H2 (##).
- Each logical block should be accompanied by a corresponding link and description. For example, if you created a “Products” block, it should contain a link to your web page and a brief description of the content that is there. You should write blocks according to the following pattern:
[Document name](URL): Short description. - If necessary, you can supplement the file with additional sections (“Examples”, “Privacy Policy”, etc.).
The LLMs.txt file can be created either manually or using special tools. If your site consists of a few pages, and you want to decide for yourself what to include in the document, choose the manual option. Using an automatic generator is optimal for large websites. It will reduce the risk of errors and save time. To check if you did everything correctly, visit llmstxt.directory and view files created by other companies.
Tools for creating LLMs.txt:
- llmstxt.firecrawl.dev – an online generator for large-scale platforms with hundreds of pages.
- llmstxtgenerator.org – an online generator for creating a basic llms.txt for small sites.
- nbdev by Fast.ai – a developer tool that automatically creates .md files.
- LLMs.txt for WP – a WordPress extension that supports the .md format and also automatically generates llms.txt for sites created on this platform.
How to Add a File to a Website and Check Its Availability
The LLMs.txt file should be located in the root directory of the web resource. Example: www.mywebsite.com/llms.txt
After uploading the file to the root of the site, you should check its availability. To do this, open a browser and enter www.mywebsite.com/llms.txt in the address bar. You can also use console commands for this purpose:
- curl -I http://www.mywebsite.com/llms.txt
- wget --spider http://www.mywebsite.com/llms.txt
- http --head http://www.mywebsite.com/llms.txt
If the accessibility check shows that the file cannot be opened, check the access rights, file upload, and server settings.
What Benefits Does the Site Owner Get from LLMs.txt
By adding LLMs.txt to the file structure of their site, its owner gains several significant benefits:
- Increased attention from AI platforms. Chatbots (ChatGPT, Claude, Copilot, Perplexity, etc.) may be more likely to link to your site in their responses to users.
- Quick access and high-quality interpretation of information. Having a “guide” helps LLMs find the content they need faster and can better explain it to users. This is especially important if your resource has documents, instructions, or an FAQ section.
- Reduced AI errors. If AI assistants frequently use your website as a source of information, LLMs.txt will allow them to respond as accurately as possible. Chatbots will no longer have to make up things due to data access issues and upset users with their inaccurate responses.
- Increased user trust. Continuous AI recommendations of relevant content from your site in responses will increase traffic to the resource and enhance the level of user trust in it.
- Resource optimization. Once the AI has access to LLMs.txt, it will no longer scan the entire site, loading the server. It will only access the content specified in the file. Under certain conditions, this can reduce your hosting costs.
The active implementation of generative artificial intelligence models on the web cannot be ignored. Today, using SEO to improve the visibility of a site in search engines is no longer enough. The readability of a resource for AI is no less important than classic optimization methods. LLMs.txt is a strategic solution that effectively adapts your site to high-quality interaction with it.
Time is the most valuable resource for business today. Almost half of it is wasted on routine tasks. Your employees are constantly forced to perform monotonous tasks that are difficult to classify as important and specialized. You can leave everything as it is by hiring additional employees, or you can automate most of the business processes using the ApiX-Drive online connector to get rid of unnecessary time and money expenses once and for all. The choice is yours!