llms.txt is a proposed Markdown file that gives compatible AI agents a concise guide to important website resources. It can make agent navigation easier, but it is not a ranking factor, does not control crawlers, and is not required for Google Search or Google’s generative AI features.
Early AI SEO discussions often called it “robots.txt for AI,” but 2026 evidence is more nuanced: adoption is real in documentation and agent workflows, while consumer AI-search citation benefits remain unproven.
For businesses investing in Generative Engine Optimization, the practical question is not simply whether the file exists. The better question is whether it solves a useful agent-access problem after technical accessibility, strong content, entity clarity, authority, and crawler access are already in place.
What Is llms.txt?
Jeremy Howard proposed the format in September 2024. Version 2 was published on August 10, 2026. The specification describes a Markdown resource that provides brief context, guidance, and links to more detailed resources an AI agent may need while completing a task.
The file is intended primarily for inference-time use. A compatible agent can inspect it, identify the resource most relevant to a user’s request, and then fetch that content rather than processing a large portion of the website.
This makes it better understood as an agent-readable navigation layer than as another SEO metadata file. For broader questions about how large language models retrieve, mention, and cite brands, our LLM SEO guide covers the wider strategy.
Is It an Official Web Standard?
No. It remains an open proposal and emerging convention rather than an IETF or W3C standard. The specification is still open to community input, and AI platforms are not required to request, parse, trust, or use the file.
Adoption is still real. The specification lists integrations with Mintlify, GitBook, Yoast SEO, AIOSEO, Wix, and other platforms, while OpenAI, Anthropic, and Gemini developer documentation publish the format for their own docs. That demonstrates a documentation use case, not a confirmed consumer-search ranking signal.
What Changed in Version 2?
The biggest change is scope. A file can now sit at the root or inside a subpath. For example, /llms.txt can describe the overall site while /docs/llms.txt can describe resources under /docs/. The proposal says the most specific applicable file should be preferred.
Version 2 also recommends clean Markdown alternatives and link relationships such as rel=”alternate” for Markdown versions and rel=”describedby” for the descriptive file. Because path-level behavior is newly specified, support may vary; a clean root file remains the safest baseline.
What Should the File Contain?
The specification is intentionally lightweight. An H1 with the project or site name is the only required section. It can then include a short blockquote summary, explanatory text, H2 sections, and Markdown links with concise descriptions.
| Element | Purpose |
| H1 | Identifies the site, project, or section |
| Summary | Gives essential context |
| Supporting text | Adds useful guidance |
| H2 sections | Groups related resources |
| Markdown links | Points agents to detailed information |
| Optional section | Holds secondary resources |
The goal is curation, not keyword coverage. A smaller set of authoritative resources is usually more useful than copying an entire sitemap into Markdown. The same principle applies to technical SEO, where crawlability, indexability, and clear architecture remain higher priorities.
A Simple llms.txt Example
# Example SaaS Platform
> Example SaaS provides scheduling and workflow software for service businesses.
Use the resources below for current product, pricing,
integration, and support information.
## Product
– [Product Overview](https://example.com/product.md): Main features and use cases.
– [Pricing](https://example.com/pricing.md): Current plans and billing information.
## Documentation
– [API Documentation](https://example.com/docs/api.md): Authentication, endpoints, and examples.
– [Integrations](https://example.com/docs/integrations.md): Supported integrations and setup instructions.
## Optional
– [Company Updates](https://example.com/news.md): Product and company announcements.
Descriptions should be factual rather than promotional: the goal is to help an agent find the right resource, not sell the brand.
llms.txt vs robots.txt, Sitemap.xml, and Schema
These technologies solve different problems.
| Technology | Main Role |
| robots.txt | Communicates access rules to compliant crawlers |
| sitemap.xml | Helps search engines discover important URLs |
| llms.txt | Curates resources for compatible AI agents |
| Schema markup | Adds structured meaning to page content |
The specification explicitly says robots.txt and this format have different purposes. Robots rules deal with acceptable automated access; the newer resource provides context and links that an agent may use on demand.
Likewise, a sitemap generally lists indexable human-readable pages, while the AI-oriented format can be more selective and may point toward Markdown resources optimized for efficient agent retrieval.
If a site still has crawl, canonical, indexing, rendering, or migration problems, those issues should normally outrank this experiment. Dexora’s Technical SEO service focuses on those established foundations first.
Does llms.txt Actually Work?
It works technically when a compatible agent deliberately fetches it, reads the structure, and follows the resources it identifies. What remains unproven is whether publishing the file creates measurable improvements in rankings, AI citations, referral traffic, or model accuracy.
The strongest large-scale usage study available in 2026 comes from Ahrefs. It analyzed 137,210 domains using Ahrefs Web Analytics in May 2026. About 28% of that sample had a valid file, yet 97% received zero requests during the month. Ahrefs cautions that its audience skews technical, so 28% should not be treated as a global adoption rate.
Even when a file is fetched, that proves access only—not use in retrieval, ranking, inference, or citations.
This is the correct 2026 conclusion: adoption is real, but proven AI-search impact remains limited.
Does Google Use It?
Google’s current guidance is explicit: Google Search does not use special AI text files such as llms.txt for rankings or visibility, including AI Overviews and AI Mode. Creating one for another service neither helps nor harms Google Search performance.
Google instead recommends the same foundations that support Search overall: crawlable and indexable pages, unique and useful content, clear technical structure, strong page experience, and established SEO best practices.
For businesses targeting Google’s generative results, our Google AI Overviews optimization guide covers the factors Google actually documents as relevant.
Does ChatGPT Require It?
No. OpenAI’s documented requirement for ChatGPT Search focuses on OAI-SearchBot, not this file. OpenAI says OAI-SearchBot is used to surface websites in ChatGPT search results and recommends allowing it in robots.txt for sites that want search inclusion.
OpenAI separately identifies GPTBot for content that may be used to improve foundation models, while ChatGPT-User handles certain user-triggered requests. These controls are independent.
This distinction matters for AI Search SEO: crawler eligibility, retrieval, citations, and model training are separate processes.
What About Perplexity, Claude, and Gemini?
Perplexity’s official guidance focuses on PerplexityBot for surfacing and linking websites in search results and does not document an llms.txt requirement. Anthropic and Gemini developer documentation publish the format, demonstrating documentation adoption rather than a confirmed consumer-search ranking benefit.
For broader multi-platform visibility, how to rank in AI search engines explains why authority, retrievability, entity clarity, and useful content matter beyond any single file.
Where Is It Most Useful?
The specification says the format is used most heavily for software documentation, where coding agents follow curated links to API references and tutorials. Documentation-heavy SaaS, API platforms, developer tools, and large knowledge bases are therefore the clearest candidates.
This prioritization follows the same principle explained in why technical SEO is the foundation of long-term search success: new enhancements are valuable only after the underlying website can be accessed, understood, and maintained properly.
How to Implement and Validate It
Start by choosing the resources an agent genuinely needs. Write a concise site description, group those resources under meaningful headings, and use factual link descriptions. Avoid turning the file into a sitemap copy, keyword list, or sales page.
After publishing, confirm that the URL returns successfully and every linked resource is current, public, and intentional. If Markdown alternatives are provided, test them separately.
Server or CDN logs can show whether bots request the file, but interpretation must remain conservative. A request proves the resource was fetched; it does not prove that the system used it in an answer.
For websites reviewing crawler access, technical health, content quality, and AI-search readiness together, Dexora recommends auditing the underlying search foundation before prioritizing experimental enhancements.
Limitations and Risks
The main limitation is simple: a perfect file has no practical value to a system that never requests it.
It also creates maintenance obligations. Outdated links, retired services, stale documentation, or incorrect product information can make the machine-readable guide less reliable. Ahrefs additionally notes that exposing a curated map of important resources can make content easier for automated systems or scrapers to discover.
The format cannot force an AI model to trust or cite a brand. Strong entity SEO and structured data remain separate parts of machine-readable brand clarity.
Should You Implement It in 2026?
For many established websites, the answer is yes, if implementation is inexpensive and the use case is clear—but treat it as an experimental agent-readability layer, not an AI ranking shortcut.
Prioritize crawlability, indexability, useful original content, internal architecture, platform-specific crawler access, entity consistency, and authority first. Then add this resource where it can make complex information easier for compatible agents to navigate.
Frequently Asked Questions
What is llms.txt?
It is a proposed Markdown resource that gives compatible AI agents concise website context and links to important content. It is intended to improve agent navigation, not traditional search rankings.
Is llms.txt an official web standard?
No. It is an emerging open proposal rather than an IETF or W3C standard. Platforms can support, ignore, or implement it differently as the agent ecosystem continues evolving.
Does llms.txt improve Google rankings?
No. Google states that Search ignores special AI text files for ranking and visibility. Publishing one for another platform neither improves nor harms performance in Google Search.
Does Google AI Overview require llms.txt?
No. Google says its generative Search features rely on established Search systems and normal SEO requirements. A page should be crawlable, indexable, useful, and eligible for Google Search.
Does ChatGPT require llms.txt?
No. OpenAI’s published guidance focuses on allowing OAI-SearchBot for ChatGPT Search inclusion. The company does not document this file as a requirement for rankings, summaries, or citations.
Is llms.txt the same as robots.txt?
No. Robots directives communicate crawler access preferences. This newer format provides context and curated resource links for compatible agents but does not grant, deny, or control crawler access.
Is llms.txt the same as an XML sitemap?
No. XML sitemaps help search engines discover URLs, while this format offers a smaller, curated guide to resources that compatible AI agents may find especially useful.
What is llms-full.txt?
It is an additional convention used by some documentation platforms to provide a larger combined Markdown export. It is not a required component of the current llms.txt v2 specification.
How can I measure whether AI bots use the file?
Check server, CDN, or bot-analysis logs for requests to the file and linked resources. Remember that a fetch proves access only; it does not prove retrieval or citation.
Should a local business create llms.txt?
It can, but most local businesses should first prioritize accurate service information, indexing, local authority, reviews, strong location pages, internal linking, and platform-specific crawler accessibility.
Final Verdict
The 2026 evidence supports a balanced position. The format is real, actively evolving, and useful in some documentation and agent workflows. Major platforms publish it for their own developer resources, and Chrome recognizes it as an optional agentic-browsing convention.
At the same time, Google Search explicitly ignores it for rankings, OpenAI documents OAI-SearchBot rather than this file as the mechanism for ChatGPT Search crawling, and large-scale log data shows that most published files in Ahrefs’ sample were never requested.
So do not implement it because someone calls it “the next robots.txt.” Implement it when it gives compatible agents a cleaner route into valuable content, keep it accurate, and measure real usage.
Want to Know If Your Website Is Ready for AI Search?
Publishing an AI-focused file is only one small part of modern search visibility. Your website still needs strong technical SEO, crawlable content, clear entity signals, authoritative information, and correct access for relevant search and AI crawlers.
Get a free SEO + AI visibility audit from Dexora Digital to identify what is actually limiting your visibility across Google, ChatGPT, Perplexity, and emerging AI search experiences and which improvements deserve priority first.
