Why llms.txt Won't Help Your AI Visibility, and Where the Real Shift Is

llms.txt is a Markdown file meant to give AI models a short summary of a website, and it does not improve visibility in search or in AI answers. Google says so in its own documentation, and server logs show that AI crawlers do not even request it. The change websites need to prepare for is agentic browsing: agents that read, click and act on pages for the user.

  • Google Search lists llms.txt among AI optimization myths: the file is not needed to appear in AI Overviews or AI Mode.
  • Google states that optimizing for generative AI search is still SEO.
  • On a site with several million monthly visits, AI crawlers made zero requests to /llms.txt in August 2026.
  • Even the experimental agent audit in Chrome Lighthouse marks a missing llms.txt as not applicable, not as a failure.
  • The work that matters now is making pages readable and usable by browser agents.

llms.txt Was Proposed as a Shortcut for Language Models

An llms.txt file is a plain Markdown document placed at the root of a site, at /llms.txt. It was proposed in 2024 as an open convention, with a simple idea: web pages are built for people, so a short curated file would help language models understand a site faster.

The format has three main parts:

  • An H1 heading with the name of the site.
  • A short summary in a blockquote, with the key information a model needs first.
  • Sections of links in Markdown to the most useful pages, sometimes with Markdown copies of those pages.

The idea is easy to understand, and many sites added the file in 2024 and 2025 in the hope of being cited more often by AI assistants. Two years later, the evidence points the other way.

Google Says llms.txt Is Not Needed for Search or AI Features

Google’s position is explicit. Its guide to optimizing for generative AI features, updated in July 2026, lists llms.txt among the myths: you do not need new machine readable files, AI text files, markup or Markdown to appear in Google Search. The guide adds that creating the file will neither help nor harm visibility in Search.

Its documentation on AI features says the same for AI Overviews and AI Mode: there are no additional requirements and no special optimizations to appear in them. The guide sums it up in one line that frames the whole debate: optimizing for generative AI search is still SEO.

Google’s John Mueller also said in January 2026 that the llms.txt files present on some of Google’s developer documentation are not an endorsement of the format for Search. In other words, an llms.txt file will not move rankings, will not make a page eligible for AI Overviews and will not replace the work on crawlability, content and authority that selection depends on.

Server Logs Show AI Crawlers Ignore llms.txt

What Google says is confirmed by what crawlers do. On a site we work with that receives several million visits a month, the AI crawler logs for August 2026 show zero requests to /llms.txt. None of the crawlers monitored, from Google, OpenAI, Microsoft, Anthropic, Apple, Perplexity and others, requested the file even once.

AI crawler requests to /llms.txt over 30 days: zero from every crawler

AI crawler requests to /llms.txt in August 2026 on a site with several million monthly visits: zero, whatever the crawler.

For a site of that size, this is not a lack of AI crawler activity: the file is simply not what these crawlers look for. Checking this on your own site takes a few minutes:

  • Filter your server logs on the /llms.txt URL over a full month.
  • Group the requests by user agent and keep only AI crawlers and assistants.
  • Compare with their requests to your key pages over the same period.

Chrome Lighthouse Confirms llms.txt Is Optional

Where Google does check for the llms.txt file is Chrome Lighthouse, in an Agentic Browsing category added in 2026. Even there, the file is treated as optional:

  • A missing file is not a failure: the audit fails only when the server returns an error while fetching llms.txt. A 404 is marked not applicable.
  • It is one check among several: the category mostly evaluates WebMCP, accessibility for agents and layout stability.
  • It is experimental: the category relies on proposed standards and requires Chrome 150 or later.

The Lighthouse documentation argues that the file can save agents some crawling time. But an audit that does not penalize its absence says a lot about its weight. For visibility, Google Search has already given the answer: no effect.

Agentic Browsing Is the Change That Actually Matters

llms.txt tried to answer a real question: how will AI read websites? The answer is arriving, and it is not a text file. With agentic browsing, assistants open pages, read them and complete tasks for the user. Chrome auto browse has been available since January 2026 for Google AI Pro and Ultra subscribers in the US, and the ChatGPT desktop app has included a built-in browser since July 2026.

These AI agents use the page itself. Google’s guide notes that browser agents may access websites through screenshots, DOM inspection and the accessibility tree. What they need is therefore on the page, not in a separate file:

  • Readability: semantic HTML, clear labels on buttons and form fields, key information stated as text.
  • Actions: forms that are easy to complete and, for advanced sites, WebMCP, a proposed standard still experimental in Chrome that lets a site declare tasks as tools an agent can call.
  • Commerce: clean product feeds and emerging protocols such as the Universal Commerce Protocol for AI shopping.
  • Discovery: the classic SEO work that makes pages retrievable, which remains the entry ticket for AI search.

Ranking these layers against real agent traffic and business priorities is the work of a GEO and GSO practice.

Where to Put the Effort Instead of an llms.txt File

If your site already has an llms.txt file, there is no need to remove it: Google says it does no harm. The point is to stop spending time on it and to stop presenting it as an AI visibility lever. The same hours are better used elsewhere.

Action Effect on AI visibility or agent use
Creating or updating llms.txt None on Google Search and AI features, according to Google
SEO fundamentals: crawlability, content, authority Required to appear in AI Overviews and AI Mode
Allowing OAI-SearchBot in robots.txt Required to be shown in ChatGPT search answers
Accessible markup and labeled forms Lets browser agents read the page and complete tasks
Monitoring AI crawlers in server logs Shows what AI systems actually read on your site

Leave llms.txt Aside and Prepare Your Pages for Agents

llms.txt was an early attempt to make websites readable by AI. Google has said it does not help search or AI features, and server logs show that AI crawlers do not use it. The real shift is agentic browsing, where agents read the page itself and act on it. Across 300+ global brands and 15 years of organic work, the priority is clear: strong SEO foundations first, then pages that agents can read and use.

A 30 minute review of your AI crawler logs and of your key journeys will show what AI systems read on your site today and what to fix first for agents.

FAQ: Common Questions About llms.txt

  • What is llms.txt?

    llms.txt is a Markdown file placed at the root of a website, at /llms.txt, that gives AI models a short summary of the site and a list of links to its main pages. It was proposed in 2024 as an open convention. Google Search does not use it, and it does not replace robots.txt or sitemaps.

  • Does llms.txt help SEO?

    No. Google Search states that you do not need new machine readable files, AI text files, markup or Markdown to appear in Google Search, including AI Overviews and AI Mode. According to Google, the file will neither help nor harm visibility. Rankings and AI citations still depend on SEO fundamentals.

  • Do AI crawlers read llms.txt?

    Rarely, if at all. On a site with several million monthly visits, AI crawlers from the main AI companies made zero requests to /llms.txt in August 2026. Server logs are the simplest way to check what happens on your own site, by filtering on the /llms.txt URL over a full month.

  • Should I delete my llms.txt file?

    There is no need to. Google says the file does no harm, so an existing llms.txt can stay online. The better decision is to stop investing time in it and to focus on SEO fundamentals, crawler access and pages that browser agents can read and use.

Read our insights