TechCognate
Services ▾
What we
do for you.

Build it. Grow it. Make it visible.

PortfolioBlogAbout
Start a Project
MenuClose ✕
Services
WordPress DevelopmentWooCommerce DevelopmentShopify DevelopmentAI Product DevelopmentBranding & Brand Identity
SEO & Digital MarketingWordPress SEOShopify SEOGuest Post Services
Performance MarketingAI Ads Videos & Creative
PortfolioBlogAbout
Start a project →
Menu

What Is llms.txt and Does Your WordPress or Shopify Site Need One?

By /
What is llms.txt and does your site need one
Home›Blog›AI

Guides · SEO

What Is llms.txt and Does Your WordPress or Shopify Site Need One?

Guide • llms.txt • WordPress and Shopify

If you run a WordPress or Shopify site, you have probably been told to add an llms.txt file so ChatGPT, Claude, Perplexity or Google’s AI features can find you. SEO plugins now offer a one-click toggle, Shopify apps sell generators, and some audits flag a missing file as a problem.

The trouble is that most of that advice skips the key question: which AI systems actually read the file? A plain text file costs little to publish, but time spent on it is time not spent on the things that do affect AI visibility, such as crawler access, clear content and being cited by other sites.

This guide explains what llms.txt is, how the format works, how it differs from robots.txt and XML sitemaps, what Google, OpenAI, Anthropic and Perplexity have said about it, and how to add one on WordPress and Shopify if you decide to. If you want a team to handle technical SEO on your site, see our WordPress SEO services.

Editorial note: this article contains no affiliate links. Facts were checked against official documentation from Google, OpenAI, Shopify, Yoast, Rank Math and the llms.txt proposal on November 10, 2026, with press reports used only to fill gaps and credited where used. AI companies change their crawlers and policies often, so confirm current details before you act.

Quick answer: llms.txt is a proposed Markdown file at your site root that gives AI tools a short, curated map of your most useful pages. As of November 10, 2026, no major AI search engine has publicly confirmed that it uses llms.txt to find, rank or cite websites, and Google says Search ignores it. It is cheap to add and harmless, but it should be a low priority behind crawler access, clear content and schema.

  • Jeremy Howard of Answer.AI proposed llms.txt on September 3, 2024. It is a community proposal, not an official web standard.
  • Google says llms.txt files won’t harm or help visibility in Google Search, “as Google Search ignores them.”
  • OpenAI, Anthropic and Perplexity document their crawlers and robots.txt rules, but we found no official statement from any of them that their search products read llms.txt.
  • Yoast SEO and Rank Math can generate the file on WordPress. Shopify now serves /llms.txt on every store by default.
  • Allowing OAI-SearchBot, Claude-SearchBot and PerplexityBot in robots.txt matters far more for AI search visibility.
15Minute read
12Sections
1Comparison table
Nov 2026Published
Reading map · section length
010203040506070809101112

Each block is one section, sized by how long it is. The orange block is the longest: Which AI Companies Say They Use llms.txt (November 2026) (about 2 min). Tap a block to jump.

What llms.txt Is and Where It Came From

llms.txt is a plain text file written in Markdown and placed at the root of a website, for example https://www.example.com/llms.txt. Jeremy Howard, co-founder of Answer.AI, published the proposal on llmstxt.org on September 3, 2024.

The idea is simple. Language models have limited context windows, and web pages are full of navigation and scripts. The proposal suggests a short, curated Markdown overview with links to the pages that matter most, so an AI tool can find the right content quickly.

The proposal also suggests publishing clean Markdown versions of important pages at the same URL with .md added. It is not a standard from the W3C, IETF or any search engine, and publishing the file does not oblige any AI company to read it.

The llms.txt File Format, Line by Line

The llmstxt.org proposal defines a short, fixed order of elements:

  1. An H1 with the site or project name. This is the only required element.
  2. A blockquote summary. One or two sentences on what the site is and who it is for.
  3. Optional Markdown paragraphs or lists with more detail, without headings.
  4. H2 sections of links. Each item uses the format [name](url): notes, where the notes are optional.
  5. An “Optional” H2 section for secondary links that an AI tool can skip when it needs a shorter context.

Here is an example for a small online store. Replace the names and URLs with your own.

# Northwind Candles

> Northwind Candles is a US online store selling hand-poured soy candles, wax melts and gift sets. We ship within the United States.

Prices are in USD. All candles use soy wax and cotton wicks.

## Products
- [All candles](https://www.example.com/collections/candles): Full candle range with scent notes and burn times
- [Gift sets](https://www.example.com/collections/gift-sets): Bundles for holidays and corporate gifting

## Help
- [Shipping policy](https://www.example.com/policies/shipping-policy): Delivery times and costs
- [Returns policy](https://www.example.com/policies/refund-policy): How returns and refunds work
- [Candle care guide](https://www.example.com/pages/candle-care): Safe burning and wick trimming

## Optional
- [Our story](https://www.example.com/pages/about): How the business started
- [Blog](https://www.example.com/blogs/news): Scent guides and seasonal ideas

A related file you will often see is llms-full.txt. It is not part of the llmstxt.org proposal itself. Documentation platforms popularized it as a single file that contains the full text of every page. Mintlify announced in November 2024 that it automatically generates both files for docs it hosts, and GitBook says it publishes /llms.txt, /llms-full.txt and Markdown versions of every page automatically. For a store or business site, llms-full.txt rarely makes sense, because your full content is already on your pages.

llms.txt vs robots.txt vs XML Sitemaps

These three files are often confused, but they do different jobs. The llms.txt proposal itself says it complements robots.txt and is not a substitute for a sitemap.

FileWhat it doesWho reads itStatus
robots.txtTells crawlers which URLs they may and may not requestSearch and AI crawlers, including Googlebot, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBotInternet standard (RFC 9309), documented by every major AI company
XML sitemapLists the URLs you want crawled and indexed, often with last-modified datesSearch engine crawlersLong-established and supported by Google and Bing
llms.txtGives a short, curated Markdown summary of your most useful pagesNo major AI search engine has confirmed reading itCommunity proposal from September 2024

The practical difference is control. robots.txt changes what AI crawlers can access. llms.txt only offers suggestions.

Which AI Companies Say They Use llms.txt (November 2026)

We looked for official statements from the four companies most site owners care about.

Google

Google has been the most direct. In June 2025, John Mueller wrote on Bluesky, “FWIW no AI system currently uses llms.txt.” In July 2025, Search Engine Roundtable reported Gary Illyes saying at a Search Central Live event that Google does not support llms.txt and is not planning to.

In May 2026, Google published its guide to optimizing for generative AI features in Search, which lists AI text files among the things you do not need to create for AI Overviews and AI Mode. On June 15, 2026, Google added a clarification to that guide. Its Search Central updates page says these files “aren’t needed for Google Search,” won’t negatively or positively affect rankings, and that “it’s fine if you want to maintain these files for other services or systems that use them.” Mueller also described llms.txt as “purely speculative for now” in a Reddit reply, as reported by Search Engine Journal.

There is one wrinkle. Chrome’s Lighthouse tool now has an Agentic Browsing category that includes an llms.txt check, which it describes as checking “for the presence of a machine-readable summary at the domain root.” That category covers browser-agent readiness, not Search rankings. Some Google developer sites also publish llms.txt files, which Mueller said in January 2026 is not an endorsement.

OpenAI

OpenAI documents three main crawlers. OAI-SearchBot surfaces websites in ChatGPT search, GPTBot collects content that may be used to train its models, and ChatGPT-User fetches pages when a user asks ChatGPT to. Its crawler page does not mention llms.txt as a way to be found in ChatGPT. OpenAI’s developer docs link to their own llms.txt index, which shows OpenAI publishes the file, not that ChatGPT reads yours. In July 2025, Search Engine Roundtable reported one SEO’s server logs showing OpenAI requesting llms.txt files, with no official comment from OpenAI. We found no OpenAI statement confirming use.

Anthropic

Anthropic updated its crawler documentation in February 2026 to describe ClaudeBot for training, Claude-User for fetches requested by users and Claude-SearchBot for improving search results, as reported by Search Engine Roundtable. That documentation explains robots.txt controls. We found no Anthropic statement that Claude’s search uses llms.txt. Anthropic’s developer docs publish an llms.txt file, but again, that is publishing, not reading.

Perplexity

Perplexity’s help center says PerplexityBot respects robots.txt and will not index content from sites that disallow it. The page does not mention llms.txt, and we found no official Perplexity statement about using it.

Where llms.txt does help today

llms.txt is most useful in developer documentation. Developers can point a coding assistant at a docs site’s llms.txt or llms-full.txt to load an API reference, which is why Mintlify and GitBook generate the files. That is different from an AI search engine discovering your store.

How to Add llms.txt on WordPress

You have three options. Use only one, because two tools writing the same file will conflict.

Option 1: Yoast SEO

Yoast added llms.txt to the free version of its plugin in June 2025. To turn it on:

  1. Go to Yoast SEO → Settings → Site features.
  2. Under AI tools, find llms.txt and turn the toggle on.
  3. Click Save changes, then visit yoursite.com/llms.txt to check it.

By default, Yoast’s developer specification says the file lists your five most recently updated posts and pages from the last 12 months, prioritizing cornerstone content, plus your five most used categories or tags. A scheduled action updates it weekly. You can switch to Manual page selection under Customize llms.txt file to choose pages yourself. Yoast writes a physical file to your site root, will not overwrite an existing llms.txt, and does not support the feature on multisite.

Option 2: Rank Math

Rank Math includes an LLMS Txt module. To set it up:

  1. Go to Rank Math SEO → Dashboard and enable the LLMS Txt module. Update the plugin if you don’t see it.
  2. Open Rank Math SEO → General Settings → Edit llms.txt.
  3. Choose the post types and taxonomies to include. Noindexed posts are left out automatically.
  4. Set the posts and terms limit (the default is 100) and add any custom links under Additional Content.
  5. Save, then open the file link at the top of the tab to preview it.

Option 3: Upload a file manually

Write the file in a text editor using the format above, save it as llms.txt in UTF-8, and upload it to the same folder as wp-config.php using your host’s file manager or SFTP. You get full control but must update it by hand. Turn off any plugin llms.txt feature first.

How to Add llms.txt on Shopify

Shopify used to make this hard, because merchants could not place files at the domain root, so older workarounds relied on apps or URL redirects. That changed in 2026.

According to Shopify’s developer changelog dated May 28, 2026, every store now includes a default /agents.md file, and /llms.txt and /llms-full.txt serve that same content by default. Shopify’s documentation says the default file covers commerce protocol discovery, read-only links for browsing products, collections and search, and links to your public policies. It is served on your primary domain only and leaves out contact details.

So most Shopify merchants already have one. To customize it:

  1. In Shopify admin, go to Online Store → Themes, click the menu on your live theme and choose Edit code.
  2. Add templates/agents.md.liquid to change all three files at once, or templates/llms.txt.liquid to change only /llms.txt.
  3. Write your content in Markdown. Only the request and agents Liquid objects are available, so global objects such as shop and collections will not render. Hard-code your key links.
  4. Save, then visit yourstore.com/llms.txt to check the result.

Once you add a custom template, you maintain it yourself, and it replaces Shopify’s generated default. If you installed an llms.txt app or set up a redirect before May 2026, check what /llms.txt shows now and remove anything you no longer need. Yoast SEO for Shopify also has an llms.txt setting, but a separate app is rarely necessary now. If you would rather hand this to specialists, our Shopify SEO services cover crawl settings and AI visibility.

What Matters More for AI Visibility

The AI companies’ own documentation points to familiar basics.

Let the right AI crawlers in

OpenAI says OAI-SearchBot is used to surface sites in ChatGPT search, and that robots.txt changes can take about 24 hours to take effect there. Perplexity says PerplexityBot follows robots.txt. Anthropic’s crawler documentation, as summarized by Search Engine Roundtable, says blocking Claude-SearchBot may reduce your visibility in search results. If your robots.txt, firewall or CDN bot protection blocks these crawlers, your site cannot be cited, no matter what llms.txt says.

You can treat training and search separately. Blocking GPTBot or ClaudeBot opts you out of model training without blocking OAI-SearchBot or Claude-SearchBot. On WordPress, you can edit robots.txt through Yoast or Rank Math. On Shopify, add a robots.txt.liquid template. Shopify warns that incorrect edits can cost you all traffic, so change it carefully.

Understand what Google-Extended does

Google-Extended is a robots.txt token, not a separate crawler. Google says it controls whether your content is used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google also says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.” Blocking it does not remove you from AI Overviews or AI Mode, which use the normal Googlebot crawl.

Publish content AI tools can quote

Google’s AI optimization guide frames its generative AI features as an extension of normal SEO. That means clear headings, direct answers near the top, accurate product details and content rendered in the HTML.

Add structured data and earn citations

Accurate Product, Organization and FAQ schema helps search engines understand your pages. Mentions and links from respected sites in your niche make it more likely that AI answers cite you. For more on schema tooling, see our guide to the best product schema tools, and for a full strategy see our LLM SEO services.

AI visibility checklist

Work through these before you spend time on llms.txt.

☐ robots.txt allows OAI-SearchBot, Claude-SearchBot and PerplexityBot
☐ Your firewall or CDN is not blocking those crawlers by default
☐ Key pages are indexed in Google and Bing
☐ Product and policy pages state prices, shipping and returns clearly
☐ Product and Organization schema validate without errors
☐ Important content loads in the HTML, not only through scripts
Then: add or review llms.txt as a small, final step.

Does Your Site Need llms.txt?

Add it, leave it or skip it?

Based on public statements and platform documentation as of November 10, 2026.

Worth adding if
→ You run developer docs, an API or a software product
→ Your SEO plugin can generate it in one click
→ You want a tidy summary ready for any tool that adopts it
→ Your crawler access and schema are already in good shape
Low priority if
→ You run a store or local business site
→ You are on Shopify, which already serves a default file
→ Your robots.txt or CDN blocks AI search crawlers
→ Someone is charging you to “optimize” it for rankings
Rule of thumb: llms.txt will not hurt you, but it will not fix weak content or blocked crawlers either. Spend your first hour on the checklist above.

Frequently Asked Questions

What is llms.txt in simple terms?

It is a short Markdown file at your site root that lists your most useful pages for AI tools, along with a one-line summary of your site.

Does llms.txt improve Google rankings?

No. Google says Search ignores llms.txt files, so they neither help nor hurt rankings, AI Overviews or AI Mode.

Does ChatGPT read llms.txt?

OpenAI has not said so publicly. Its crawler documentation covers robots.txt and OAI-SearchBot, not llms.txt.

Is llms.txt the same as robots.txt?

No. robots.txt controls what crawlers may access. llms.txt only suggests which pages are most useful and cannot block anything.

Does Shopify create llms.txt automatically?

Yes. Since May 2026, Shopify serves /llms.txt by default, mirroring each store’s /agents.md file. You can customize it with a theme template.

Do I need llms-full.txt too?

Usually not. It mainly suits documentation sites.

Is the Yoast llms.txt feature free?

Yes. Yoast says the feature is available in the free version of Yoast SEO.

Should I block GPTBot or Google-Extended?

That depends on whether you want your content used for model training. Blocking them does not block OAI-SearchBot or remove you from Google Search. Our LLM SEO team can review your crawler rules with you.

Final Take

llms.txt is a sensible idea with a simple format, and it has a real home in developer documentation. But as of November 10, 2026, no major AI search engine has publicly confirmed that it uses the file to find, rank or cite websites, and Google says Search ignores it. Treat it as optional housekeeping, not an AI SEO tactic.

If you are on WordPress, turn on the Yoast or Rank Math feature if you like, keep the list short, and move on. If you are on Shopify, you already have one. Then put your real effort into crawler access, clear pages, schema and citations, which every AI search provider’s documentation points back to. If you want help with that work, our Shopify SEO services and WordPress SEO services can take it on.

Sources

Jaykishan Panchal, Founder & CEO, TechCognate
About the author

Jaykishan Panchal

Founder & CEO, TechCognate · SEO Strategist & WordPress Expert

Jaykishan Panchal is the founder of TechCognate and an SEO strategist with 15+ years of hands-on experience driving organic growth for businesses worldwide. He specialises in technical SEO, content strategy and GEO (Generative Engine Optimisation), helping brands stay visible in both traditional search and AI-driven discovery.

  • Technical SEO
  • Content Strategy
  • AI & GEO
  • Google Ads
  • E-Commerce SEO
LinkedInX / @TechCognate

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top