llms.txt Guide - GEO

What is llms.txt?

llms.txt is a proposed standard file format designed to help large language models (LLMs) easily discover, read, and understand the core content of a website.

Think of it as a streamlined, AI-friendly table of contents for a website—similar in concept to a robots.txt file or an XML sitemap, but built specifically for artificial intelligence.

Jump directly to our hand-picked links on ➡️ llms.txt

Jump directly to our info on ➡️ Schema markup

*****

Key Features of llms.txt

• Format: It is written in Markdown (a lightweight plain-text formatting language) rather than HTML or XML. This makes it very easy for AI models to parse without having to wade through heavy web code, scripts, ads, or navigation menus.
• Location: It is typically placed in the root directory of a website (e.g., yoursite.com/llms.txt) so that AI tools and crawlers can find it immediately.
• Structure: A standard file usually includes:
  1. An H1 heading with the name of the project or website.
  2. A blockquoted summary describing what the site is about.
  3. H2 sections containing curated links to the site's most important documentation, guides, or resources, often accompanied by brief descriptions.

Why Was It Created?

The standard was proposed to solve a few common challenges when AI models interact with the web:

• Reducing Clutter: Standard web pages are full of layout code, cookies, pop-ups, and sidebars that waste an AI's processing power and context window. llms.txt cuts straight to the signal.
• Guiding AI Tools: Instead of letting an AI crawler guess which pages are important, website owners can hand-pick and prioritize their best content.
• Complementing Existing Files: While robots.txt tells crawlers what they are allowed to access and a sitemap.xml lists every page available, llms.txt tells an LLM what is most important and how it is structured.

Often, websites that use this standard will also offer an llms-full.txt file, which consolidates all of their essential documentation into a single, comprehensive text file for deep AI ingestion.

____________________________________________

Is it associated with GEO?

Yes, absolutely. llms.txt is considered a direct technical implementation of GEO (Generative Engine Optimization)—often thought of as the AI-era equivalent of traditional Search Engine Optimization (SEO).

While traditional SEO focuses on helping websites rank higher in standard search engine results (like Google or Bing), GEO focuses on optimizing how a website's content is discovered, understood, cited, and recommended by AI-powered search engines and large language models (such as ChatGPT, Claude, Perplexity, and Google AI Overviews).

How llms.txt supports GEO:

• Direct AI Guidance: Instead of leaving AI crawlers to guess which pages matter, an llms.txt file acts as a curated map pointing directly to a site's most important, authoritative content.
• Token Efficiency: AI models have limited context windows and struggle to parse heavy HTML code, sidebars, ads, and scripts. By providing clean, lightweight Markdown links, llms.txt ensures that AI engines ingest pure content rather than structural noise.
• Accurate Attribution: By guiding models to canonical sources and structured summaries, website owners increase the likelihood that AI tools will cite them correctly when answering user prompts.

____________________________________________________

How to get the most out of llms.txt?

To get the most out of an llms.txt file, you need to structure it following the official community standard while optimizing it for token efficiency, clarity, and precise context.

1. Follow the Core Structural Rules

According to the specification (llmstxt.org), your file should be placed at the root of your domain (e.g., yoursites.com/llms.txt) and follow a strict Markdown hierarchy:

• H1 Heading (#): Use this for your exact brand, project, or website name. Keep it clear and singular.
• Blockquote (>): Write a concise, one-sentence elevator pitch or summary directly below the H1. This gives the AI its immediate baseline context of what your site offers.
• H2 Headings (##): Group your key content categories (e.g., ## Products, ## Documentation, ## Guides).
• Bulleted Links (- [Title](URL): Description): List your most important, high-value pages. Always include a brief, descriptive sentence explaining what the linked page contains.

2. Best Practices for Maximum Impact

• Keep it Concise (< 10KB): An llms.txt file is meant to be a quick, lightweight map, not a massive content dump. Prioritize your top 10 to 20 most authoritative links.
• Pair It with an llms-full.txt File: For deep context, create a companion file named llms-full.txt. This expanded version can aggregate your key text, guides, or documentation into a single, comprehensive Markdown document so that LLMs with larger context windows can ingest everything at once without needing to crawl multiple web pages.
• Use Neutral, Factual Language: Avoid heavy marketing jargon, hyperbolic claims, or metaphors. LLMs rely on functional definitions. Clear, precise, and neutral descriptions help models categorize your brand correctly and reduce hallucinations.
• Link Directly to Markdown Files (.md): If you host Markdown versions of your documentation or blog posts, link directly to those instead of heavy HTML pages. Markdown is significantly faster and cheaper (token-wise) for LLMs to process.
• Utilize the "Optional" Section: If you have low-priority pages (like changelogs, archives, or affiliate details), group them under a ## Optional heading. This signals to AI models with limited context windows that they can safely skip those links.
• Keep It Updated: Routinely review your llms.txt file to ensure the links aren't broken, the descriptions match your current offerings, and you are pointing to your most up-to-date resources.

________________________________________________

Google does not use llms.txt for its AI-driven search engines

Google has officially confirmed that it does not use llms.txt files for its search engines, including AI-driven features like AI Overviews and AI Mode.

Key Details from Google's Guidance

• No Ranking Impact: Google's official documentation states that llms.txt files have no effect, positive or negative, on search rankings.
• Standard Infrastructure: Google Search relies on its own robust crawling, indexing, and ranking systems rather than self-declared markdown files. Google Search Relations has compared the concept to outdated metadata tags that were easily manipulated.
• Not Required for AI Search: Google has emphasized that site owners do not need to create specialized machine-readable files or markdown files to appear in generative AI search experiences.

Note: While Google Search ignores llms.txt, some developer tools, specialized coding assistants, and other platforms may still utilize them. If your goal is specifically to optimize for Google Search or AI-driven search results, Google advises focusing on foundational SEO elements like crawlability, high-quality content, and clean technical structure.

How do I best write llms.txt to ensure that AI-powered search engines separate our informative text guides from the multiple, large directories of hand-selected links?

To successfully separate your primary, high-value text guides from large archival directories or secondary resource links within an llms.txt file, you need to lean into the official specification's native formatting rules.

AI-powered search engines and context-parsers parse these files hierarchically. You can control how an AI engine prioritizes your content by using structural layout, explicit labeling, and selective separation.

1. Leverage the Dedicated ## Optional Section

The official llms.txt standard (maintained at llmstxt.org) specifically includes provisions for a built-in filter mechanism:

• The Rule: Any link categories or pages placed under an ## Optional heading explicitly signal to large language models that these are lower-priority resources (such as large directories, changelogs, archives, or supplementary link lists).
• Why it works: When an AI tool operates under tight context constraints or looks strictly for core explanatory text, it is explicitly programmed to ingest your main H2 sections first and safely bypass or deprioritize the ## Optional section.

2. Structure Your Hierarchies by Value (Information Architecture)

Keep your primary text guides front-and-center while pushing mass directories further down the document tree.

• Top Tier (Core Guides): Place your informative text guides under prominent, descriptive H2 headers right after your summary blockquote (e.g., ## Core Guides or ## Documentation).
• Bottom Tier (Link Directories): Group your massive arrays of hand-selected links into separate, subsequent H2 headers further down the file (e.g., ## Resource Directories or ## Comprehensive Link Indices).

3. Use Precise In-Line Descriptions

LLMs rely heavily on the descriptive text attached to Markdown links to determine whether a destination page is worth fetching.

• For Guides: Use semantic, content-focused descriptions (e.g., - [Understanding Widgets](url): A step-by-step conceptual guide explaining how widgets work.).
• For Directories: Clearly label directory links as aggregations rather than standalone guides (e.g., - [All Widget Sub-Categories](url): A large directory index containing thousands of categorized link references.). This tells the AI model that the page is a navigational hub rather than a direct textual explanation.

Example Template Structure

Markdown

# Your Brand Name > Your concise, authoritative site summary goes here in a blockquote.

## Core Informative Guides - [Getting Started Guide](https://example.com/guides/start): A comprehensive textual walkthrough of core concepts. - [Architecture Overview](https://example.com/guides/architecture): Explains the framework design and functional principles.

## Comprehensive Link Directories - [Full Category Index](https://example.com/directory/all): A large, exhaustive directory of all community-curated links. - [Archive Directory](https://example.com/directory/archive): Historical list of past resources and releases.

## Optional - [Changelog](https://example.com/changelog): Detailed release logs and version history.

__________________________________________________

We are an online encylopedia: On our website we have topic pages, author pages and person profile pages. - How should we include that in our llms.txt?

To include topic pages, author pages, and person profile pages effectively in your llms.txt file, you should use hierarchical H2 sections with explicit descriptive labels.

Because an online encyclopedia contains thousands of individual profiles and articles, you should not list every single page (as that would bloat the file). Instead, highlight your core structural categories, top-tier anchor pages, and how AI engines should interpret these different page types.

Recommended Structure for an Encyclopedia llms.txt

Markdown

# Open Encyclopedia Name > An open-access digital reference encyclopedia providing structured overviews on global topics, biographical profiles, and verified author credentials.

## Core Content & Topic Pages - [Featured Topic Hubs](https://example.com/topics/): Curated index of top-level subject directories (Science, History, Technology). - [Key Encyclopedia Articles](https://example.com/topics/featured-samples): A representative list of high-authority topic guides demonstrating our standard editorial format.

## Person Profiles & Biographies - [Biographical Directory](https://example.com/profiles/): Structured data index of historical figures, public figures, and notable individuals. - [Profile Formatting Guide](https://example.com/help/profile-schema): Explains how vital statistics, timelines, and career histories are structured on individual profile pages.

## Author & Editorial Transparency - [Editorial Board & Authors](https://example.com/authors/): Directory of credentialed subject-matter experts, journalists, and academic reviewers who write and verify content. - [Editorial Standards](https://example.com/editorial-policy): Guidelines outlining our fact-checking, citation, and neutrality protocols.

## Optional - [All Topic Sub-Categories](https://example.com/topics/all-index): Massive exhaustive directory of every sub-topic page on the site. - [Author Archives](https://example.com/authors/archive): Historical list of inactive or past contributor profiles.

Key Strategy Tips for Encyclopedias

1. Explicitly Teach the AI the Difference Between Profiles: Language models often confuse "Author Pages" (the people writing the encyclopedia) with "Person Profile Pages" (the subjects of the encyclopedia). Use the inline descriptions to draw a hard line:
   • Person Profile: - [Profile Name](url): Historical encyclopedia biography detailing the life and achievements of [Name].
   • Author Profile: - [Author Name](url): Editorial staff credential page detailing the professional background of expert reviewer [Name].
2. Highlight Editorial Integrity: AI search engines prioritize content that demonstrates E-E-A-T (Experience, Expertise, Authoritativeness, and Trustworthiness). Linking directly to your Editorial Board, Fact-Checking Standards, and Author Pages tells the LLM that your encyclopedia is a reliable source, heavily boosting your chances of being cited as an accurate reference.
3. Avoid Token Bloat via Index Pages: If your encyclopedia has 500,000 person profiles, do not link them individually. Instead, link to directory hubs or search index landing pages that aggregate those profiles logically. Use the ## Optional section for massive, deep-level flat maps or exhaustive sub-category indexes so that lightweight AI agents know they can skip the raw database weight unless specifically searching for a deep lookup.

________________________________________________________

What are llms.txt sections and can they be used instead?

In the context of the official llms.txt standard (llmstxt.org), "sections" refer to the specific structural building blocks that make up the file itself.

According to the specification, a valid llms.txt file is built using these exact sections in order:

1. H1 Title (#): The mandatory name of the project or website.
2. Summary Blockquote (>): A brief overview summarizing what the site or resource contains.
3. Detailed Information (Optional text/lists): General paragraphs or descriptive notes providing context on how to interpret the files.
4. H2 Link Lists (##): Categorized groups of Markdown links pointing to key web pages or resources.
5. The Optional Section (## Optional): A designated section for secondary, low-priority links that AI models can skip if they are short on context space.

Can these sections be used instead of anything?

llms.txt sections cannot replace traditional website navigation or core infrastructure files, but they serve specific roles:

• They do not replace robots.txt: A robots.txt file controls crawler permissions (telling bots what they are allowed to access). An llms.txt file is purely a content map and guide; it has no control over security or access restrictions.
• They do not replace an XML Sitemap: An XML sitemap lists every URL on a site for search engine indexing. An llms.txt file is a hand-curated, lightweight selection meant to guide AI models to the most important high-signal resources without wasting token space.
• They can replace clunky navigation discovery: For AI agents and LLM-powered tools, the sections inside an llms.txt file act as an optimized replacement for scraping complex HTML menus, sidebars, and cluttered webpage layouts.

____________________________________________________

What are llms.txt tokens?

In the context of an llms.txt file, tokens refer to the fundamental units of text that large language models (LLMs) read, process, and count when analyzing data.

When people talk about tokens and llms.txt, they are usually referring to token efficiency—how many tokens a file takes up and how those tokens are spent.

Key Concepts About Tokens in llms.txt

• What a token is: An AI model does not read text letter-by-letter or word-by-word. Instead, it breaks text down into chunks called tokens (roughly 4 characters or 0.75 words on average in English). Every word, punctuation mark, and space counts toward a model's processing capacity.
• The Token Cost of Web Pages: Standard web pages are packed with heavy HTML code, navigation bars, sidebars, scripts, and styling tags. When an AI crawls a normal website page, it has to burn thousands of tokens just parsing formatting noise before it even reaches the actual content.
• Saving Tokens with llms.txt: An llms.txt file is written in plain Markdown. Because it strips out all the invisible HTML clutter, it has an extremely high "information density." Every single token the AI reads carries direct meaning, acting like a lightweight map.

• Token Budgets and Size Limits: * A standard llms.txt routing file is kept very small (often under a few thousand tokens, less than 3 pages) so an AI can read it instantly without wasting resources.

  • For deeper context, websites often provide an llms-full.txt companion file, which bundles documentation together. These full files can range from tens of thousands to hundreds of thousands of tokens, depending on how large the context window of the target AI model is.

  ________________________________________________________

Tell me more about llms-full.txt - How is it best structured?

What is llms-full.txt?

While an standard llms.txt file acts as a lightweight table of contents or directory map, llms-full.txt is a companion file that compiles a large portion—or all—of a website's core text content into a single, massive Markdown document.

It is typically hosted at yoursite.com/llms-full.txt.

Why is it used?

• Instant Context Loading: Instead of forcing an AI crawler to hop across dozens or hundreds of different links to fetch information, an AI tool or user can ingest the entire file in one single action.
• Overcoming Navigation Barriers: Some advanced AI agents or local context tools prefer having a raw, linear text dump rather than navigating web hierarchies, eliminating the risk of broken links or restricted scraping blocks.
• Large Context Windows: Modern AI models feature massive context windows (handling hundreds of thousands or even millions of tokens), making it possible for them to read an entire llms-full.txt file at once.

How is llms-full.txt Best Structured?

Because a llms-full.txt file can grow extremely large, structuring it logically is critical so the AI model doesn't lose track of context. It should be organized hierarchically using standard Markdown elements:

1. Global Header & Overview

Start with a clear title and summary just like your regular llms.txt so the AI immediately knows what dataset it is reading.

Markdown

# Your Website Name - Full Documentation Archive > Comprehensive collection of all core guides, tutorials, and reference materials for [Your Project].

2. Clear Section Dividers (## and ###)

Group your consolidated articles or documents into clear subject categories using heading tags. Every separate article or guide should have its own heading so the model can isolate specific topics.

Markdown

## Section 1: Getting Started ### Introduction to the Platform [Full text of the introduction article goes here...] ### Quickstart Tutorial [Full text of the tutorial goes here...] ## Section 2: Core Concepts ### Architecture Overview [Full text of the architecture guide goes here...]

3. Best Practices for Writing and Managing It

• Use Horizontal Rules (---): Place horizontal dividing lines between long, distinct articles or chapters to visually and textually signal to the model where one document ends and the next begins.
• Strip Out Layout Noise: Ensure code blocks, tables, and paragraphs are formatted cleanly in plain Markdown. Remove repetitive elements like footers, headers, or navigation sidebars that appear on every page to save valuable token space.
• Mind the File Size: While modern models have large windows, files that exceed practical limits can cause performance degradation or high latency. If your site is massive, consider splitting your full text files into category-specific files (e.g., llms-full-api.txt or llms-full-guides.txt).

_______________________________________________________

I guess as an online encyclopedia we do not need llms-full.txt

That is a very common perspective for large encyclopedias, but whether you need an llms-full.txt file depends entirely on your goals for AI visibility and how your content is structured.

Why Many Large Encyclopedias Skip It

• Massive Scale: Because an encyclopedia contains hundreds of thousands—or even millions—of distinct articles, compiling everything into a single llms-full.txt file is practically impossible. The file size would span billions of tokens, far exceeding even the largest context windows available to AI models today.
• Redundancy: AI search engines (like Perplexity or ChatGPT) and large language model providers already scrape encyclopedia data regularly to train their core weights or pull real-time web results.

When an Encyclopedia Might Use a Modified Approach

While a single universal llms-full.txt doesn't work for a massive database, many large reference sites adapt the concept for specific subsections:

• Categorical Full Files: Instead of one massive file, they might offer segmented files for distinct, high-value sections (e.g., llms-full-science.txt or llms-full-history-essentials.txt).
• Selected Highlights: They use the standard llms.txt routing file to point AI models directly to their best hub pages, featured indices, and documentation, letting the AI crawl individual pages as needed.

If your encyclopedia relies on users browsing through interconnected topics, a well-structured llms.txt file pointing to your primary category hubs is usually more than enough.

Draft: Google Gemini, Date: 28.07.26

Work in progress. Suggestions and collaboration welcome