LLMs.txt Best Practices: How to Structure and Maintain One
The llms.txt best practices that matter come down to one idea: a good llms.txt is a curated, machine-readable index of your most important content, not a dump of every URL on your domain. Get the structure right, keep the descriptions honest, and update the file every time you publish. Get any of those wrong and the file becomes noise an AI system learns to skip.
This guide covers the practices, not the concept itself. If you need the full explanation of what llms.txt is and why it exists, start with our LLMs.txt guide. Here, we're focused on how to build one that actually holds up: structure, content selection, maintenance, and the mistakes that quietly make a file useless.
What a good llms.txt actually does
An llms.txt file sits at your domain root (yoursite.com/llms.txt) and gives AI systems a fast, structured overview of your site: what it is, what it covers, and which pages are worth reading. It's markdown, so a model parses it natively without stripping HTML, navigation chrome, or ads.
The job of the file is narrow. It's not a copy of your sitemap, and it's not a place for marketing copy. It's a shortlist. Think of the difference between a table of contents and a full-text search index. Your sitemap is the index. Your llms.txt is the table of contents, curated by you, pointing at the pages you'd want an AI system to read first.
That framing drives every practice below.
Structure: the core of llms.txt best practices
The llms.txt specification (proposed at llmstxt.org) defines a consistent, minimal shape. Following it matters more than any individual field, because consistency is what makes the file machine-parseable at scale.
Start with an H1 of your site name. One line, the name of the site or company. Not a slogan, not a keyword-stuffed title.
Follow with a blockquote summary. A short paragraph, in blockquote markdown (>), describing what the site is and what it covers. This is the equivalent of a meta description, but for a model instead of a search snippet. Keep it factual: what you do, who it's for, what's on the site.
Group links into sections with H2 headings. Common sections are things like "Blog," "Docs," "Product," or "Guides," depending on what your site actually has. Each H2 becomes a labeled bucket a model can scan.
List links with one-line descriptions. Under each H2, a markdown list of links, each with a short, factual description of what that page covers. Not a headline. A description. "How to configure hreflang tags for a multilingual blog" tells a model more than "The Ultimate Guide to Multilingual SEO."
Put your most important content first. Within each section, order matters. Cornerstone and pillar content, the pages you'd most want cited by ChatGPT and other assistants, should lead. Models parsing large files may truncate or weight earlier content more heavily, and even when they don't, order signals priority to anyone (human or machine) working through the file.
Keep the markdown clean. No embedded HTML, no custom syntax, no tables where a list would do. Standard markdown, headings, links, and short blockquotes. The entire value proposition of llms.txt is that it's easier to parse than scraped HTML. Undermining that with messy formatting defeats the purpose.
Here's the shape in practice:
## Superblog
> A fully-managed blogging platform for companies serious
> about organic growth. 90+ Lighthouse scores, automatic
> SEO, and zero maintenance.
Blog
- [Blog SEO Guide](https://superblog.ai/blog/blog-for-seo): How to rank a company blog in 2026
- [WordPress Alternatives](https://superblog.ai/blog/wordpress-alternatives): Platforms compared for growth teams
Docs
- [Getting Started](https://superblog.ai/docs/getting-started): Launch a blog on your domain in minutes
Four sections, clear labels, factual one-liners. That's the whole pattern.
Content practices: curate, don't dump
The single most common mistake is treating llms.txt like a second sitemap and listing every URL the site has ever published. That's backward. A well-curated llms.txt with 30 to 50 strong pages is more useful to an AI system, and to you, than one listing 2,000 thin ones.
Curate for quality, not completeness. Include cornerstone guides, pillar pages, your best evergreen content, and pages that answer questions people actually ask. Skip thin posts, outdated announcements, and pages that exist only for internal navigation.
Keep descriptions factual, not promotional. "The complete guide to X" tells a model nothing it can use. "Explains how to configure X for Y use case" tells it what the page contains and when to reach for it. Descriptions are metadata, not ad copy. Save the persuasion for the page itself.
Include your pillar and cornerstone pages explicitly. A definitive guide on a topic your business is known for belongs near the top of its section, since these are the pages you most want an AI system citing.
Skip listing thin or duplicate pages. Tag archives, near-duplicate variations, and low-value utility pages add noise without adding coverage. If a page wouldn't earn a spot in a hand-picked "start here" list, it doesn't earn a spot in llms.txt either.
No keyword stuffing. Some early adopters treat llms.txt as a new place to cram target keywords into descriptions. It reads as spam to a model the same way it reads as spam to a person, and it works against the file's actual purpose, which is clarity.
Maintenance: a stale llms.txt is worse than none
An llms.txt file is only as good as its last update. If it points to pages that no longer exist, omits everything published in the last six months, or still describes a product you've since repositioned, it actively misleads whatever reads it. A 404 in a citation is worse for trust than not being cited at all.
Regenerate on every publish, not on a schedule. Weekly or monthly regeneration sounds reasonable until you realize an AI system might read your llms.txt the day after you publish your best piece of content this quarter, and the file still doesn't mention it. Treat regeneration as part of your publish flow, not a maintenance chore you get to later.
Automate it if you can. Manual llms.txt upkeep works for a five-page site. It breaks down for anything actively publishing. If your platform doesn't generate the file automatically, build the regeneration into your deploy pipeline so it's not a task a human has to remember.
Audit periodically for dead links and drift. Even automated files drift when categories change, posts get unpublished, or a site restructures. A quarterly pass to confirm every linked URL resolves and every description still matches its page keeps the file trustworthy.
Superblog handles this by generating and updating llms.txt automatically on every deploy, toggled on in Settings > SEO. There's no separate step to remember and no stale file sitting around from three redesigns ago.
What not to do
A short list of the mistakes that undo everything above:
- Don't list every URL on the domain. That's what sitemap.xml is for. llms.txt is a curated shortlist, not a mirror of your site structure.
- Don't write descriptions like ad copy. "Best-in-class," "revolutionary," and "game-changing" describe nothing a model can use to decide whether a page answers a question.
- Don't include thin or low-value pages just to pad the file. A shorter, sharper llms.txt beats a longer, weaker one.
- Don't let it go stale. An outdated llms.txt is a worse signal than having none, because it actively points to the wrong things.
- Don't use non-standard markdown or embed HTML. It breaks the one advantage llms.txt has over scraping your pages directly: clean, predictable parsing.
Honest expectations: what llms.txt actually is right now
llms.txt is a proposed convention, not an official web standard ratified by a governing body, and adoption is real but not universal. Some companies (Anthropic, Mintlify, and Zapier among them) publish one. Some AI crawlers are known to request the file when it exists. But there's no public confirmation that every major AI provider parses llms.txt for every query, and treating it as a guaranteed citation mechanism overstates what it is today.
That's not a reason to skip it. The file costs nothing to maintain properly, and if AI crawlers or agents do request it, you want a clean, current one waiting rather than nothing. It's the same bet robots.txt and sitemap.xml represented in their early days: cheap to implement, asymmetric upside if the standard sticks. Treat it as good infrastructure hygiene with a plausible payoff, not a guaranteed traffic channel.
It's worth being precise about which crawler does what, because "AI crawlers" isn't one thing. GPTBot is OpenAI's model training crawler and has nothing to do with whether ChatGPT cites you in a live search answer. OAI-SearchBot is the separate crawler behind ChatGPT's search citations; blocking GPTBot does not block OAI-SearchBot. Google-Extended governs only Google's AI training data collection and has no bearing on AI Overviews, which draw from standard Googlebot crawling and ranking. Anthropic runs ClaudeBot for training, Claude-User for on-demand fetches, and Claude-SearchBot for Claude's web search. PerplexityBot powers Perplexity's retrieval. None of these read llms.txt as a substitute for crawl access: if robots.txt blocks the crawler, a pristine llms.txt won't change that. The file is a citation aid layered on a site that's already crawlable, not a workaround for one that isn't. For the full breakdown of what each crawler does, see our guide to AI SEO.
llms.txt vs robots.txt: two different jobs
The two get conflated because both are plain-text files at your domain root, but they solve different problems. robots.txt is an access control file. It tells crawlers what they're allowed to fetch. llms.txt is an inclusion guide. It tells an AI system, once it has access, what's worth reading and in what order.
You need both, and they don't substitute for each other. A permissive robots.txt with no llms.txt means crawlers can reach everything but have no curated starting point. A perfect llms.txt behind a robots.txt block that disallows the relevant crawler means nothing gets read regardless of how well the file is structured. Check both. For the full breakdown of which crawlers to allow and how crawling feeds into indexing and ranking, see our guide to SEO crawling.
How to check yours
Verifying your own llms.txt takes about two minutes:
- Fetch it directly. Open
yourdomain.com/llms.txt in a browser. Confirm it loads as plain markdown, not a 404 or an HTML error page. - Check the structure. H1 site name, blockquote summary, H2 sections, linked lists with descriptions. If any of those pieces are missing, the file is incomplete relative to the specification.
- Spot-check the links. Click through a handful. Confirm they resolve, and confirm the one-line description actually matches what's on the page.
- Compare against a reference file. Our free LLMs.txt Generator builds one from your sitemap in about a minute. It's a structural reference, not a validator: it generates a fresh file to compare your existing one against, or a starting point if you don't have one yet.
- Recheck after any redesign. URL changes, content pruning, and category restructuring all break links inside an existing llms.txt if it isn't regenerated to match.
If you're publishing on Superblog, this whole checklist is mostly moot. The file regenerates automatically on every deploy, structured according to the practices above, with no manual step to forget.
FAQ
Does llms.txt use a specific file format?
Yes. Markdown, following a loose but consistent shape: H1 site name, a blockquote summary, H2 section headings, and linked lists with one-line descriptions under each. Plain markdown parses more reliably than embedded HTML or custom syntax.
How often should I update my llms.txt file?
Every time you publish or materially change content, ideally as part of your deploy process rather than a manual task on a schedule. A file that's a month behind your actual content is a file quietly pointing AI systems at outdated information.
Should I list every page on my site?
No. List your best, most important, most evergreen pages. A shorter, curated file that leads with your cornerstone content is more useful than a complete mirror of your sitemap, both to a model reading it and to anyone auditing it later.
Is llms.txt required for AI search visibility?
No. It's a proposed convention with growing but not universal adoption. It doesn't replace crawl access, clean HTML, or structured content. It's a low-cost addition layered on top of a site that's already crawlable, not a substitute for the fundamentals.
What's the difference between llms.txt and robots.txt?
robots.txt controls access: what crawlers are allowed to fetch. llms.txt is an inclusion guide: what's worth reading once a crawler has access, and in what priority order. They serve different jobs and neither replaces the other.
Can I check llms.txt for errors automatically?
There's no dedicated validator built into most tooling today, including ours. Our LLMs.txt Generator generates a reference file from your sitemap so you can compare it against what you currently have, but manual review, checking that links resolve and descriptions are accurate, is still the reliable method.
The maintenance-free version
Every practice above is either a one-time structural decision or a recurring maintenance task: regenerate on publish, prune stale links, keep descriptions honest. On most platforms, that upkeep falls on whoever remembers to do it, which in practice means it happens less often than it should.
Superblog generates llms.txt automatically, structured to these practices, and refreshes it on every deploy as part of the normal publish flow. It's a toggle in Settings > SEO, not a task on anyone's list. Combined with automatic JSON-LD schemas, XML sitemaps, and IndexNow submission, it's one piece of the SEO layer that runs without anyone having to remember it exists. Start a 7-day trial, no card required, and see what your own llms.txt looks like on day one.
Want an SEO-focused and blazing fast blog?
Superblog let's you focus on writing content instead of optimizations.
superblog
Superblog is a blazing fast blogging platform for beautiful reading and writing experiences. Superblog takes care of SEO audits and site optimizations automatically.
ESC
Type to search...
Loading...
No results found
↑↓ Navigate
↵ Open
esc Close
