A robots.txt file for a blog tells crawlers which paths they are allowed to fetch. It lives at one fixed location, yourdomain.com/robots.txt, and every well-behaved bot reads it before crawling. The one rule most blogs need is short: allow everything, point to your sitemap, and block only your admin and internal-search URLs. If you do nothing else, that is enough. The mistakes that hurt blogs come from doing too much, not too little.
This guide covers what robots.txt controls (and what it does not), a copy-paste default you can adapt, the 2026 decision of which AI crawlers to allow or block, the errors that quietly deindex blogs, and how to test your file before it does damage.
What Robots.txt Is and Is Not
Robots.txt is a crawl-control file, not an indexing-control file. That distinction is the single most misunderstood thing about it, and getting it wrong is how blogs disappear from search.
Here is what the file actually does: when a crawler arrives at your domain, it fetches /robots.txt first and reads the rules. A Disallow line tells the bot not to request that path. That is the whole mechanism. It governs fetching, nothing more.
Here is what it does not do:
It does not remove pages from search results. A page you Disallow in robots.txt can still be indexed. If another site links to that URL, Google can add the URL to its index based on the link alone, even though it never fetched the page. You have seen the result: a search listing with the URL and no description, showing "No information is available for this page." The page is blocked from crawling but not from indexing.
Noindex does not belong in robots.txt. To keep a page out of the index, you use a noindex directive, which lives in the page's HTML <meta> tag or in an X-Robots-Tag HTTP header, not in robots.txt. Google dropped support for the unofficial Noindex: robots.txt line in 2019, so any such line is ignored today.
The two directives conflict when combined. If you both Disallow a page and add a noindex tag to it, the crawler is blocked from fetching the page, so it never sees the noindex tag, so the page can stay indexed anyway. To reliably remove a page from search, you must allow crawling and serve noindex. Let the bot in so it can read the instruction to leave.
The rule of thumb: use robots.txt to stop bots wasting time on pages that have no business being crawled. Use noindex to stop pages appearing in search. They solve different problems.
A Sensible Default Robots.txt for a Blog
Most blogs are overthinking this file. You want search engines and AI answer engines to read every post, so the default posture is open. You only fence off the handful of paths that generate thin or duplicate URLs, or that expose an admin surface.
Here is a default you can copy and adapt. The comments explain each block.
# robots.txt for https://yourdomain.com
# Allow all crawlers to read the blog by default
User-agent: *
Allow: /
# Block the admin and login surface (no SEO value, keep bots out)
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /login/
# Block internal search-result pages (thin, near-infinite duplicate URLs)
# /search/ with trailing slash; bare /search would also block /search-tips...
Disallow: /search/
Disallow: /*?s=
Disallow: /*?q=
# Point every crawler to your sitemap
Sitemap: https://yourdomain.com/sitemap.xml
A few notes on the choices:
Allow all by default. Your posts, category pages, and author pages are the entire point of the blog. They should never sit behind a Disallow. The User-agent: * block with Allow: / sets an open baseline that every bot inherits.
Block internal search results. A site-search URL like /search?q=anything can generate a near-infinite set of low-value pages. Blocking the pattern keeps crawlers focused on real content. Adjust the exact parameter (?s=, ?q=, ?query=) to match your platform.
The Sitemap line is not optional. It tells every crawler where your full URL list lives, which speeds up discovery of new posts. Point it at your real sitemap path.
Be careful with tag and archive pages. A common instinct is to Disallow every tag page. Resist it. Tag and category pages often carry internal-linking value and can rank in their own right. If some tag pages are genuinely thin, the correct tool is a noindex tag on those pages, not a robots.txt block, so Google can still crawl through them to your posts. Only block archive patterns in robots.txt if they produce true duplicates, such as endless date-based pagination, and even then prefer noindex plus canonical tags.
That is the entire file most blogs need. If yours is longer than this, check that every extra line earns its place.
The AI-Crawler Decision for 2026
The new question robots.txt has to answer is which AI crawlers to allow. This is where blog owners now spend their deliberation, and where the wrong reflex costs you visibility. The key is that AI crawlers are not one category. They do three different jobs, and blocking the wrong class removes you from AI answers entirely.
There are three roles, and the user-agent tokens are public and current as of 2026:
*Google-Extended is a control token, not a separate crawler; Googlebot does the fetching.
The trade-off is now clear. If you object to your writing being used to train models, block the training class. That is a legitimate stance and it costs you nothing in live traffic. But if you block the search-index or real-time-fetch classes, you are removing your blog from AI answers and from agentic browsing, which is a fast-growing discovery channel. For a business blog that wants to be cited when someone asks ChatGPT or Perplexity a question in your space, blocking those crawlers works against you.
Two facts worth stating plainly. First, the training class is not uniform. Blocking GPTBot and ClaudeBot keeps your posts out of future training runs and removes you from no live product. Google-Extended is the partial exception: it governs Gemini training and grounding, so blocking it can also keep your content out of Gemini's grounded answers. It has no effect on Google Search ranking or AI Overviews. Second, the user fetchers differ by vendor. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user initiated the action, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic is the exception: it states its bots, including Claude-User, honor robots.txt, and that disallowing Claude-User stops it retrieving your content for user queries. Plan around what each token actually respects, not what you wish it did.
Here is a training-opt-out block you can add if that is your choice. It blocks training while leaving every AI search and citation crawler allowed:
# Opt out of AI model training, stay visible in AI search and answers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
On Superblog this is a toggle: flip GPTBot off in Settings and the robots.txt is rewritten for you.
Note what is not in that list. OAI-SearchBot, Claude-SearchBot, and PerplexityBot stay allowed, so your posts remain eligible to be cited in AI search. That is the deliberate position for most blogs chasing organic and AI-driven discovery. If you want the fuller picture of who these bots are, our AI crawlers guide breaks down each one, and agentic browsing covers the real-time-fetch channel in depth.
Common Blog Robots.txt Mistakes
Almost every serious robots.txt failure falls into one of these buckets. Each one has deindexed real blogs.
Blocking CSS and JavaScript. Google renders your pages to understand them, and rendering needs your stylesheets and scripts. An old habit was to Disallow: /wp-includes/ or block asset directories to save crawl budget. Do not. If Google cannot fetch your CSS and JS, it sees a broken page and judges it accordingly. Leave assets fully crawlable.
Shipping the staging block to production. Staging sites are usually locked down with a site-wide block:
User-agent: *
Disallow: /
Those two lines tell every crawler to fetch nothing. When a site launches and that file gets copied to production unchanged, the entire blog becomes uncrawlable overnight. This is the most expensive robots.txt mistake there is, and it is disturbingly common. The first thing to check on any launch is that the production robots.txt does not disallow everything.
Prefix-matching accidents. Every Disallow path is prefix-matched, even without a *, so it blocks any URL that starts with those characters. Disallow: /blog blocks not just /blog but everything starting with those characters, including /blog-post-one. If you meant the directory, write Disallow: /blog/ with the trailing slash, or you will wipe out posts you meant to keep. Test every pattern before shipping it.
Robots.txt on the wrong host. Robots.txt is scoped to its exact host, including the subdomain. A file at yourdomain.com/robots.txt does not govern blog.yourdomain.com. If your blog runs on a subdomain, the robots.txt that matters is blog.yourdomain.com/robots.txt, and it must be served from that host. Blogs hosted on a subdirectory like yourdomain.com/blog are governed by the root domain's file instead. Know which host serves the crawlers your blog needs, and edit the right file.
How to Test Your Robots.txt
Never ship a robots.txt change to a revenue-driving blog without testing it. Two checks catch nearly everything.
Google Search Console's robots.txt report. In Search Console, the robots.txt report shows the file Google last fetched, flags syntax errors, and lets you see how Google reads it. Use it to confirm Google is fetching the file you think it is, and that no critical path is blocked. When you push a change, you can request a refetch so Google picks it up sooner.
A live validator. Before you deploy, paste your file into our robots.txt validator to check the syntax and confirm that a given URL is allowed or blocked for a specific user-agent. It catches the trailing-slash wildcard traps and the accidental site-wide Disallow before they reach production. Test the URLs that matter most, your posts and your homepage, against Googlebot and against the AI crawlers you care about.
After any change, spot-check a real post URL in Search Console's URL Inspection tool to confirm it is still crawlable and indexed. If you want a deeper look at how blocking interacts with what actually gets indexed, see our guides on blog indexing and SEO crawling.
How Superblog Handles Robots.txt
On Superblog, robots.txt is generated for you with sane defaults, so there is no plugin to install and no file to hand-edit. Your posts, categories, and sitemap are exposed to search and AI crawlers by default. The sitemap line points at the sitemap Superblog builds and updates automatically on every deploy.
For the AI-training decision, Superblog gives you a GPTBot toggle so you can opt out of OpenAI model training without touching a raw file or leaving your search visibility behind. The defaults keep the search and citation crawlers allowed, which is the right posture for a blog that wants to be discovered in AI answers as well as in Google. Superblog also generates an llms.txt file, covered in our guide to llms.txt and AI search. No maintenance, no plugin, no risk of shipping a staging block to production.
If you want a blog where the crawl-control layer is correct by default instead of one wrong line away from disaster, start a 7-day free trial (no credit card required) and publish on infrastructure that gets robots.txt right for you.
FAQ
Do blogs need a robots.txt file?
Technically no, a blog works without one, and the absence of a robots.txt file means crawlers assume everything is allowed. In practice you want one, because it lets you point crawlers to your sitemap and block admin and internal-search URLs from wasting crawl attention. A minimal robots.txt with an Allow: / baseline and a Sitemap: line is worth having on every blog.
Should I block AI crawlers from my blog?
Only the ones whose job you object to. Blocking training crawlers like GPTBot and ClaudeBot keeps your content out of model training while costing you no live traffic. Google-Extended is the exception in that class: blocking it also opts you out of Gemini grounding, though it never touches Google Search ranking or AI Overviews. Blocking search-index crawlers like OAI-SearchBot, Claude-SearchBot, and PerplexityBot removes your blog from AI answers and citations, which most business blogs do not want. Decide per class, not all at once.
Does robots.txt stop a page from being indexed?
No. Robots.txt stops crawling, not indexing. A blocked page can still appear in search results if other sites link to it, usually with no description. To keep a page out of search, allow crawling and add a noindex meta tag so the crawler can read the instruction to leave.
Where should the robots.txt file be located?
At the root of the exact host it governs, always at /robots.txt. For a subdomain blog it must be at blog.yourdomain.com/robots.txt. For a subdirectory blog at yourdomain.com/blog, the root domain's yourdomain.com/robots.txt applies. A robots.txt on the wrong host controls nothing on your blog.
Can I block SEO tools from crawling my blog?
Yes. You can add Disallow rules for third-party SEO crawler user-agents such as AhrefsBot or SemrushBot if you do not want them mapping your content. This is optional and mostly a competitive-privacy choice. It has no effect on your Google rankings, since Googlebot is a separate user-agent.
Will blocking Google-Extended hurt my search ranking?
No. Google-Extended controls only whether your content is used for Gemini training and grounding. Google states it does not affect inclusion in Google Search and is not a ranking signal, and it does not affect AI Overviews. Blocking it opts you out of Gemini training and grounding; it does not touch Search rankings or AI Overviews.
How do I test my robots.txt before publishing it?
Use two checks. Google Search Console's robots.txt report shows the file Google fetched and flags errors, and its URL Inspection tool confirms a given post is crawlable. Before deploying, run your file through a robots.txt validator to confirm each URL is allowed or blocked for the user-agents you care about, which catches wildcard traps and accidental site-wide blocks.