# Robots.txt for Blogs: What to Block, What to Allow (2026)
Author: Sai Krishna
Author URL: https://superblog.ai/blog/author/sai-krishna/
Published: 2026-08-06
Meta Title: Robots.txt for Blogs: What to Block, What to Allow (2026)
Meta Description: A robots.txt file tells crawlers where they can go on your blog. What to block, what to allow, the 2026 AI-crawler decision, and a copy-paste example.
URL: https://superblog.ai/blog/robots-txt-for-blogs/

![Robots.txt for Blogs](https://prod.superblogcdn.com/site_cuid_ckox4in4f002nl8lhcib41g2u/images/robots-txt-for-blogs-1785016876790-compressed.png)

A robots.txt file for a blog tells crawlers which paths they are allowed to fetch. It lives at one fixed location, `yourdomain.com/robots.txt`, and every well-behaved bot reads it before crawling. The one rule most blogs need is short: allow everything, point to your sitemap, and block only your admin and internal-search URLs. If you do nothing else, that is enough. The mistakes that hurt blogs come from doing too much, not too little.

This guide covers what robots.txt controls (and what it does not), a copy-paste default you can adapt, the 2026 decision of which AI crawlers to allow or block, the errors that quietly deindex blogs, and how to test your file before it does damage.

## What Robots.txt Is and Is Not

Robots.txt is a crawl-control file, not an indexing-control file. That distinction is the single most misunderstood thing about it, and getting it wrong is how blogs disappear from search.

Here is what the file actually does: when a crawler arrives at your domain, it fetches `/robots.txt` first and reads the rules. A `Disallow` line tells the bot not to request that path. That is the whole mechanism. It governs fetching, nothing more.

Here is what it does not do:

**It does not remove pages from search results.** A page you `Disallow` in robots.txt can still be indexed. If another site links to that URL, Google can add the URL to its index based on the link alone, even though it never fetched the page. You have seen the result: a search listing with the URL and no description, showing "No information is available for this page." The page is blocked from crawling but not from indexing.

**Noindex does not belong in robots.txt.** To keep a page out of the index, you use a `noindex` directive, which lives in the page's HTML <meta> tag or in an `X-Robots-Tag` HTTP header, not in robots.txt. Google dropped support for the unofficial `Noindex:` robots.txt line in 2019, so any such line is ignored today.

**The two directives conflict when combined.** If you both `Disallow` a page and add a `noindex` tag to it, the crawler is blocked from fetching the page, so it never sees the `noindex` tag, so the page can stay indexed anyway. To reliably remove a page from search, you must allow crawling and serve `noindex`. Let the bot in so it can read the instruction to leave.

The rule of thumb: use robots.txt to stop bots wasting time on pages that have no business being crawled. Use `noindex` to stop pages appearing in search. They solve different problems.

## A Sensible Default Robots.txt for a Blog

Most blogs are overthinking this file. You want search engines and AI answer engines to read every post, so the default posture is open. You only fence off the handful of paths that generate thin or duplicate URLs, or that expose an admin surface.

Here is a default you can copy and adapt. The comments explain each block.

```
# robots.txt for https://yourdomain.com

# Allow all crawlers to read the blog by default
User-agent: *
Allow: /

# Block the admin and login surface (no SEO value, keep bots out)
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /login/

# Block internal search-result pages (thin, near-infinite duplicate URLs)
# /search/ with trailing slash; bare /search would also block /search-tips...
Disallow: /search/
Disallow: /*?s=
Disallow: /*?q=

# Point every crawler to your sitemap
Sitemap: https://yourdomain.com/sitemap.xml

```

A few notes on the choices:

**Allow all by default.** Your posts, category pages, and author pages are the entire point of the blog. They should never sit behind a `Disallow`. The `User-agent: *` block with `Allow: /` sets an open baseline that every bot inherits.

**Block internal search results.** A site-search URL like `/search?q=anything` can generate a near-infinite set of low-value pages. Blocking the pattern keeps crawlers focused on real content. Adjust the exact parameter ( `?s=`, `?q=`, `?query=`) to match your platform.

**The Sitemap line is not optional.** It tells every crawler where your full URL list lives, which speeds up discovery of new posts. Point it at your real sitemap path.

**Be careful with tag and archive pages.** A common instinct is to `Disallow` every tag page. Resist it. Tag and category pages often carry internal-linking value and can rank in their own right. If some tag pages are genuinely thin, the correct tool is a `noindex` tag on those pages, not a robots.txt block, so Google can still crawl through them to your posts. Only block archive patterns in robots.txt if they produce true duplicates, such as endless date-based pagination, and even then prefer `noindex` plus canonical tags.

That is the entire file most blogs need. If yours is longer than this, check that every extra line earns its place.

## The AI-Crawler Decision for 2026

The new question robots.txt has to answer is which AI crawlers to allow. This is where blog owners now spend their deliberation, and where the wrong reflex costs you visibility. The key is that AI crawlers are not one category. They do three different jobs, and blocking the wrong class removes you from AI answers entirely.

There are three roles, and the user-agent tokens are public and current as of 2026:

Role

User-agent tokens

What blocking costs you

Training

`GPTBot`, `ClaudeBot`, `Google-Extended`\*

Keeps your posts out of future model training runs. Removes you from no live product ( `Google-Extended`\\* is the exception: it also governs Gemini grounding, so blocking it can keep you out of Gemini's grounded answers).

Search indexing

`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`

Removes you from those products' AI answers and citations.

Real-time user fetch

`ChatGPT-User`, `Claude-User`, `Perplexity-User`

Stops live retrieval where honored (Anthropic; OpenAI and Perplexity may fetch regardless, see below)

\* `Google-Extended` is a control token, not a separate crawler; Googlebot does the fetching.

The trade-off is now clear. If you object to your writing being used to train models, block the training class. That is a legitimate stance and it costs you nothing in live traffic. But if you block the search-index or real-time-fetch classes, you are removing your blog from AI answers and from agentic browsing, which is a fast-growing discovery channel. For a business blog that wants to be cited when someone asks ChatGPT or Perplexity a question in your space, blocking those crawlers works against you.

Two facts worth stating plainly. First, the training class is not uniform. Blocking `GPTBot` and `ClaudeBot` keeps your posts out of future training runs and removes you from no live product. `Google-Extended` is the partial exception: it governs Gemini training and grounding, so blocking it can also keep your content out of Gemini's grounded answers. It has no effect on Google Search ranking or AI Overviews. Second, the user fetchers differ by vendor. OpenAI says robots.txt rules may not apply to `ChatGPT-User` because a user initiated the action, and Perplexity says `Perplexity-User` generally ignores robots.txt for the same reason. Anthropic is the exception: it states its bots, including `Claude-User`, honor robots.txt, and that disallowing `Claude-User` stops it retrieving your content for user queries. Plan around what each token actually respects, not what you wish it did.

Here is a training-opt-out block you can add if that is your choice. It blocks training while leaving every AI search and citation crawler allowed:

```
# Opt out of AI model training, stay visible in AI search and answers

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

```

On Superblog this is a toggle: flip GPTBot off in Settings and the robots.txt is rewritten for you.

Note what is not in that list. `OAI-SearchBot`, `Claude-SearchBot`, and `PerplexityBot` stay allowed, so your posts remain eligible to be cited in AI search. That is the deliberate position for most blogs chasing organic and AI-driven discovery. If you want the fuller picture of who these bots are, our [AI crawlers guide](/blog/ai-crawlers-guide/) breaks down each one, and [agentic browsing](/blog/agentic-browsing/) covers the real-time-fetch channel in depth.

## Common Blog Robots.txt Mistakes

Almost every serious robots.txt failure falls into one of these buckets. Each one has deindexed real blogs.

**Blocking CSS and JavaScript.** Google renders your pages to understand them, and rendering needs your stylesheets and scripts. An old habit was to `Disallow: /wp-includes/` or block asset directories to save crawl budget. Do not. If Google cannot fetch your CSS and JS, it sees a broken page and judges it accordingly. Leave assets fully crawlable.

**Shipping the staging block to production.** Staging sites are usually locked down with a site-wide block:

```
User-agent: *
Disallow: /

```

Those two lines tell every crawler to fetch nothing. When a site launches and that file gets copied to production unchanged, the entire blog becomes uncrawlable overnight. This is the most expensive robots.txt mistake there is, and it is disturbingly common. The first thing to check on any launch is that the production robots.txt does not disallow everything.

**Prefix-matching accidents.** Every `Disallow` path is prefix-matched, even without a `*`, so it blocks any URL that starts with those characters. `Disallow: /blog` blocks not just `/blog` but everything starting with those characters, including `/blog-post-one`. If you meant the directory, write `Disallow: /blog/` with the trailing slash, or you will wipe out posts you meant to keep. Test every pattern before shipping it.

**Robots.txt on the wrong host.** Robots.txt is scoped to its exact host, including the subdomain. A file at `yourdomain.com/robots.txt` does not govern `blog.yourdomain.com`. If your blog runs on a subdomain, the robots.txt that matters is `blog.yourdomain.com/robots.txt`, and it must be served from that host. Blogs hosted on a subdirectory like `yourdomain.com/blog` are governed by the root domain's file instead. Know which host serves the crawlers your blog needs, and edit the right file.

## How to Test Your Robots.txt

Never ship a robots.txt change to a revenue-driving blog without testing it. Two checks catch nearly everything.

**Google Search Console's robots.txt report.** In Search Console, the robots.txt report shows the file Google last fetched, flags syntax errors, and lets you see how Google reads it. Use it to confirm Google is fetching the file you think it is, and that no critical path is blocked. When you push a change, you can request a refetch so Google picks it up sooner.

**A live validator.** Before you deploy, paste your file into our [robots.txt validator](/tools/robots-txt-validator) to check the syntax and confirm that a given URL is allowed or blocked for a specific user-agent. It catches the trailing-slash wildcard traps and the accidental site-wide `Disallow` before they reach production. Test the URLs that matter most, your posts and your homepage, against `Googlebot` and against the AI crawlers you care about.

After any change, spot-check a real post URL in Search Console's URL Inspection tool to confirm it is still crawlable and indexed. If you want a deeper look at how blocking interacts with what actually gets indexed, see our guides on [blog indexing](/blog/blog-indexing/) and [SEO crawling](/blog/seo-crawling/).

## How Superblog Handles Robots.txt

On Superblog, robots.txt is generated for you with sane defaults, so there is no plugin to install and no file to hand-edit. Your posts, categories, and sitemap are exposed to search and AI crawlers by default. The sitemap line points at the sitemap Superblog builds and updates automatically on every deploy.

For the AI-training decision, Superblog gives you a `GPTBot` toggle so you can opt out of OpenAI model training without touching a raw file or leaving your search visibility behind. The defaults keep the search and citation crawlers allowed, which is the right posture for a blog that wants to be discovered in AI answers as well as in Google. Superblog also generates an `llms.txt` file, covered in our guide to [llms.txt and AI search](/blog/llms-txt-ai-search/). No maintenance, no plugin, no risk of shipping a staging block to production.

If you want a blog where the crawl-control layer is correct by default instead of one wrong line away from disaster, [start a 7-day free trial](https://write.superblog.ai) (no credit card required) and publish on infrastructure that gets robots.txt right for you.

## FAQ

### Do blogs need a robots.txt file?

Technically no, a blog works without one, and the absence of a robots.txt file means crawlers assume everything is allowed. In practice you want one, because it lets you point crawlers to your sitemap and block admin and internal-search URLs from wasting crawl attention. A minimal robots.txt with an `Allow: /` baseline and a `Sitemap:` line is worth having on every blog.

### Should I block AI crawlers from my blog?

Only the ones whose job you object to. Blocking training crawlers like `GPTBot` and `ClaudeBot` keeps your content out of model training while costing you no live traffic. `Google-Extended` is the exception in that class: blocking it also opts you out of Gemini grounding, though it never touches Google Search ranking or AI Overviews. Blocking search-index crawlers like `OAI-SearchBot`, `Claude-SearchBot`, and `PerplexityBot` removes your blog from AI answers and citations, which most business blogs do not want. Decide per class, not all at once.

### Does robots.txt stop a page from being indexed?

No. Robots.txt stops crawling, not indexing. A blocked page can still appear in search results if other sites link to it, usually with no description. To keep a page out of search, allow crawling and add a `noindex` meta tag so the crawler can read the instruction to leave.

### Where should the robots.txt file be located?

At the root of the exact host it governs, always at `/robots.txt`. For a subdomain blog it must be at `blog.yourdomain.com/robots.txt`. For a subdirectory blog at `yourdomain.com/blog`, the root domain's `yourdomain.com/robots.txt` applies. A robots.txt on the wrong host controls nothing on your blog.

### Can I block SEO tools from crawling my blog?

Yes. You can add `Disallow` rules for third-party SEO crawler user-agents such as `AhrefsBot` or `SemrushBot` if you do not want them mapping your content. This is optional and mostly a competitive-privacy choice. It has no effect on your Google rankings, since Googlebot is a separate user-agent.

### Will blocking Google-Extended hurt my search ranking?

No. `Google-Extended` controls only whether your content is used for Gemini training and grounding. Google states it does not affect inclusion in Google Search and is not a ranking signal, and it does not affect AI Overviews. Blocking it opts you out of Gemini training and grounding; it does not touch Search rankings or AI Overviews.

### How do I test my robots.txt before publishing it?

Use two checks. Google Search Console's robots.txt report shows the file Google fetched and flags errors, and its URL Inspection tool confirms a given post is crawlable. Before deploying, run your file through a [robots.txt validator](/tools/robots-txt-validator) to confirm each URL is allowed or blocked for the user-agents you care about, which catches wildcard traps and accidental site-wide blocks.


---
This blog is powered by Superblog. Visit https://superblog.ai to know more.
---