Crawl Budget Experiment: 9 Stuck URLs, Zero Manual Indexing Requests

On July 25, 2026, nine posts on this blog were sitting unindexed. One had been live for 17 days without Google fetching it once. Another was in the worst state Search Console reports: URL is unknown to Google, despite returning HTTP 200 and sitting in a submitted sitemap.

The standard advice at that point is to open Search Console and click Request Indexing. We did the opposite. We wrote down a protocol, wrote down the decision rules, committed both to git, and did nothing for twelve days.

All nine indexed on their own. Median time from publish to the crawl that indexed them was 15 days, with a range of 6 to 25. No manual requests, no pinging, no resubmission.

The result is not the interesting part. Plenty of people have noticed that stuck pages eventually unstick. The interesting part is that the decision rules were written before the data existed, they were committed with a timestamp twelve days ahead of the outcome, and they included the outcomes that would have falsified our hypothesis. That is ordinary practice in science and close to unheard-of in SEO writing, where the usual shape is "we did a thing and it worked."

What this study does not prove

Read this before the results, not after.

• There is no control group. We cannot show that these pages would have indexed more slowly with intervention. We can only show that intervention was not necessary here. If you take one thing away, take that distinction.
• This is not evidence that Request Indexing never helps. It is evidence that on one mid-authority domain, at one publishing cadence, in one 12-day window, nine deferred URLs resolved without it.
• n = 9 tracers, on a single domain at DR 64. That is a case study, not a law. A DR 20 domain and a DR 80 domain may behave nothing alike.
• The window is confounded. Internal links were added to five of the nine tracers on July 25, the same day the experiment started. Exact details are in the confounds section below. We are not going to pretend that did not happen.
• A tenth post from the same period is still not indexed as of September 8, at 42 days old, and its coverage state has moved backwards and then forwards again. The self-healing is not universal, and reporting only the tracers that worked would be dishonest.

The pre-registration

The protocol was committed to our repository on July 25, 2026 at 20:59 +0100 (commit 9ed02fd, message docs: crawl budget experiment design, cancel manual indexing requests). The outcome was committed on August 6, 2026 at 20:20 +0100 (commit 10b7b52). Twelve days apart, in that order.

Being straight about how much that is worth to you: the repository is private, so you cannot go and check those hashes yourself. Publishing them is a commitment rather than a proof. It fixes the timestamps and the commit identifiers in public, which means if anyone ever gets access and the dates do not match what is written here, this post is falsified. That is a weaker guarantee than a public repo and a stronger one than an unsourced claim, and it is worth saying out loud in a study whose entire premise is that the rules were written first.

The governing decision, verbatim from the July 25 commit:

NO manual GSC "Request Indexing" for any URL. We test whether Google's organic crawl absorbs the 3/day cadence unaided. Manual requests would contaminate the measurement; every indexing event from here on is attributable to crawl budget + internal links + sitemap only.

The hypothesis, also verbatim from that commit:

The domain's crawl budget, not content quality, is the binding constraint on indexing speed.

And the three decision rules, exactly as written before any of the data came in. These are lifted verbatim from commit 9ed02fd:

• ≥6/9 tracers crawled AND cohort ≥80% indexed → crawl budget is sufficient with internal links doing the routing. Take the 4/day call seriously.
• Tracers move but cohort < 80% → budget absorbs 3/day with 1-2 week latency. Hold 3/day, keep the internal-link discipline (every new post gets 2+ inbound links from indexed posts at publish time), re-test Aug 8.
• [Post H] / [Post D] still uncrawled at 24/13 days → budget-constrained. Escalation levers, in order: (a) "recent posts" block on a high-crawl marketing-site surface (NOT the frozen homepage), (b) prune/noindex low-value crawl consumers (paginated /blog/N/ pages showed up in GSC page data), (c) external-link pushes (newsletter, communities), (d) only then reconsider manual requests, as a labeled intervention.

The bracketed labels are the one edit made to these quotes. The commit names the two URLs in full; they are referred to throughout this post as Post D and Post H, and every tracer is labelled rather than named. That is a commercial decision, not a methodological one: the tracers are ordinary posts on a competitive blog and the set of them, with dates, is a map of what we publish and when. Nothing in the data below depends on knowing which URL is which.

Note what that third rule does. It names, in advance, the two specific URLs whose continued failure would have counted as a loss, and it names the day count at which we would have called it. Both of those URLs indexed. Had they not, this post would have been a different post, and we would have owed you that one too.

Methodology: how we ran this crawl budget experiment

Reproducible on any site with Search Console access. The whole thing costs about twenty minutes of API calls.

1. Property:sc-domain:superblog.ai. Blog mounted at superblog.ai/blog/, so blog and marketing site share one domain and one crawl budget.
2. Baseline sweep, July 25. Run the Search Console URL Inspection API against every post published in the study window (28 URLs, published July 8 to July 20). Record coverageState, lastCrawlTime, googleCanonical, and sitemap membership for each.
3. Define tracers. Every URL not in the Submitted and indexed state becomes a tracer. Nine qualified. Freeze the list; do not add or drop members later.
4. Pre-commit the decision rules to version control before the next observation. This is the step that does the work.
5. Zero intervention for the duration. No Request Indexing, no sitemap resubmission, no IndexNow pings, no content edits to the tracers.
6. Re-inspect at the checkpoint and record per-URL state transitions.
7. Corroborate with a second, independent measurement (see below), because the Inspection API alone is not enough.

lastCrawlTime is a value, not a history

One methodological warning, and it is the most useful thing in this post. The URL Inspection API returns lastCrawlTime, not a crawl history. If Google crawls a page twice, the first fetch is gone from the record forever. This means time-to-first-crawl is not retrospectively recoverable. You can only measure it by snapshotting on a schedule while the experiment runs, which is exactly why the July 25 baseline matters: it establishes that these nine URLs had no crawl before that date, so any timestamp we later observe inside the window is bounded.

We lost one measurement to this. Post I was indexed by the August 6 checkpoint but has since been re-crawled (lastCrawlTime now reads 2026-08-10T06:21:05Z), which overwrote the crawl that actually indexed it. Its first-crawl date is unrecoverable and appears in the table below as a bound, not a value.

The second measurement

Because crawl timestamps are fragile, we cross-checked every tracer against a completely independent source: the date each URL first recorded an impression in Search Console performance data. Performance data is a daily time series and does not get overwritten.

This turns out to be a good proxy. Across the five tracers with enough volume to produce a signal, first impression followed the recorded crawl by 1 to 4 days, median 2. If a page's first impression lands well before its recorded lastCrawlTime, you know the record has been overwritten and your latency figure is wrong.

Results

All nine tracers. Crawl times are UTC as Search Console reports them. State column is a fresh index_inspect run on August 10, 2026, four days after the checkpoint.

Median publish-to-crawl across the eight measurable tracers: 15 days. Every single one was in a submitted sitemap the entire time, which is the point. Google knew the URLs existed and was choosing when to spend a fetch on them.

The crawls arrive in two clusters, not a smooth distribution: three on July 25 and 26, then nothing for six days, then five in a single wave on August 2 to 4. That shape matters more than the median. Google did not work through a queue at a steady rate. It took a bite, went away, and came back.

The cohort behind the tracers

Tracers are the interesting cases by construction, so here is the unselected population. Indexing rate for all posts published in the window, measured at three checkpoints:

The July 25 dip to 68% is the study's most useful number, because it is the one that looks like failure. Split by age, it stops looking like failure and starts looking like latency: posts 9 days or older were 14 of 16 indexed (87.5%), while the July 17 to 20 tail was 5 of 12 (42%). Nothing was broken. The recent tail simply had not been reached yet.

By August 6 the entire July 8 to 20 cohort was 28 of 28. One hundred percent, with zero manual requests, having passed through a reading of 42% two weeks earlier.

The scraper-pollution check

Any study that quotes Search Console impression or position figures owes you this section, and most do not have one.

Search Console counts automated SERP scrapers as impressions. Commercial rank trackers cycle keyword lists through geographically distributed proxy pools, and every one of those SERP fetches can land in your performance data as a real-looking impression at a real-looking position. We have found this on our own property at scale: one unrelated page on this domain has a query family carrying roughly 8,570 impressions in a 30-day window that is almost entirely a rank tracker, enough to drag the page's reported average position from the low 40s to 63.

Four signatures we screen for:

1. Permutation families. Hundreds of impressions each across near-identical word orderings of the same small token set. Humans do not distribute their searches evenly across "seo and subdomains", "subdomains and seo", and "subdomain and seo". A keyword list in a rank tracker does exactly that.
2. Impossible device splits. 1,200 desktop impressions against 9 mobile from a single country is not a human population.
3. Zero clicks at high volume. Even at position 60, thousands of genuine impressions produce a click trickle. Sustained exact zero does not happen to a page humans see.
4. Step-function onset. Organic demand does not multiply thirtyfold in four days and hold. Scraper schedules do.

The primary result of this study is immune to all of it. Index state comes from the URL Inspection API, which reports Google's own coverage verdict for a URL. It is not derived from impressions, so no amount of scraper traffic can move it. That is a deliberate design choice, not luck.

We did run the check anyway, on the one tracer carrying enough impression volume to bother with. Post B, 145 impressions across July 11 to August 9:

• Signature 1: clean. One dominant head query (69 impressions at position 14.8) plus a scatter of single-impression misspellings and near-misses of that same phrase, each appearing once or twice. Human typo variance, not systematic enumeration.
• Signature 3: clean. One genuine click, on July 31.
• Signature 4: clean. Zero impressions every day from July 11 through July 25, first impressions on July 26 (4), then a steady 6 to 15 per day. A smooth onset one day after the crawl, not a burst.
• Signature 2: ambiguous, and we are flagging it rather than waving it through. The geography is desktop-skewed: GBR 19 desktop and 0 mobile, IND 26 desktop and 2 mobile. In isolation that shape is mildly bot-like. At 19 to 26 impressions per country it is far below the volume at which a device split carries any diagnostic weight, and the other three signatures point the other way.

Verdict: genuine, with a noted weak desktop skew.

Why a query-level audit can never see the whole page

One more caveat worth publishing, because it constrains everybody doing this kind of audit. The query-dimension rows for that page sum to roughly 79 impressions against a page-level total of 145, because Search Console withholds queries below its anonymization threshold. A query-level pollution audit can therefore only ever inspect part of a page's traffic. On low-volume pages that missing fraction can be the majority of it. Treat a clean query-level audit on a small page as weak evidence, not proof.

Confounds, stated exactly

Internal links were added during the window. On July 25, the same day the experiment began, eight internal links were inserted across six donor posts pointing at five of the nine tracers: two each into Posts D and G, two into Post B, one into Post C, one into Post H. The other four tracers (A, E, F and I) got no new link and functioned as an informal control group.

The honest read on whether that mattered: the signal is confounded and does not separate. The August 2 to 4 crawl wave lifted linked and unlinked tracers alike, and the four tracers that got no new link on July 25 indexed across the full range, from 6 days to ≤17. Meanwhile the three earliest indexers (A, B and C) all had inbound links from elsewhere on the marketing site that predated the study. You could tell a link story or a "Google came back to the domain" story from this data and neither would be disprovable. We are not claiming the links caused the indexing. We are not claiming they did not.

If you want the mechanics underneath all of this, how Googlebot decides which URLs are worth a fetch and in what order, that is covered in our guide to SEO crawling. The parallel question of how AI crawlers treat the same URLs, and why their behavior is not a substitute signal for Googlebot's, is in the AI crawlers guide.

Two other changes touched the domain during the window. The site continued publishing at three posts per day throughout, which is itself a change in crawl demand, and a homepage redesign shipped on July 22. Neither targeted the tracers, but neither is nothing.

One confound is cleanly excluded by timing. A hub page was rebuilt on August 4 to route links into several tracers including Post H. That post was crawled on August 2, two days earlier, so the hub change cannot explain its crawl. It is a possible confound for continued indexing, not for the event we measured.

We cannot prove a negative to you. There is no artifact anywhere that demonstrates nobody clicked Request Indexing. The timestamped pre-registration commit and the standing written directive are the strongest evidence available, and they are evidence of intent, not proof of compliance. Any zero-intervention study by anyone has this hole in it. Ours included.

The inconvenient parts

Indexing is not traffic, and the gap is enormous. All nine tracers combined produced 2 clicks and 198 impressions over July 11 to August 9. Post H, the headline rescue, the one that sat 17 days uncrawled and then indexed at day 25, has recorded zero impressions. Not zero clicks. Zero impressions. It is indexed and it is invisible.

Some of that is the eligibility window: five of the nine were only indexed on August 2 to 4 and had less than a week of exposure inside the measurement period. But that does not rescue Post H, which was crawled on August 2 and surfaced for nothing across the seven days of performance data available since. Getting indexed removed a blocker. It did not create demand.

Two posts published just after the baseline were excluded from the tracer set by the freeze rule. Both are worth reporting. As of August 11:

• Post J, published July 26, spent fifteen days reporting URL is unknown to Google, the same worst-case state Post D started in. It was crawled on August 11 at 04:33Z and now reports Submitted and indexed. Publish to crawl: 16 days, inside the 6-to-25-day range the tracers produced and one day past their median.
• Post K, published July 28, is still not indexed at 42 days, and its state has oscillated in both directions. On August 10 it reported Discovered - currently not indexed. On August 11 it reported URL is unknown to Google, with no robots state, no indexing state and no fetch state returned at all. On September 8 it is back to Discovered - currently not indexed, with referring URLs recorded from two internal pages. Six weeks in a sitemap, twice discovered, never indexed.

The regression is the more useful of the two observations, because it contradicts the tidy progression this kind of study invites you to assume. Coverage state is not monotonic. A URL Google has discovered can stop being a URL Google admits to knowing about, without the page changing, without the sitemap changing, and without anyone touching it. If you are tracking indexing over time, do not model these states as a one-way ladder, and do not read a single URL is unknown to Google as a permanent verdict. Two pages on this domain have cleared that exact state unaided, Post D at 14 days and Post J at 16. A third, Post K, entered it and came back out to Discovered without ever being indexed, which is the clearest evidence here that these labels describe a queue position rather than a judgement about the page.

Note that we have not linked to Post K from anywhere in this article, and that is deliberate. It is still under observation. An internal link from a post that will pick up traffic is exactly the kind of intervention this study was built to exclude, and adding one now would destroy our ability to say anything clean about how long it took. Linking it would probably help it. That is the reason not to do it yet.

And the whole premise is unflattering. This is a study about a blog whose posts routinely went uncrawled for one to three weeks. A well-linked, high-authority site does not produce a dataset like this because it does not have this problem. The mid-authority middle is where the effect is observable, which is exactly why the honest version of the number is worth publishing and the flattering version would not have been.

What we concluded, and what we changed

The first pre-committed rule fired: at least 6 of 9 tracers crawled, cohort at or above 80%. Actual result was 9 of 9 and 95%. Under the rule as written, that made raising publishing cadence from 3 to 4 posts per day defensible.

We held at 3 per day anyway. The gate was a constraint being tested, not a target to max out, and once crawl budget stopped being the binding limit the cadence question became a content-supply question instead. Recording that here because a study that only reports the decisions matching its own rules is not reporting decisions.

The zero-intervention rule became permanent. Requesting indexing is now off the table on this property, not because we think it is harmful, but because we now have a measured baseline for what the domain does unaided and every manual request contaminates it.

How to run this on your own site

1. Sweep every URL published in your last 30 days through the URL Inspection API. Record coverageState and lastCrawlTime.
2. Freeze the not-indexed set as your tracers. Do not adjust membership later.
3. Write your decision rules now, including the outcome that would make you conclude you were wrong, and commit them somewhere timestamped.
4. Do nothing for two weeks. Two weeks is the minimum: our median was 15 days, so a 7-day study on this domain would have concluded, wrongly, that the pages were dead.
5. Re-inspect. Cross-check every crawl date against first-impression date in performance data.
6. Before quoting any impression or position figure, run the four-signature pollution check. Segment by query cohort. Never cite a raw page-level average.

The step people skip is step 3, and it is the only one that changes what the exercise is worth. Without it you will find a story in whatever you observe.

Limitations, collected

For anyone citing this: n = 9 tracers plus a 38-URL cohort, one domain, DR 64, blog on a subdirectory, publishing 3 posts per day, window July 25 to September 8, 2026. No control group. No randomization. Internal links added to five of nine tracers on day zero, confounding the link variable beyond separation. Time-to-first-crawl is bounded rather than exact for one tracer and is upper-bounded generally by the API's single-timestamp limitation. Pre-registration is attested by commit hash and timestamp in a private repository, so it is a public commitment rather than an independently checkable artifact. Non-intervention is attested by that same written directive, not proven. One contemporaneous post remains unindexed at 42 days, with a coverage state that oscillated rather than held, and it is counted against the finding rather than excluded from it. Indexing latency was 6 to 25 days, so any conclusion about faster or slower windows is extrapolation.

What the study supports: on this domain in this window, nine URLs that Google had deferred for 6 to 25 days all indexed without manual requests, and a cohort reading of 42% on a recent tail resolved to 100% by waiting.

What it does not support: any general claim that Request Indexing is useless, unnecessary, or harmful.

FAQ

Does Request Indexing speed up indexing?

This study cannot answer that, and neither can any study without a control group. What it can show is that intervention was not required here: nine URLs Google had deferred for 6 to 25 days all indexed with zero manual requests, and a cohort reading of 42% on July 25 became 100% by August 6. "Not necessary on this domain in this window" is a much smaller claim than "never helps", and we are only making the smaller one. We no longer use manual requests on this property, for a measurement reason rather than a safety one: we now have a baseline for what the domain does unaided, and every manual request would contaminate it.

How long should a page take to index?

For the stuck pages in this study, publish to the crawl that indexed them ran 6 to 25 days, with a median of 15. The arrivals were clustered, not steady: three crawls on July 25 and 26, six days of nothing, then five in a single wave on August 2 to 4. Higher-authority, better-linked sites index much faster than this, so treat 15 days as one mid-authority domain's number and not a benchmark. The transferable part is the observation window. A 7-day check on this domain would have concluded, wrongly, that the pages were dead. Two weeks is the minimum before the data means anything.

Why is my page "Discovered, currently not indexed"?

In this dataset it meant Google knew the URL existed and had not yet chosen to spend a fetch on it. Seven of the nine tracers sat in that state on July 25, and all seven resolved without intervention, the slowest at 25 days from publish. Sitemap membership did not accelerate it; every tracer was in a submitted sitemap the entire time, which is what makes the state a scheduling decision rather than a discovery failure. Read it as queue position, not as a verdict on the page, at least until it is well past your own measured median. That said, waiting is not a guarantee. One contemporaneous post of ours is still unresolved at 42 days, and it did not sit still: it went from Discovered - currently not indexed to URL is unknown to Google and back to Discovered. These states are not a one-way ladder, and six weeks in a submitted sitemap has not been enough for this one.

The blog studied here runs on Superblog, which is why publish timestamps are known to the minute and every URL was in a generated sitemap from the moment it went live. If you want to run the same protocol on your own blog, the data you need comes from Search Console, not from us. Superblog offers a free trial if you want the publishing side handled.