Googlebot 429 responses: an App Engine rate-limit case study
Published October 23, 2026 · Ithy engineering observation · Prepared with AI assistance from deployment evidence and the linked public documentation.
Googlebot can reach a public page and still lose crawl requests to a rate limiter. In one measured Ithy window, 532 of 5,984 article requests from verified Google crawler IPs returned HTTP 429. Several crawler IPs shared the subnet used as the limiter’s key. Separating verified readers into individual IP buckets preserved the existing per-IP limit and removed that particular grouping conflict.
This note documents a real operational correction, with an aggregate dataset and a local CSV analyzer. It does not claim a search-ranking recovery, an experiment with randomized traffic, or a universal remedy for Google traffic loss.
Two observed windows
| Observation | Window | Total | HTTP 200 | HTTP 429 | HTTP 404 | 429 rate |
|---|---|---|---|---|---|---|
| Before the correction | October 22, 22:55–23:42; 47 minutes | 5,984 | 5,448 | 532 | 4 | 8.890% |
| After the correction | October 23, 05:40:07–05:57:50; about 17 minutes 43 seconds | 241 | 238 | 0 | 3 | 0% observed |
The first bounded log query returned all 5,984 matching records below its 10,000-record limit. Fourteen verified Google IPs shared three normalized subnets. The largest observed calendar-minute count was 65 requests for one IP and 244 for one subnet. Calendar bins are not identical to a rate limiter’s moving windows, and process-local limiter state can differ between instances.
The later query returned all 241 matching records, also below its limit. Seven verified IPs were represented. The three 404s were real missing-page responses, not rewritten as successful articles. No request-level server 5xx appeared in that target-version observation window. Raw IPs, requested URLs and customer text are omitted from the shared dataset.
Download the aggregate observations. Calculate each rate as 429 responses ÷ all matching responses × 100. The download is sufficient to reproduce the percentages, but cannot independently reproduce the private log query or crawler identity verification.
Verify the crawler and the client address
A Googlebot user-agent string is a claim, not authentication. We matched the observed client addresses against Google’s then-current official common-crawler IP ranges. Google documents IP-list and DNS verification approaches in Verifying Google crawlers; its common-crawler prefix file is public. A stored list needs a refresh policy and a safe fallback when stale.
There was also a proxy boundary. The application runs behind App Engine’s frontend. Its documented X-Forwarded-For handling appends the client IP and the forwarding-rule IP. Client-supplied entries farther left are not authoritative. We used that documented appended position only with an explicitly configured trusted frontend. See the App Engine flexible request-header reference. Copying the first forwarded address or trusting an arbitrary header would let clients select a false identity.
Change the grouping, retain the protections
The scoped correction applies only to canonical public article GET or HEAD requests without query parameters, with both the expected crawler agent and a verified Google IP. Each verified IP still has the same 100-request-per-minute limit. Ordinary reader, account, sign-in and generation limits remain. Private answers still require their normal access checks. The page content is the same for readers and crawlers.
Invalid, missing or stale verification inputs fall back to the ordinary limit. There is no network lookup in a page request. Rate-limited responses also retain a useful Retry-After value. RFC 6585 defines status 429, and Google explains crawl effects in its HTTP and network-error guidance.
What this demonstrates—and what remains uncertain
The measurements identified real crawl throttling and the deployed application recorded use of the verified-reader bucket. The follow-up window contained no 429s. However, it was shorter and much quieter; traffic, timing and load were not controlled. These observations do not establish the full causal improvement, long-term error rate, indexing outcome, or any increase in impressions or clicks. They do not establish the cause of an earlier ranking decline.
A useful investigation starts with exact populations and timestamps, separates startup health checks from real requests, verifies identities independently, and keeps successful requests in the denominator. Record the actual change, test privacy and spoofed-header behavior, and observe the deployed policy before drawing a conclusion. The CSV analyzer makes the arithmetic reproducible without sending a visitor’s logs to Ithy.
More source-checked guides and tools · How to assess Ithy research