Crawler response CSV analyzer
Calculate the proportion of HTTP 429, 404 and server-error responses in your own crawl observations. Paste a CSV or choose a local file. Processing stays in your browser; this tool does not crawl websites, verify crawler identities, upload the file or call an AI service.
Use a status column for individual responses, or add count for aggregated totals. An optional window column separates comparison periods. Extra columns are ignored. Counts must be positive whole numbers. The tool supports 2 MiB, 20,000 rows and 12 windows. Remove IP addresses, URLs, queries, credentials and other unnecessary private fields before sharing a CSV with anyone.
No file has been analyzed. Inputs are not saved automatically.
| Window | Total | 2xx | 3xx | 404 | 429 | 429 rate | 5xx |
|---|
A worked example with real observations
Our Googlebot rate-limit case study compares a 47-minute window containing 5,984 verified Google crawler article requests, including 532 responses with status 429, with a later 17-minute-43-second window containing 241 verified requests and no 429s. The observed rates are 532 ÷ 5,984 × 100 = 8.890% and 0 ÷ 241 × 100 = 0%. Three later requests returned genuine 404s; they remain in the denominator.
Download the aggregate example CSV. It contains counts and window labels, without IPs or requested URLs. Different traffic, window lengths and system load prevent a controlled causal or ranking conclusion.
Interpret the result before changing a server
- A user-agent string alone does not establish that a request came from a search engine. Verify identities separately using the crawler operator’s documented method.
- Keep successful requests and errors in the same population. Dividing 429s by errors alone exaggerates the overall rate.
- Separate article requests from health checks, private account routes and application APIs. Compare like populations and record the exact UTC boundaries.
- A real missing page should return 404. A blocked request and an unavailable server need different investigation.
- Zero observed errors in a short window does not establish zero future risk. A rate calculation does not prove indexing, impressions or clicks.
The Google HTTP and network-error documentation explains how responses affect crawling. For a reproducible observation log, use the revision-record tool and retain your configuration change and measurement boundaries.