To investigate whether AI discovery was blocked at Ithy, we checked a fixed 12-hour HTTP log window against the providers' published crawler IP ranges. There were 3,018 request records with both a matching declared agent and a matching published IP range. None returned HTTP 429. These observations support reachability during that window; they do not measure citations, human visits or ranking.
What was measured
The window begins October 23, 2026 at 16:00:00 UTC and ends just before October 24 at 04:00:00 UTC. A read-only query selected App Engine nginx.request records for named AI discovery and user-fetch agents. It returned 3,037 records, below the 10,000-record retrieval limit. The comparison used the published provider prefix files retrieved on October 24.
The aggregate download contains agent category, IP-list-match classification, HTTP status and request count. It contains no IP addresses, requested URLs, account identifiers, prompts or article bodies. These are request records across the site, potentially including supporting resources and redirects, not unique article readers. Log arrival delay and provider identification limits still apply.
The observed responses
| Declared agent | HTTP 200 | HTTP 301 | HTTP 404 | HTTP 499 | Total |
|---|---|---|---|---|---|
| OAI-SearchBot | 548 | 3 | 0 | 0 | 551 |
| ChatGPT-User | 131 | 3 | 1 | 2 | 137 |
| PerplexityBot | 2,325 | 0 | 0 | 0 | 2,325 |
| Claude-User | 5 | 0 | 0 | 0 | 5 |
| Total | 3,009 | 6 | 1 | 2 | 3,018 |
Another 19 records declared one of the selected agents without a matching range in this check. They are separately labeled in the aggregate CSV. An unmatched record is not automatically a fake bot: alternate retrieval infrastructure, changing lists and proxy behavior can affect identification. A range match is also not a cryptographic attestation of a historical request.
No selected record returned 429 or a 5xx status. The two 499 records describe a client connection closing in these nginx logs; they do not establish why it closed. The 404 remains a 404 rather than being counted as a successful content retrieval.
Search indexing and user fetches are different
OpenAI distinguishes OAI-SearchBot from ChatGPT-User and GPTBot: search discovery, user-triggered access and model training are separate uses. The public ranges used here were SearchBot and ChatGPT-User.
Perplexity describes PerplexityBot as search discovery and separately describes user-directed retrieval. We checked its published PerplexityBot ranges. Anthropic likewise distinguishes Claude-SearchBot, Claude-User and its training crawler; its documentation links the published ranges used in this check.
Google AI impressions need their own measurement boundary
Google's Search generative AI performance report covers AI Overviews and AI Mode impressions. Its chart groups by property; its Pages table groups by URL. These totals can differ. Dates use Pacific Time, and page exports have a 1,000-row limit. This AI report includes data from the ordinary Web performance report, so adding their impressions would double-count overlapping exposure.
Here is an original fictional example, assuming both links qualify as shown in the same AI feature. It illustrates aggregation, not Ithy's private report data:
| Display | Links shown | Property impressions | Page A impressions | Page B impressions |
|---|---|---|---|---|
| First | A and B from the same site | 1 | 1 | 1 |
| Second | A only | 1 | 1 | 0 |
| Total | Two displays | 2 | 2 | 1 |
The page-column sum is 3 while the property total is 2. Neither value supplies a click count or proves that three different people read an article. Preserve the report, period, filters, aggregation level and row limit before comparing results.
The separate Search generative AI control determines eligibility for the covered Google AI features. Inclusion is the default; child properties can inherit or override a parent. This setting does not improve ordinary Web rankings or control AI training, and eligibility does not guarantee a citation. Check the actual property before changing anything.
What this changes—and what it cannot establish
For this observed window, widespread AI-crawler throttling was not supported by the data. We retained the existing access controls and rate limits. The next useful work was improving existing explanations and references, rather than creating an unneeded crawler exemption.
A 200 response alone does not show that an answer cited the page. An AI citation is not a human click, and a click is not a completed research session or purchase. Bing's AI Performance documentation describes citation reporting as a separate observation. Different providers and reporting windows cannot simply be added together.
There is no before/after ranking experiment here. One site and one short window cannot establish behavior for all publishers, all bots or future traffic. Absence of an agent from this selected window is not proof that the provider never accesses the site.
Apply the method to your own site
To reproduce this article's counts, save the aggregate CSV and the Python summary script in the same folder, then run the command below. The script uses only Python's standard library and the downloaded aggregate; it makes no network requests and reads no raw logs. The separate synthetic methods example uses invented records for teaching; it does not contain this study’s observations.
python3 ithy-ai-crawler-summary.py
The output separates 3,018 matched request records from 19 claimed-only records and preserves the individual agent and HTTP-status totals. Recomputing the aggregate does not independently authenticate the source logs or establish citations, readers, clicks or ranking gains.
- Choose a fixed UTC window and one request-log source. Record the retrieval limit and whether it was reached.
- Check the appropriate public robots directives and actual HTTP responses; keep private-page authentication intact.
- Compare both the agent and the actual client address against the provider's current documented ranges. Keep claimed-only traffic separate.
- Publish only safe aggregate data if you share a result. Preserve real errors in the counts.
- Use actual citation and referral reports to measure the later steps instead of inferring them from crawling.
Google's official AI-search guidance emphasizes useful original content and ordinary search requirements; it says Google Search does not use llms.txt for visibility or rankings. A special text file is not a substitute for a crawlable, accurate page.
