Crawler visits

The requests that AI crawlers make to this brand’s site, read from its own logs, and how each one is verified.

A crawler visit is a request to the project’s website whose user agent names an AI crawler, as the site’s own logs record it. Only the logs show the real crawler. A request from anywhere else can only claim to be one.

What is kept

• A row is kept only when its user agent names an AI crawler and its host is on the project’s site. Every other row is dropped before storage.
• FullReach AI stores daily counts per page, crawler and HTTP status. It stores no single request and no address.

Verified, spoofed or unverifiable

Most operators publish the addresses their crawlers send from. FullReach AI checks each request against that list, then drops the address.

Result	Meaning
Verified	The address is in the operator’s published list.
Spoofed	The address is outside the list. Somebody else sent a request that claims to be the crawler.
Unverifiable	The operator publishes no list, or the log carried no address.

Spoofed requests have their own count, and every other count leaves them out.

Three ways to send logs

Upload a log file

On the Crawler visits page, choose a file in Combined Log Format (Apache, nginx), CSV with a header row, or NDJSON. The browser reads the file and sends only the rows that name an AI crawler. The other rows stay on your computer.

Push from your CDN or server

An admin creates an ingest key on the Crawler visits page. Your log shipper posts records to the endpoint that the page shows, with the key as a Bearer token:

curl -X POST "$ENDPOINT" \
  -H "Authorization: Bearer $INGEST_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"timestamp": "2026-09-20T10:00:00Z", "request_method": "GET",
        "request_url": "https://example.com/pricing", "response_status": 200,
        "user_agent": "Mozilla/5.0 ... GPTBot/1.3", "client_ip": "20.0.0.5"}]'

The body can be a JSON array, an object that holds records, or NDJSON. Each record can have the shape above, the shape of a Vercel drain record, or the shape of an Akamai DataStream 2 record. The answer counts the records kept and the records dropped, by reason.

Connect Cloudflare

An admin gives a Cloudflare API token with Analytics Read on the zone, and nothing else, together with the zone ID. FullReach AI reads the zone each day. Cloudflare sends sampled counts without addresses, so these requests are unverifiable. The page labels them as claimed by user agent.

What a log cannot show

• ChatGPT’s agent mode sends a plain browser user agent, so no log row names it.
• FullReach AI’s own daily collection can make a platform fetch a page of this site. Such a fetch counts as a request that a person asked for.