Evidence · document 08

Crawler Logs

In the workspace: /crawler-log-analyzer/

Availability

Included on every plan, including the free trial

This document is open to every reader on every plan. Availability tells you what a plan includes; it does not decide what you may read.

What it is for

Every other instrument here asks what a crawler would find. Your access log records what crawlers actually did: which paths they requested, how often, and what your server answered. It is the only evidence of crawler behaviour that is not a simulation, and it is sitting unread on most hosts.

How to use it

Download an access log from your hosting control panel. A single day is enough to start; a week reads better.
Upload it here, within the stated size and line bounds.
Read the summary before the detail. The response-code distribution usually contains the finding.
Follow up the paths crawlers request most and the ones your server answered with an error.

Reading the result

The summary groups requests by response class and by path. A large share of 4xx answers to a crawler means budget spent on nothing; a crawler hammering one low-value path means budget spent in the wrong place. Malformed lines are counted rather than silently dropped, so you can see how much of your file was readable.

What it cannot tell you

It reads the file you uploaded and nothing else, so it describes the period that file covers. A user agent claiming to be Googlebot may not be, and this instrument reports the claim rather than verifying it. It also cannot tell you whether a crawled page was indexed afterwards.

What it keeps

Nothing, and this one is worth stating plainly: your log file is processed and never persisted. The detail rows you see are held in this browser session only.