To verify search engine crawlers, check the requesting IP address against the crawler operator’s published verification method. A user-agent string alone is not proof. For Googlebot, use either forward-confirmed reverse DNS or the appropriate official IP range list. Keep the request time, original IP, verification result and source used together in your server-log record.
Google documents two verification approaches: command-line DNS checks for individual requests and IP range matching for automated checks. Both begin with a trustworthy client IP. This guide explains how to verify search engine crawlers behind a proxy, separate unknown traffic from confirmed crawler visits and use the results in a technical investigation without treating a crawl as proof of indexing.
What log fields do you need to verify a crawler?
You need the original client IP and request timestamp to verify a crawler, plus enough request detail to understand what happened. Keep the raw record while adding a separate verification result. This preserves evidence if your interpretation changes.
- Identity: original client IP, claimed user agent and trusted proxy path.
- Request: hostname, requested path, method and timestamp with timezone.
- Response: status code, response size and any relevant cache or firewall action.
A report containing only “Googlebot visited” cannot show whether the bot received an article, redirect or challenge page. The technical SEO audit workflow uses those distinctions to turn a log observation into a specific check. Limit access to raw logs and avoid exporting unnecessary personal or authentication data into a shared editorial spreadsheet.
How do you verify Googlebot with reverse and forward DNS?
Verify Googlebot with reverse and forward DNS by resolving the logged IP to a hostname, checking the official hostname pattern and resolving that hostname back to the original IP. Both directions matter; a plausible-looking name alone is insufficient.
- Run a reverse lookup on the original client IP.
- Compare the returned hostname with Google’s documented pattern for the claimed crawler category.
- Run a forward lookup on that complete hostname.
- Confirm that the returned address set contains the original client IP.
Google’s request verification documentation separates common crawlers, special crawlers and user-triggered fetchers. Use the matching category rather than accepting every Google-owned hostname as Googlebot. Check domain boundaries exactly: a name containing “googlebot.com” somewhere inside it is not necessarily a hostname beneath that domain.
How do you match crawler IP addresses to official ranges?
Match crawler IP addresses by testing whether the logged address belongs to a network in the operator’s current official range file. Parse IPv4 and IPv6 as IP addresses and networks, rather than comparing text prefixes.
- Download the range file from the operator’s documented HTTPS location.
- Validate the response and record when it was retrieved.
- Match the original address against the correct crawler category.
Google links its current range files from the verification documentation. Other operators define their own methods. Bing’s crawler verification help is the starting point for Bingbot checks; do not apply Google’s domain rules to Microsoft traffic. Keep the downloaded range version when investigating an incident, since a later list does not necessarily prove ownership at an earlier date.
How do proxies affect crawler verification?
Proxies affect crawler verification because the origin server may log the proxy address instead of the visiting crawler. Restore the client IP only through the trusted proxy configuration for your hosting path.
- Identify which CDN or reverse proxy connects to the origin.
- Use its documented client-IP field only for requests from that trusted service.
- Check whether direct origin access can bypass those assumptions.
Cloudflare’s HTTP header documentation describes fields including CF-Connecting-IP and the behavior of forwarded addresses. A header supplied by an arbitrary direct client is not identity evidence. Ask the hosting administrator to confirm the trust boundary before using it for an allow rule. Otherwise, an attacker may provide the address you intended to verify.
What should you do when crawler verification fails?
When crawler verification fails, record the traffic as unverified and investigate the reason before changing access rules. A failed lookup can reflect a temporary DNS problem or an incorrect log field, as well as a false crawler claim.
- Distinguish a definite range mismatch from a lookup timeout.
- Check whether the claimed bot belongs to a different documented category.
- Review request behavior and existing security controls before making exceptions.
Do not give unverified traffic privileged access merely because it uses a familiar user agent. Equally, do not remove ordinary reader access while investigating one uncertain bot. The guide to crawling and indexing controls explains why robots.txt is a voluntary crawler instruction, not an authentication mechanism. Security decisions and search visibility decisions should use the evidence relevant to each.
How does crawler verification support a technical SEO audit?
Crawler verification supports a technical SEO audit by making response analysis trustworthy. Once the requester is confirmed, compare the requested URL and returned response with the version intended for public search.
- Check whether important articles return successful responses to verified crawlers.
- Inspect repeated redirects, missing files or firewall challenges.
- Correlate response changes with deployment and cache changes.
A verified request for a historic URL can reveal an obsolete redirect target. Compare that request with the historical URL map and test the destination. Keep the conclusion narrow: the log proves that a request received a particular response at a particular time. Indexing, ranking and citation require separate evidence.
What else should you know about crawler verification?
Does a Googlebot user agent prove a visit is genuine?
No. A client can send a Googlebot user-agent string without being Googlebot. Verify the source IP using Google’s documented method before treating the visit as confirmed.
Does verified crawling prove indexing?
No. Verified crawling shows that the crawler requested a resource. It does not show whether the resource was selected for an index or displayed in search results.
Should every unknown bot be blocked?
No. Unknown identity is a reason to investigate, not a complete access policy. Use observed behavior, your publication’s requirements and hosting controls to decide what access is appropriate.
How should you document the verification result?
Add the verification method, checked date and outcome beside the original log entry. Include it in your measurement baseline only as crawl evidence, with indexing and visits recorded separately. That makes future comparisons understandable to another editor or administrator.
Explore more in Technical SEO.
Written by
Theo Bennett
Theo Bennett is an editorial pen name used by Approve SEO for technical SEO and measurement coverage. The byline focuses on crawler verification, website maintenance, analytics interpretation, software contracts and the limits of benchmark data. Articles attributed to Theo draw on official documentation and other identified primary sources, with practical checks that readers can reproduce on their own sites. Examples are labelled when they are illustrative rather than measured results. Theo represents an editorial function, not an individual specialist. The portrait is an original AI-generated illustration. Source suggestions and corrections are handled through Approve SEO’s editorial contact route.