Approve SEO

A clearer view of search.
Practical knowledge for people who publish.

The publication
Technical SEO

Search Engine & AI Crawler Directory

Approve SEO Research DeskPublished 5 min read

The search engine and AI crawler directory is a reference list of Googlebot, Bingbot and AI bots from OpenAI, Anthropic and Perplexity. Compare 11 entries by operator, crawler purpose, user-agent token, robots.txt behavior and IP verification method. The directory distinguishes search indexing bots, AI training crawlers, user-requested fetchers and Google-Extended, a content-use control without its own HTTP user agent. Each crawler entry links to official documentation and records the date checked. Use the list to identify web crawlers in server logs and review access rules without confusing search visibility with training permission.

Explore the crawler directory

11 selected search, training and user-triggered crawler/control entries. Operator documentation describes stated behavior; this directory is not an independent compliance test.

Download the dataset (CSV)

11 records

Dataset change log

  • September 11, 2026 — initial sourced dataset assembled for editorial review; not published.

What does this search engine and AI crawler directory cover?

The first edition covers common search bots, selected AI search and training agents, user-directed fetchers, and Google-Extended. It is a practical reference for publishers deciding how different products may access their pages.

  • Search: Googlebot, bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot.
  • Training: GPTBot and ClaudeBot.
  • User requests: ChatGPT-User, Claude-User and Perplexity-User.
  • Content-use control: Google-Extended, including its documented grounding scope.

Advertising fetchers, specialist image bots and general-purpose scraping tools fall outside this edition. A missing identity means it is outside our present coverage, not that the operator has no other bots.

How do search crawlers differ from training bots?

Search crawlers support discovery and retrieval; training bots collect material that may inform model development. A publisher can make different access decisions for those purposes when an operator exposes independent controls.

  • Identify the product you want to reach before choosing a token.
  • Check whether its search and training preferences are independent.
  • Review grounding separately when a control covers both training and answer-time use.

OpenAI’s crawler documentation separates OAI-SearchBot from GPTBot. Permission for one should not be assumed to grant or deny the other. The directory preserves those distinctions instead of treating every request carrying an AI brand as the same activity.

Which user-triggered agents follow robots.txt?

User-triggered agents have different documented robots policies, so the word “User” does not establish a common rule.

  • ChatGPT-User: OpenAI says robots.txt rules may not apply.
  • Perplexity-User: Perplexity says requested fetches generally ignore robots.txt.
  • Claude-User: Anthropic includes this agent within its stated robots-respecting policy.

The Perplexity crawler reference distinguishes its search bot from its user-requested fetcher.

Anthropic’s policy describes three separate identities. These are operator statements; the directory does not certify that every request you receive complies with them.

Is Google-Extended a separate crawler?

Google-Extended is a robots.txt product token without its own HTTP user-agent string. It controls specified uses of content collected through existing Google crawlers.

Google’s crawler reference includes future Gemini training and specified Gemini and Vertex AI grounding uses within that scope. It says the token does not affect Google Search inclusion or ranking.

A log search for “Google-Extended” therefore cannot establish whether this preference is respected. Separate the request you can observe from the downstream use you are asking the operator to permit or restrict.

How do you verify a crawler in server logs?

Verify a crawler using the request’s source IP and the operator’s current method. A user-agent header is a claim anyone can copy.

  1. Capture the source IP, timestamp, requested URL, response status and user agent.
  2. Open the verification source for the claimed operator.
  3. Match the IP against the current published ranges, where supported.
  4. For DNS verification, check the operator’s required hostname suffix and resolve it forward to the original IP.
  5. Record the verification result separately from the claimed bot name.

Microsoft’s verification instructions require both directions of the DNS check when that method is used. Behind a CDN, use the trusted client-IP field configured by your hosting provider; an arbitrary forwarded header is not reliable evidence.

Can you allow search and block training?

You can express separate search and training preferences where the operator documents independent tokens. This limited example allows OpenAI search crawling while declining GPTBot training collection:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

This is an illustrative fragment, not a replacement robots.txt file. Preserve existing private-path restrictions in the appropriate named groups and test the resulting file. It does not set Anthropic, Google or Perplexity policy, and it is not a security barrier against unauthorized access.

How should a publisher test a crawler policy?

A publisher should test the final access path, including robots.txt, the CDN and the origin server. A permissive file cannot help a request rejected before it reaches the page.

  • Keep a dated copy of the original rules and the reason for each change.
  • Check one intended public URL and one intentionally restricted path.
  • Review genuine crawler requests after the operator has refreshed its rules.
  • Compare response codes before and after changing firewall controls.

Our technical SEO audit explains the broader checks around access, while robots.txt versus noindex separates crawling decisions from indexation instructions. Avoid changing several control layers at once when you need to identify the cause of a problem.

How is the crawler directory maintained?

The Research Desk checked this edition against primary documentation on September 11, 2026. The maintenance schedule is monthly, with additional checks after operator announcements. This is a documentation review, not an independent test of crawler behavior.

  • Unknown source revision dates remain unreported rather than inferred from our visit.
  • Examples illustrate documented strings; they are not immutable matching patterns.
  • IP-list links point to operator feeds rather than frozen address inventories.

The change log records this initial 11-entry edition. Corrections should include the affected entry and an official source so the evidence can be checked and the downloadable record updated.

What else should publishers know about crawler access?

Does blocking a crawler remove indexed pages?

Blocking crawling is not a reliable removal method. Existing URLs can remain discoverable, and a blocked crawler may not read a new noindex instruction. Match the removal method to the platform and the intended outcome.

Can an allow rule guarantee an AI citation?

An allow rule cannot guarantee a citation. It expresses access permission; selection, indexing and attribution remain separate decisions. Measure verified visits and resulting referrals separately using a measurement baseline.

Written by

Approve SEO Research Desk

Approve SEO Research Desk is a collective publication byline for documentation-based analysis of tools, technical changes and measurement. The byline describes the work behind a piece; it is not a fictional person or a claim about team size. Its articles focus on what a source actually says, what remains uncertain and which practical checks a website owner can perform. Product comparisons explain selection criteria instead of presenting unsupported hands-on ratings. Update coverage links to the original announcement and keeps the announcement date separate from the article’s publication date. Readers can report inaccuracies or suggest better primary evidence through the editorial contact form.

Keep exploring

More in this section ↗