AI Search (GEO)
AI Search (GEO)High

AI Crawlers Allowed

Checks that GPTBot, ClaudeBot, PerplexityBot, and Google-Extended are not blocked in robots.txt.

What this check measures

We parse `/robots.txt` and simulate the user-agent matching for each major AI crawler: GPTBot (OpenAI), ChatGPT-User (OpenAI search), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google AI), CCBot (Common Crawl).

Why it matters

If AI crawlers cannot fetch your site, you cannot appear in ChatGPT search results, Claude citations, Perplexity answers, or Google AI Overviews. This is the #1 miss for GEO (Generative Engine Optimization) — most sites accidentally block AI either by a restrictive default or by copying anti-AI advice from a blog post.

search

How our audit detects it

Fetch /robots.txt. Simulate robots.txt parser for each AI user-agent against the root URL. Flag if any is blocked.

Typical findings

  • error_outline`User-agent: GPTBot\nDisallow: /` — explicitly blocks all AI search.
  • error_outlineDefault `User-agent: *\nDisallow: /private/` accidentally cascades to block AI because of specific rules.
  • error_outlineNo robots.txt at all — AI is technically allowed, but no explicit invite either.

How to fix

Add explicit `Allow: /` blocks for each AI crawler. If you want AI search but not training: allow ChatGPT-User, ClaudeBot, PerplexityBot; block GPTBot, Google-Extended. See /prompts/allow-ai-crawlers for stack-specific copy-paste.

Copy-paste fix prompt for your stack

Lovable · Cursor · Bolt · v0 · Replit · Windsurf · Claude Code · Base44

View the fix prompt →

Frequently asked questions

Does allowing GPTBot mean OpenAI trains on my content?expand_more
Yes — GPTBot is explicitly for training. To appear in ChatGPT search without training, allow only ChatGPT-User.
Does allowing AI hurt SEO?expand_more
No. AI crawlers are separate from Googlebot. Allowing them does not affect Google Search rankings.

Want this checked on your site?

Pantra runs the full audit (SEO, Security, GEO, Performance, Schema, Technical, Images) in 10 seconds and generates stack-specific fix prompts.

Scan my site

Related checks

AI Search (GEO)

JSON-LD Structured Data

Checks for JSON-LD structured data in the page <head> — essential for rich results and AI citations.

AI Search (GEO)

Server-Side Rendering

Checks that main content renders in the initial HTML response, not only after JavaScript executes.

AI Search (GEO)

llms.txt File

Checks for a /llms.txt file — an AI-specific guide to your site content.

AI Search (GEO)

Static Content Density

Checks how much meaningful text ships in the initial HTML without JS execution.

Learn more in the Glossaryarrow_forward