Add a robots.txt that allows search + AI crawlers — Windsurf

Without a robots.txt, defaults apply — which is usually fine for Google but leaves you invisible to ChatGPT, Claude, and Perplexity unless their bots are explicitly allowed.

rocket_launch

Fixing this in Windsurf

Codeium agentic AI IDE with Cascade

Windsurf Cascade can walk the entire project and apply multi-file edits in one pass. Paste the prompt below into your Windsurf chat and the fix rolls out across the project in one pass.

Using a different tool? Pick your stack:

Lovable Cursor Bolt v0 Replit Windsurf Claude Code Base44

The prompt for Windsurf

Copy and paste this into your Windsurf chat exactly as-is.

Using Windsurf Cascade, apply these edits across the project in one pass:

Add a robots.txt

1. Create /public/robots.txt.
2. Allow GPTBot, ClaudeBot, PerplexityBot and Google-Extended (one User-agent block each, all with `Allow: /`).
3. Add a `Sitemap:` line pointing to your sitemap.xml.

Why this matters

robots.txt is the first file any crawler requests. It tells bots what they can and cannot fetch, and at what rate. No file = default-allow for most good bots, which is fine — but you lose the ability to explicitly invite AI crawlers or block abusive scrapers.

In 2023-2025, OpenAI, Anthropic, Perplexity, and Google all introduced AI crawlers that respect `robots.txt` and only fetch content from sites that allow them. Many developers blocked them by accident through a restrictive default config; many others left them disabled by never creating the file.

If you want to appear in ChatGPT search, Claude responses, Perplexity citations, or Google AI Overviews, you must explicitly allow the respective bot. This is the single most important GEO (Generative Engine Optimization) step and takes 30 seconds.

How to use this prompt in Windsurf

1. Open your Windsurf project.
2. Copy the prompt above with the copy button.
3. Paste into the Windsurf chat and send.
4. Review the diff, accept the changes, redeploy.
5. Verify the fix using the checklist below.

Common mistakes to avoid

error_outlineShipping `User-agent: *\nDisallow: /` from a template — blocks everything.
error_outlineBlocking GPTBot, ClaudeBot, PerplexityBot because of a blog post that said "block AI crawlers" — cost yourself all AI-search visibility.
error_outlineForgetting to add `Sitemap: https://yoursite.com/sitemap.xml` at the end.
error_outlineUsing `Disallow: /api` which usually is correct, combined with `Disallow: /` which is not.
error_outlineHaving both `public/robots.txt` and `app/robots.ts` — Next.js picks one unpredictably.

How to verify the fix worked

check_circleVisit `https://yoursite.com/robots.txt` — returns the file content as plain text.
check_circleIn Google Search Console → Settings → Crawling → robots.txt report — no errors, no warnings.
check_circleTry a `curl -s https://yoursite.com/robots.txt | grep "gptbot"` — confirms GPTBot line exists.
check_circleTest a blocked URL with Google's robots.txt tester (Search Console).

Frequently asked questions

Does disallowing a URL in robots.txt remove it from Google?expand_more

No. robots.txt blocks crawling, not indexing. To remove a URL from the index use `<meta name="robots" content="noindex">` or a `noindex` HTTP header.

Which AI crawlers should I allow?expand_more

GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google AI), CCBot (Common Crawl), Applebot-Extended (Apple AI).

Does allowing AI crawlers mean they use my content for training?expand_more

GPTBot and Google-Extended are specifically for training. ClaudeBot and PerplexityBot are primarily for search. If you only want AI search citations without training, allow ClaudeBot and PerplexityBot only.

Want all 34 prompts tailored to your Windsurf site?

Pantra scans your site in 10 seconds, detects the stack, and generates the exact prompts that apply — only the ones you actually need.

Scan my site

Related Windsurf prompts

AI Search / GEO