Add a robots.txt that allows search + AI crawlers — Base44

Without a robots.txt, defaults apply — which is usually fine for Google but leaves you invisible to ChatGPT, Claude, and Perplexity unless their bots are explicitly allowed.

rocket_launch

Fixing this in Base44

No-code + AI app builder

Base44 generates reusable blocks — many fixes are applied by regenerating the affected block with clearer instructions. Paste the prompt below into your Base44 chat and the fix rolls out across the project in one pass.

Using a different tool? Pick your stack:

Lovable Cursor Bolt v0 Replit Windsurf Claude Code Base44

The prompt for Base44

Copy and paste this into your Base44 chat exactly as-is.

In Base44, regenerate the affected blocks with these exact instructions:

Add a robots.txt

1. Create /public/robots.txt.
2. Allow GPTBot, ClaudeBot, PerplexityBot and Google-Extended (one User-agent block each, all with `Allow: /`).
3. Add a `Sitemap:` line pointing to your sitemap.xml.

Why this matters

robots.txt is the first file any crawler requests. It tells bots what they can and cannot fetch, and at what rate. No file = default-allow for most good bots, which is fine — but you lose the ability to explicitly invite AI crawlers or block abusive scrapers.

In 2023-2025, OpenAI, Anthropic, Perplexity, and Google all introduced AI crawlers that respect `robots.txt` and only fetch content from sites that allow them. Many developers blocked them by accident through a restrictive default config; many others left them disabled by never creating the file.

If you want to appear in ChatGPT search, Claude responses, Perplexity citations, or Google AI Overviews, you must explicitly allow the respective bot. This is the single most important GEO (Generative Engine Optimization) step and takes 30 seconds.

How to use this prompt in Base44

1. Open your Base44 project.
2. Copy the prompt above with the copy button.
3. Paste into the Base44 chat and send.
4. Review the diff, accept the changes, redeploy.
5. Verify the fix using the checklist below.

Common mistakes to avoid

error_outlineShipping `User-agent: *\nDisallow: /` from a template — blocks everything.
error_outlineBlocking GPTBot, ClaudeBot, PerplexityBot because of a blog post that said "block AI crawlers" — cost yourself all AI-search visibility.
error_outlineForgetting to add `Sitemap: https://yoursite.com/sitemap.xml` at the end.
error_outlineUsing `Disallow: /api` which usually is correct, combined with `Disallow: /` which is not.
error_outlineHaving both `public/robots.txt` and `app/robots.ts` — Next.js picks one unpredictably.

How to verify the fix worked

check_circleVisit `https://yoursite.com/robots.txt` — returns the file content as plain text.
check_circleIn Google Search Console → Settings → Crawling → robots.txt report — no errors, no warnings.
check_circleTry a `curl -s https://yoursite.com/robots.txt | grep "gptbot"` — confirms GPTBot line exists.
check_circleTest a blocked URL with Google's robots.txt tester (Search Console).

Frequently asked questions

Does disallowing a URL in robots.txt remove it from Google?expand_more

No. robots.txt blocks crawling, not indexing. To remove a URL from the index use `<meta name="robots" content="noindex">` or a `noindex` HTTP header.

Which AI crawlers should I allow?expand_more

GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google AI), CCBot (Common Crawl), Applebot-Extended (Apple AI).

Does allowing AI crawlers mean they use my content for training?expand_more

GPTBot and Google-Extended are specifically for training. ClaudeBot and PerplexityBot are primarily for search. If you only want AI search citations without training, allow ClaudeBot and PerplexityBot only.

Want all 34 prompts tailored to your Base44 site?

Pantra scans your site in 10 seconds, detects the stack, and generates the exact prompts that apply — only the ones you actually need.

Scan my site

Related Base44 prompts

AI Search / GEO