Let AI crawlers in, or go invisible in AI answers
Most sites invisible to ChatGPT, Perplexity, and Google AI Overviews aren't losing on content. They block the crawler before it reads a single page.
AI can read your robots.txt and tell you exactly which bots are shut out. But it can't see your live file unless you paste it or let it browse, so give it the real thing.
Fastest path: one prompt, end to end
Open yoursite.com/robots.txt, copy the whole file, and paste it into this prompt in ChatGPT or Claude:
🤖 AI prompt — paste into ChatGPT / Claude
You are a technical SEO auditing a site's robots.txt for AI-crawler access.
Here is my full robots.txt:
[paste the entire file]
The AI crawlers that matter in 2026 are: GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot, Claude-User (Anthropic); PerplexityBot, Perplexity-User (Perplexity); Google-Extended (Google's Gemini/AI-training opt-out token); plus CCBot, Bytespider, Amazonbot, Applebot-Extended.
Do this and return ONE markdown table:
1. For EACH crawler above, say whether my file ALLOWS or BLOCKS it, and quote the exact line that decides it.
2. Flag any bot blocked by a generic wildcard (User-agent: *) that was probably not intended.
3. Recommend the exact lines to ADD to allow the AI crawlers I should let in.
4. If you are unsure whether a user-agent string is current, open its official docs to confirm; do not invent a user-agent name. If you can't browse, say which ones you couldn't verify.
Columns: Crawler | Allowed or Blocked | Deciding line | Recommended change
Warn me clearly if Googlebot (Search) is blocked anywhere.
Ask the AI to confirm the user-agent names against each vendor's live docs. The names drift, and a stale block is worse than none.
Or do it in 5 steps
- Pull your live robots.txt at yoursite.com/robots.txt. Hosting platforms, security plugins, and CDNs rewrite it without telling you.
- Match each AI bot to allow or block. Allow GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-SearchBot, Claude-User, and Google-Extended unless you have a specific reason not to.
- Never confuse Google-Extended with Googlebot. Google-Extended only opts you out of Gemini/AI training. Googlebot is regular Search. Block the wrong one and you either do nothing or kill your search traffic.
- Edit and re-deploy. On Shopify, edit robots.txt.liquid. On WordPress, check the SEO plugin's crawler settings, not just the raw file.
- Re-verify with curl. Run
curl -A "GPTBot" https://yoursite.com/your-page and read the raw HTML. If the content you care about isn't in that output, no AI engine sees it either, because these bots don't run JavaScript.
Why AI-first: the model reads a robots.txt file and maps every crawler to allowed/blocked in seconds. Its one weakness is a stale user-agent name, so make it confirm the names against official docs.
What blocking actually costs, and two traps
- A 2026 check of 120+ prominent sites found nearly 45% block at least one major AI crawler, most-blocked being GPTBot, CCBot, and ClaudeBot, usually caught by a generic anti-scraper rule.
- These bots don't execute JavaScript. GPTBot and ClaudeBot sometimes fetch JS files but never run them. Client-side-rendered content looks like a blank shell. Use server-side rendering or static generation for anything you want quoted.
- llms.txt is a nice-to-have, not the fix. Google has said plainly it doesn't use it, and no major engine guarantees it reads the file. Ship one once your access and content are solid; don't mistake it for the fix.
The legacy tokens anthropic-ai and claude-web are deprecated. Rules targeting them do nothing against current Claude traffic.
Re-check after every plugin, CDN, or hosting change.