Free tool · no signup

robots.txt generator.
AI-aware. Per-bot toggles. Copy-paste ready.

Most generators only know about Googlebot and Bingbot. This one has dedicated buckets for the 7 AI/LLM crawlers that now matter — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, anthropic-ai, Bytespider — and three preset policies for the most common situations. Test the result with /tools/robots-tester.

Preset
Per-bot policies
Search engines
  • GooglebotGooglebot
  • BingbotBingbot
  • DuckDuckBotDuckDuckBot
  • Yahoo SlurpSlurp
  • YandexBotYandexBot
  • BaiduspiderBaiduspider
AI / LLM crawlers
  • GPTBot (OpenAI crawler)GPTBot
  • ClaudeBot (Anthropic crawler)ClaudeBot
  • PerplexityBotPerplexityBot
  • Google-Extended (Gemini training)Google-Extended
  • CCBot (Common Crawl)CCBot
  • anthropic-ai (legacy Claude UA)anthropic-ai
  • Bytespider (ByteDance)Bytespider
Social preview bots
  • Facebook external hitfacebookexternalhit
  • TwitterbotTwitterbot
  • LinkedInBotLinkedInBot
Site-wide settings
/robots.txt
# Catch-all — applies to any user-agent not overridden below
User-agent: *
Disallow:
Disallow: /admin
Disallow: /api/internal

# Search engines
User-agent: Googlebot
Disallow:

User-agent: Bingbot
Disallow:

User-agent: DuckDuckBot
Disallow:

User-agent: Slurp
Disallow:

User-agent: YandexBot
Disallow:

User-agent: Baiduspider
Disallow:

# AI / LLM crawlers
User-agent: GPTBot
Disallow:

User-agent: ClaudeBot
Disallow:

User-agent: PerplexityBot
Disallow:

User-agent: Google-Extended
Disallow:

User-agent: CCBot
Disallow:

User-agent: anthropic-ai
Disallow:

User-agent: Bytespider
Disallow:

# Social preview bots
User-agent: facebookexternalhit
Disallow:

User-agent: Twitterbot
Disallow:

User-agent: LinkedInBot
Disallow:

# Sitemaps
Sitemap: https://yoursite.com/sitemap.xml
17 bot groups · 2 disallow paths · 1 sitemap line

Vyrable's AI Visibility tracker shows you exactly which of these AI crawlers are actually hitting your site — so you can choose the policy with data, not vibes.