⚙️ Free Robots.txt Tools

AI-Friendly Robots.txt Generator

Generate a custom robots.txt file to control how AI bots like ChatGPT, Claude, and Perplexity crawl your website.

🤖 AI Crawler Access

Toggle which AI bots are allowed to crawl your content for LLM training and RAG search.

GPTBot

OpenAI — ChatGPT web crawler

ChatGPT-User

OpenAI — Real-time browsing for ChatGPT users

ClaudeBot

Anthropic — Claude web crawler

PerplexityBot

Perplexity — Perplexity AI search engine

Google-Extended

Google — Gemini and Google AI features

Bytespider

ByteDance — TikTok/ByteDance AI crawler

🛡️ General Rules

Block /admin/Block /api/Block /private/Block /tmp/

🌐 Global Settings

Include Sitemap URL

Add Crawl-delay (seconds)

📄 robots.txt LIVE

1User-agent: GPTBot
2User-agent: ChatGPT-User
3User-agent: ClaudeBot
4User-agent: PerplexityBot
5User-agent: Google-Extended
6User-agent: Bytespider
7Allow: /
8Crawl-delay: 1
9 
10# --- General Web Crawlers ---
11User-agent: *
12Disallow: /admin/
13Disallow: /api/
14Disallow: /private/
15Disallow: /tmp/
16Crawl-delay: 1
17 
18Sitemap: https://example.com/sitemap.xml

Next Steps: Deploy & Verify

Upload the generated robots.txt file to your website's root directory (e.g., https://yoursite.com/robots.txt).

Verify with our Robots.txt Auditor →