Free Robots.txt Generator
Build a robots.txt file that tells crawlers which site areas they may request and where to find your sitemap. Use it to protect crawl budget on private paths, not to remove already indexed pages.
Pick the crawlers you care about — including AI bots like GPTBot, ClaudeBot, and PerplexityBot — set allow/disallow rules per agent, point to your sitemap, and copy a valid robots.txt for your site root. Built for founders and marketers shipping fast.
All crawlers (*)
Default rules for every bot
User-agent: * Disallow:
How it works
Save the output as robots.txt at your domain root (https://yoursite.com/robots.txt). Each block tells one user-agent what it may crawl. An empty Disallow: line means “allow everything”; Disallow: / blocks the whole site. Crawl-delay is honored by Bing and some bots but ignored by Google.
What does robots.txt do?
Robots.txt gives compliant crawlers request rules for a host, such as disallowing private app paths and pointing to a sitemap. It does not secure content and does not reliably remove a public URL that has already been indexed.
Should I block AI crawlers in robots.txt?
Allow retrieval crawlers when you want AI search systems to be able to fetch public pages for answers. Keep private paths disallowed for every crawler. Blocking a named crawler is a product and data-use decision, not a general SEO improvement.
Related free tools
get the full growth toolkit
buyer-intent listening, ai-visibility tracking, and more — free to start.