Robots.txt Generator — Create robots.txt File Online [2026]

A robots.txt file is a plain text file placed at the root of your website (e.g. https://example.com/robots.txt) that follows the Robots Exclusion Protocol. It tells web crawlers which URLs they are allowed or not allowed to request from your site. It is advisory — well-behaved bots respect it, but malicious scrapers may ignore it.

Use Disallow in robots.txt to prevent a page from being crawled (fetched). Use a noindex meta tag or X-Robots-Tag header to allow crawling but prevent indexing. Disallowing a page stops crawlers from seeing it at all — if other sites link to that page, it may still appear in search results without any content snippet.

GPTBot is OpenAI's web crawler used to collect data for training AI models. You can block it by adding User-agent: GPTBot followed by Disallow: /. Whether to block it is a content strategy decision — blocking it prevents your content from being used in AI training but has no effect on your Google search ranking.

The robots.txt file must be placed at the root of your domain: https://yourdomain.com/robots.txt. It cannot be in a subdirectory. If you use a subdomain, each subdomain needs its own robots.txt file.

Blocking Googlebot from crawling pages will prevent those pages from being indexed, which removes them from search results. For most sites, you should only disallow pages you do not want indexed, such as admin panels, staging paths, or duplicate content. Incorrectly blocking your whole site (Disallow: /) will remove it from Google entirely.