Robots.txt Generator
Build a valid robots.txt file for your site.
How to use the Robots.txt Generator
- Pick whether to allow or block crawlers.
- Add any paths to disallow and your sitemap URL.
- Copy the file and upload it to your site root as robots.txt.
About the Robots.txt Generator
The Robots.txt Generator builds a valid robots.txt file: a User-agent line, the Disallow rules for the access mode you choose (allow everything, block the whole site, or allow except for the paths you list), and optional Crawl-delay and Sitemap lines. Paths are normalised to start with a slash. You copy the result and upload it to the root of your domain, where crawlers look for it at /robots.txt.
It is the standard way to keep search engines out of admin pages, carts, internal search results and other URLs you would rather not have crawled. It is generated in your browser from the choices you make.
The crucial thing to understand: robots.txt is a polite request, not a lock. It asks well-behaved crawlers not to crawl the listed paths — but it does not make a page private or secure. Malicious bots simply ignore it, and a disallowed page can still appear in search results if other sites link to it. For genuine privacy, use a noindex tag or password protection, not robots.txt. Note too that Google ignores the Crawl-delay directive. Once your important pages are crawlable, list them with the sitemap generator and set page-level tags with the meta tag generator.
Frequently asked questions
Does robots.txt make a page private?
No. It only asks compliant crawlers not to fetch a path. It does not hide, secure or password-protect anything, and well-behaved bots are the only ones that obey it. For real privacy, use authentication or a noindex directive.
Can a disallowed page still appear in Google?
Yes. If other sites link to a URL you have disallowed, it can still be indexed and shown, sometimes without a description. To keep a page out of results entirely, use a noindex tag rather than robots.txt.
Does Crawl-delay work on Google?
No. Google ignores the Crawl-delay directive and manages its own crawl rate. Some other crawlers do respect it, which is why the tool still offers the option.
Where do I put the file?
At the root of your domain, reachable at /robots.txt — for example, example.com/robots.txt. Crawlers only look there; a robots.txt placed in a subfolder is not used.
Will blocking a path remove it from search?
Not reliably. Blocking crawling can actually stop Google from seeing a noindex tag on that page. To remove a page, allow it to be crawled and add a noindex directive, or use the search engine’s removal tools.