Robots.txt Tester
Enter a website and see its robots.txt rules, crawl-delay and declared sitemaps.
How to use the Robots.txt Tester
- Enter a website URL or domain.
- Click Test robots.txt.
- See the rules grouped by user-agent, plus any sitemaps.
About the Robots.txt Tester
The Robots.txt Tester fetches a website's /robots.txt and shows you what it actually says: the rules grouped by User-agent, each Allow and Disallow path, any Crawl-delay, and every Sitemap line it declares. Just enter a domain and it reads the live file for you.
This is the quickest way to confirm that a site is not accidentally blocking search engines, to see which sections are disallowed, and to check that your sitemap is actually referenced from robots.txt — a small detail that helps search engines find it. If no robots.txt exists, that is reported too, which by default means the whole site is crawlable. To create or edit the file, use the robots.txt generator, and to count the URLs in a sitemap it points to, the sitemap URL counter.
Frequently asked questions
What does the tester show?
The robots.txt rules grouped by user-agent — the Allow and Disallow paths and any crawl-delay — plus every sitemap the file declares and whether a robots.txt was found at all.
What if a site has no robots.txt?
The tool reports that none was found. With no robots.txt, search engines are generally free to crawl the whole site.
Does Disallow with an empty value block the site?
No — a Disallow line with nothing after it means the opposite: full access is allowed for that user-agent. The tester labels this clearly.
Can I just enter a domain?
Yes. Enter a domain or any page URL and the tool checks the robots.txt at the site root automatically.
Does it change anything on the site?
No. It only reads the public robots.txt file and displays it; it never modifies anything.