Create a correct robots.txt file in seconds. Choose a preset or customize crawl rules, disallow paths, crawl-delay, and your sitemap, then copy a ready-to-use robots.txt that tells search engines exactly how to crawl your site.
How to use the robots.txt generator
Start from a preset (allow all, block all, or sensible WordPress defaults) or build your own rules. Add the paths you want to disallow or allow, an optional crawl-delay, and your sitemap URL. Copy the result and upload it as robots.txt in your site’s root directory, reachable at yourdomain.com/robots.txt.
Frequently asked questions
Where does robots.txt go?
In the root of your domain, so it loads at yourdomain.com/robots.txt. Search engines check this file before crawling.
Will robots.txt hide a page from Google?
No. Disallow stops crawling but does not guarantee a page is removed from search. To keep a page out of the index, use a noindex meta tag instead, and do not block it in robots.txt.
Should I add my sitemap to robots.txt?
Yes. Adding a Sitemap line helps search engines discover all your URLs faster. This generator includes a field for it.