robots.txt Generator
Build a valid robots.txt in seconds.
User-agent group 1
robots.txt
User-agent: * Disallow: /admin Disallow: /private Allow: / Sitemap: https://example.com/sitemap.xml
About this tool
Assemble a correct robots.txt with per-user-agent allow and disallow rules, an optional crawl-delay, and your sitemap URL. Copy or download the result.
How to use
- Add user-agent groups (or use * for all crawlers).
- Add Allow and Disallow paths, and an optional crawl-delay.
- Add your sitemap URL, then copy or download robots.txt.
Examples
Output
User-agent: * Disallow: /admin Allow: / Sitemap: https://example.com/sitemap.xml
Frequently asked questions
At the root of your domain: https://example.com/robots.txt.
No. robots.txt is a crawler hint, not access control. Sensitive pages should be protected server-side.