Robots.txt Generator
Write a small, reviewable robots.txt for compliant crawlers. Specify paths and sitemap locations, then publish the downloaded file at the root of the intended origin.
Browser-based · No account required · Copy and export
Workspace
Customize your settings below
robots.txt is public and is not access control or a reliable way to keep URLs out of search results.
How to use Robots.txt Generator
- Select a crawler user-agent and choose the default open or closed policy.
- Enter allow/disallow paths, one per line. Add the first sitemap URL and any additional sitemap URLs.
- Generate and review the rules. Upload as /robots.txt and verify the public response from the intended host.
When to use this tool
Prepare crawler rules and sitemap declarations for a website you manage. Robots directives control crawling, not authentication or guaranteed removal from search results.
Crawling rules do not secure private content
Robots rules request crawler behavior. An Allow or Disallow path starts at the origin root, and a more specific matching rule can override a broader path. Sitemap declarations point to absolute locations. Blocking crawling does not reliably remove an already known URL from search results and does not restrict a human visitor.
Worked example
User-agent: *
Disallow: /admin/
Allow: /admin/help/
Sitemap: https://example.test/sitemap.xmlLimits and output checks
This creates one user-agent group and does not fetch a live robots file or simulate every crawler. User-agent identifiers and paths are checked to prevent accidental directive/comment injection. Use authentication for private resources and appropriate indexing controls for search visibility.
Reference: Robots Exclusion Protocol.
Frequently asked questions
Where should I publish the file?
At /robots.txt on the exact protocol and host it should control. Subdomains can have separate rules.
Can I list several sitemaps?
Yes. Add one absolute HTTP(S) URL per line in Additional sitemap URLs. Duplicate declarations are removed.
Does Disallow guarantee no indexing?
No. It controls fetching by compliant crawlers, not access security or a guarantee that a URL will never appear in search.
Crawl directives are not access control
Choose the user-agent group and supported paths, then inspect every generated rule before placing it in the appropriate source file.
Scope of this workflow
The tool neither retrieves a live robots file nor models every crawler. A disallow rule is not authentication or an automatic removal from search.
XML Sitemap Generator - URL Lists & Validation · All developer tools · Privacy policy · Contact PixelOkay