Tools Central

robots.txt Generator and Validator

Create a robots.txt file, detect errors, and simulate access to a URL for a specific bot.

100% local processing: the content and URLs entered never leave your browser.

1. Generate a safe base

One entry per line for paths. The generator adds '/', encodes spaces, and deduplicates rules.

The protocol, domain, and port define the scope of the file.Valid site URL
Use "*" for all robots or a product token like Googlebot.Valid user-agent
Examples: /admin/, /search?, /*.pdf$
A more specific Allow rule can reopen part of a blocked path.

2. Edit and validate

Also paste an existing file: the analysis updates without a network request.

1groups4rules1sitemaps139bytes

3. Test a URL

The simulator applies exact User-agent groups, the most specific rule, "*" and "$" according to RFC 9309.

Crawling allowedNo applicable rule: crawling is allowed by default.

Three points to check before publication

Exact scope

Place the file at the root. Its rules only apply to the same protocol, host, and port.

Crawling, not deindexing

A blocked URL may still appear in results. To prevent indexing, use protection or noindex on an accessible page.

Not a privacy tool

The file is public and reveals the listed paths. Never use it as access control for sensitive data.

Back to SEO tools

Similar Tools

XML Sitemap Generator and ValidatorCreate a URL sitemap or an index, validate the XML, dates, limits, and scope, then download the file locally.
Hreflang Generator and CheckerCreate HTML, HTTP, or sitemap annotations, then locally check self-references, canonicals, and backlinks between multilingual pages.
Structured Data ExtractorExtract JSON-LD, Microdata, and RDFa from HTML locally, explore entities and relationships, then export a bounded JSON report.
HTML Head AuditorLocally audit title, description, canonical, robots, viewport, charset, hreflang, Open Graph, Twitter, and JSON-LD from pasted HTML.
Google Snippet PreviewTest a title, meta description, and URL on desktop and mobile with an estimated width in pixels.
PDF Fields → JSONExport the list of fields from a PDF form (AcroForm) to JSON format.

Create and manage a reliable robots.txt file

The robots.txt file tells crawling bots which paths they can request. The generator prepares a clean base, the validator detects common errors, and the simulator explains the rule applied to a URL.

Use Cases

  • Prepare a new siteCreate a User-agent group, block paths that add no crawling value, and declare an absolute sitemap URL.
  • Audit an existing filePaste its content to identify invalid directives, duplicates, malformed paths, accidental HTML, or a size greater than 500 KiB.
  • Diagnose a blocked URLTest a URL with a specific product token to identify the selected group and the most specific Allow or Disallow rule.

Recommended Method

First, define the exact origin of the site. A file served over HTTPS does not govern HTTP, a different subdomain, or another port. It must be accessible at /robots.txt.

Start with few rules and explicit paths. When multiple rules match, the most specific one wins; in case of equal specificity, Allow prevails over Disallow.

Test important bots and a few representative URLs, publish the file in UTF-8 with the text/plain type, and then check the HTTP response from the outside.

Limits to Know

  • Blocking crawling does not always prevent a URL from appearing in a search result. A URL known through links may remain indexed without a snippet.
  • robots.txt is public and does not protect any data. Use authentication and server permissions for sensitive content.
  • The standard defines the common foundation, but each bot may support additional directives. Google notably ignores Crawl-delay.

Frequently Asked Questions

What is the difference between Allow and Disallow?
Disallow prohibits requests for the matching paths. Allow creates a more specific exception in an otherwise blocked space. An empty value blocks nothing.
Do uppercase letters and URL parameters matter?
Yes. Path comparison is case-sensitive and may include the query string. The characters * and $ serve as wildcard and end marker, respectively.
Are my data sent or recorded?
No. Generation, analysis, simulation, and export of the file are done locally in the browser.

Before blocking a path

Ensure that the path does not contain resources necessary for rendering pages, such as stylesheets or scripts. A short rule can have a much broader effect than expected.