🤖 robots.txt Generator

🎯 Preset

Apply a starting point for a popular CMS or framework, or pick Custom (empty) to start from a blank skeleton.

👥 User-agent groups

{{ groups.length }} group(s)

Each group declares one or more user-agent tokens and the rules that apply to them. Use * for the wildcard. Paths should start with /; * and $ are supported as glob/anchor.

Group {{ gIdx + 1 }}

{{ ua }}
No allow rules.
  • {{ rule }}
  • {{ rule === '' ? '(empty — allow all)' : rule }}

🗺️ Sitemap URLs

Emit Sitemap: lines at the bottom of the file. RFC 9309 requires absolute URLs.

📝 Header comment (optional)

📤 robots.txt

{{ lineCount }} line(s). Save as robots.txt at the root of your domain.

🔍 Test a URL against the rules

Paste a URL (or path) and a user-agent. The tester simulates the longest-match rule (Allow wins on tie).

{{ testResult.allowed ? 'ALLOWED' : 'BLOCKED' }} — {{ testResult.path }} for user-agent {{ testResult.userAgent }}
{{ testResult.reason }}

⚠️ Warnings

{{ warn.severity }} {{ warn.location }}
{{ warn.message }}
⚠️

ℹ️ About robots.txt Generator

What a robots.txt file does

A robots.txt file is a plain-text file served at the root of a domain (https://example.com/robots.txt) that tells well-behaved crawlers which URLs on the host they may fetch. It implements the Robots Exclusion Protocol, originally defined in 1994 and standardized by the IETF in RFC 9309 in September 2022. The protocol is voluntary: robots.txt expresses intent, not enforcement. Honest crawlers (Googlebot, Bingbot, Slurp, DuckDuckBot, AhrefsBot, GPTBot, ClaudeBot, etc.) read the file before fetching URLs; malicious scrapers ignore it.

This tool builds a robots.txt file from a structured editor (user-agent groups, allow/disallow rule lists, optional crawl-delay, sitemap URLs, free-form comment) and serialises it in canonical form. It also parses an existing robots.txt back into the structured input, ships curated presets for popular CMS and framework stacks, and warns about common configuration mistakes such as the "block everyone" pattern, missing leading slashes, non-portable crawl-delay, and non-absolute sitemap URLs.

Grammar at a glance (RFC 9309 §2.2)

  • Group — one or more User-agent: lines followed by zero or more Allow: / Disallow: lines. A blank line ends the group.
  • User-agent — case-insensitive token. * is the wildcard; specific tokens match the user-agent product token exactly.
  • Allow / Disallow — URL path beginning with /. Glob-style * matches any sequence of characters; $ at the end anchors to end-of-URL. An empty Disallow: value is the idiomatic way to permit everything.
  • Sitemap — full absolute URL (Sitemap: https://example.com/sitemap.xml). Not tied to any specific group; conventionally placed after the rule blocks.
  • Crawl-delay — non-standard extension, not defined by RFC 9309. Bing, Yandex, Baidu, and others honor it; Google ignores it and exposes its own rate control through Google Search Console.
  • Comments — start with # and run to end of line. Inline comments are allowed.
  • Case — directive names are case-insensitive; values (paths) preserve case.

Matching rules (Google + RFC 9309 §2.2.4)

  • Crawlers pick the most specific user-agent group — an exact product-token match wins over the * wildcard.
  • Within the matched group, the longest matching path wins. If an Allow rule and a Disallow rule both match a URL with the same length, Allow takes precedence.
  • If no rule in the matched group matches a URL, the URL is allowed by default.
  • Allow: with an empty path is treated as no rule (it never matches).
  • Disallow: with an empty path is the idiom for "no restrictions in this group".

Common mistakes

  • Forgetting the leading slash — Disallow: admin/ matches nothing. This tool auto-prefixes / and warns.
  • User-agent: *  Disallow: / on production — blocks the entire site from indexing. Acceptable for staging; catastrophic on the live site. The tool emits a high-severity warning when this combination appears.
  • Relative sitemap URLs — Sitemap: /sitemap.xml is invalid. RFC 9309 requires absolute URLs.
  • Using noindex in robots.txt — Google dropped support in 2019. Use the X-Robots-Tag header or <meta name="robots"> in HTML instead.
  • Blocking CSS / JS — Googlebot needs to fetch render-blocking assets to render the page; blocking /css/ or /js/ can hurt ranking. Most modern CMS presets allow them.
  • Trusting robots.txt for security — it is publicly readable. Disclosing /admin/ or /secret-staging/ in robots.txt actively advertises those paths. Use HTTP authentication or IP allow-lists for real privacy.

Use cases

  • Author a new robots.txt — start from a preset (WordPress, Drupal, Magento, Next.js, etc.) or a blank skeleton and tune the rule list.
  • Audit an existing robots.txt — paste the contents of https://example.com/robots.txt back into the parser to see how it would be modelled and whether any rules are auto-corrected.
  • Block AI training crawlers — the "Block AI scrapers" preset opts out the largest LLM training fleets (GPTBot, CCBot, ClaudeBot, anthropic-ai, Google-Extended, PerplexityBot, Bytespider, Applebot-Extended). Compliance is voluntary.
  • CI lint — call the API endpoint from a pipeline to fail the build when a high-severity warning appears (e.g. User-agent: * + Disallow: / introduced on the production branch).
  • Reference sitemap files — declare one or more Sitemap: https://... lines so search engines discover the XML sitemap without manual submission.

Practical baseline

# robots.txt — generated per RFC 9309 (Robots Exclusion Protocol).
# Reference: https://www.rfc-editor.org/rfc/rfc9309.html

User-agent: *
Disallow: /admin/
Disallow: /tmp/
Disallow: /search

Sitemap: https://example.com/sitemap.xml

A single User-agent: * group covers every crawler. Specific bot groups (e.g. User-agent: GPTBot) are matched before the wildcard. End the file with one or more absolute Sitemap: URLs so search engines discover them automatically.

Security rating

  • Status — RFC 9309 (September 2022) is the current IETF standard. Earlier behavior followed the 1994 informal specification with vendor extensions.
  • Strength — robots.txt is an advisory mechanism for cooperative crawlers. It is not an access-control mechanism: any client can ignore it and fetch the URLs anyway.
  • Recommended usage — every production website. Combine with X-Robots-Tag headers / <meta name="robots"> for page-level indexing control, and with HTTP authentication or IP filters for real access control.
  • Known weaknesses — voluntary compliance, publicly readable (exposes paths you might consider sensitive), and the Crawl-delay extension is not portable across crawlers.
  • Industry guidance — Google Search Central's robots.txt documentation, Bing Webmaster Tools robots.txt reference, and the original robotstxt.org informational page.

References

API Usage Example:

POST /api/tools/security/robots-txt-generator
Content-Type: application/json

{
    "operation": "generate",
    "input": {
        "groups": [
            {
                "userAgents": [
                    "*"
                ],
                "allow": [
                    "/public/"
                ],
                "disallow": [
                    "/admin/",
                    "/tmp/"
                ]
            }
        ],
        "sitemaps": [
            "https://example.com/sitemap.xml"
        ]
    }
}
Ready