Free · No signup · Runs in your browser

Robots.txt Generator & Tester

Generate a correct robots.txt, then verify exactly how crawlers interpret it — longest-match rules included.

Start from a preset 
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Scope of the tester

The tester parses the rules you paste and simulates how a compliant crawler would evaluate the paths you enter. It performs no network requests — nothing is fetched, crawled or uploaded. That makes it safe for testing draft files, staging rules, or hypothetical changes before they go anywhere near production.

Rule evaluation, precisely

  • Group selection. Rules apply per user-agent group. A specific group (Googlebot) fully replaces the * group for that bot; the two are not merged.
  • Longest match wins. Among all matching Allow/Disallow patterns, the one whose path matches the most characters decides the outcome.
  • Allow wins ties. If the longest Allow and the longest Disallow match the same length, the crawler may fetch the URL.
  • Empty Disallow. Disallow: with no path allows everything — commonly used to grant full access after more specific groups.
  • Wildcards & anchors. * matches any sequence of characters; $ marks the end of the path. /cart$ blocks only the exact path /cart, not /cart/item/1.

Frequently asked questions

What does robots.txt actually control?
Crawling, not indexing. A Disallow rule stops compliant crawlers from fetching a URL, but the URL can still appear in search results if other pages link to it. To keep a page out of the index use a noindex robots meta tag on the page itself, and leave robots.txt open so crawlers can see that tag.
How does the tester decide between Allow and Disallow?
It uses the same precedence model as Google: the most specific rule wins, measured by path length. If an Allow and a Disallow match the same path equally long, Allow wins. Wildcards (*) match any character sequence and $ anchors the end of the path, exactly as specified in the robots.txt RFC.
Does this tool crawl my website?
No. The tester evaluates the rules you paste against the paths you type, entirely in your browser. It never requests your site, which means you can safely test drafts of a robots.txt file before publishing it.
Should every site have a robots.txt file?
Most sites benefit from at least a minimal one: pointing to the XML sitemap and keeping clearly non-public paths out of crawl budgets. You are also better off explicitly allowing AI and search crawlers you do want, instead of leaving their access ambiguous.
Where do I upload the generated file?
Place it at https://yourdomain.com/robots.txt — the very root of the domain, named exactly robots.txt in lower case. On WordPress, plugins or hosting panels often expose an editor for it; otherwise upload via FTP/SFTP or your host file manager.

Related free tools

More utilities in the WebDesignKw toolbox.

SERP Snippet Checker

See how your title and description render in search results — desktop and mobile — before you publish.

FAQ Schema Generator

Turn questions and answers into valid FAQPage JSON-LD — reorder, validate, copy or download in seconds.

Put this into practice

Want results instead of numbers?

Our team builds and manages the systems behind these metrics — websites, campaigns and SEO that compound.

Get Started Today
Get a Free Consultation Telegram