Robots.txt Generator

Build a valid robots.txt with allow/disallow rules, crawl-delay and sitemap. Free, in your browser.

robots.txt

Place the file at your site root (/robots.txt). Remember: robots.txt guides crawlers but doesn't enforce privacy.

Free to use — premium coming soon

FREE
  • Allow/disallow rules
  • Sitemap & crawl-delay
  • Copy & download
PREMIUM
  • Remove ads
  • Per-bot rules & templates

About the Robots.txt Generator

A robots.txt file is a plain-text instruction sheet that lives at the root of your domain (for example example.com/robots.txt) and tells web crawlers which parts of your site they may or may not request. This generator builds that file for you: pick the user-agents, list the paths you want to block or permit, add your sitemap URL, and copy the finished output. It is the implementation of the long-standing Robots Exclusion Protocol, the same convention search engines like Google and Bing have followed for decades, now formalised as an internet standard (RFC 9309).

Use this tool when you are launching a site, restructuring URLs, or want to keep crawlers out of low-value areas such as internal search results, faceted-filter URLs, cart and checkout paths, staging directories, or admin folders. A clean robots.txt helps search engines spend their crawl budget on pages that matter and stops thin or duplicate URLs from clogging the crawl queue. It is not an SEO magic button, but on large sites it can meaningfully reduce wasted crawling. For a brand-new small site you may need only a Sitemap line and an open policy.

Each rule group starts with a User-agent line, followed by one or more Disallow and Allow directives whose values are URL paths matched from the left. Disallow: /private/ blocks that folder and everything under it; Allow can carve out an exception inside a blocked path; an empty Disallow value (Disallow:) blocks nothing. Field names are case-insensitive but the path values are case-sensitive, and the longest, most specific matching rule wins. The Sitemap directive takes a full absolute URL and can appear multiple times. Note that Google ignores crawl-delay, while Bing and Yandex still honour it, so this generator labels that directive as optional and non-universal.

Important: robots.txt is a request, not a lock. Compliant crawlers obey it, but the file is publicly readable by anyone and badly behaved bots can ignore it entirely. Because it merely lists paths, it can actually advertise the sensitive folders you hoped to hide. It is therefore not a security control and not a reliable way to keep a page out of search results, since a disallowed URL can still be indexed if other sites link to it. To truly remove a page from search use a noindex tag or X-Robots-Tag header (the page must stay crawlable for that to be seen), and protect private data with passwords or server-side access controls. This tool runs entirely in your browser, so nothing you type is uploaded or stored.

Frequently asked questions

Where does the robots.txt file need to be placed?

It must live in the root directory of the host it applies to and be reachable at https://yourdomain.com/robots.txt. A file in a subfolder or with a different name is ignored, and each subdomain (and the http vs https version) needs its own file.

Does Disallow stop a page from showing up in Google?

Not reliably. Disallow only asks crawlers not to fetch the URL, but Google can still index that URL if other pages link to it, sometimes showing it with little or no description. To actually keep a page out of results, use a noindex meta tag or X-Robots-Tag and leave the page crawlable, or require a password.

Is crawl-delay supported by search engines?

It is supported by some, not all. Google ignores crawl-delay entirely (adjust crawl rate in Search Console instead), while Bing and Yandex still respect it. Because it was never part of the original standard, treat it as optional and crawler-specific.

Can I use robots.txt to protect private or admin pages?

No. robots.txt is publicly viewable, so listing /admin/ or /api/ in it can broadcast exactly which paths exist. Polite bots obey it but malicious ones do not. Use authentication, IP restrictions, or server-side access control for anything that must stay private.

What happens if my site has no robots.txt file at all?

A missing file (returning a 404) is treated as fully allowed, so crawlers may access every public URL. That is fine for many small sites. Add a robots.txt when you want to block specific paths, declare your sitemap, or manage crawl traffic on a larger site.

From our blog

Which Meta Tags Actually Matter in 2026 (and Which to Stop Using)

By the Super Simple Digital Tools Team · Updated June 2026

Meta tags have quietly narrowed over the years. A decade ago a page head was stuffed with keywords, author tags, and revisit-after directives; in 2026 the list that earns its place is short. The four that carry real weight are the title, the meta description, the robots directive, and the viewport tag, supported by a canonical link and, for anything shareable, Open Graph and Twitter Card tags. Generating them correctly is less about cleverness and more about staying inside the display limits search engines enforce and giving each crawler an unambiguous instruction.

The title tag is the single most influential on-page element. It is a genuine ranking factor, it forms the clickable headline in results, and it is increasingly cited by AI answer engines. Keep it close to 50 to 60 characters because Google truncates around 580 to 600 pixels on desktop, and lead with the term you most want to be found for. The meta description is not a ranking factor, but it is your best lever on click-through rate: write 150 to 160 characters of specific, active copy, and front-load the message within the first 120 characters so it survives on mobile.

The robots tag is where pages get accidentally hidden or wrongly exposed. The default, index,follow, rarely needs stating, but you should explicitly set noindex,nofollow on pages that add no search value, login screens, internal search results, duplicate filters, or staging URLs. Pair this thinking with a canonical link on pages that have near-duplicate versions; relying on Google to auto-pick the right one is a gamble, and an explicit canonical removes the ambiguity. The viewport tag, meanwhile, is non-negotiable: mobile usability has been a ranking input since 2015.

Two tags deserve a clear verdict. The meta keywords tag is dead, ignored by Google since at least 2009, and there is no upside to including it. Open Graph and Twitter Card tags, by contrast, are very much alive but live outside SEO: they decide whether a shared link shows a crisp title, description, and image or an empty grey box. If your robots tag sets noindex, be aware some social crawlers read that as a signal to suppress the preview entirely, so keep the two concerns in mind together.

Order matters more than most people expect. Put the charset declaration first so it lands within the first 1024 bytes, then viewport, title, description, canonical, then the Open Graph and Twitter blocks. Generating the tags from a single set of inputs keeps them consistent, your social title should not drift from your search title without reason, and removes the small syntax mistakes, a missing attribute or a stray quote, that quietly break previews. Once you have the markup, the real work is per-page wording, not the boilerplate.

  • Write a unique title and description for every page; duplicated meta data across pages dilutes relevance and confuses search results.
  • Front-load the keyword in the title and the key message in the first 120 characters of the description so nothing important is cut on mobile.
  • Use noindex,nofollow for thank-you, login, and internal-search pages, but never accidentally leave it on a page you want found.
  • Always add Open Graph and Twitter Card tags to pages you expect to be shared, including a 1200x630 preview image, so links render with a proper card.

Read the full guide →

Tool by the Super Simple Digital Tools Team. Reviewed by our editorial team. Free to use, no signup required.

Related tools