mtools.lol

Robots.txt Generator

Create a robots.txt file that controls what search engines and AI crawlers may visit.

How it works

  1. 1Choose whether crawlers may visit the site.
  2. 2List paths to block, exceptions and sitemaps.
  3. 3Download robots.txt and put it at your site root.

Features

  • Disallow and allow rules.
  • Sitemap lines.
  • One click to block known AI training crawlers.
  • Paths validated.

Frequently asked questions

Where does the file go?

At the root of your domain: https://example.com/robots.txt.

Does robots.txt hide pages?

No. It asks well-behaved crawlers not to visit; blocked pages can still be indexed if linked elsewhere. Use noindex to keep a page out of search.

Is anything sent?

No. It runs in your browser.

Related tools

Sitemap GeneratorTurn a list of page URLs into a valid XML sitemap for Google and Bing.Website Metadata CheckerCheck a page's SEO and sharing metadata: title, description, canonical, headings, structured data and more.Open Graph PreviewSee how a link will look when shared on Facebook, LinkedIn, X, Slack and WhatsApp, and which tags are missing.Barcode GeneratorMake EAN, UPC, Code 128 and other barcodes as PNG or SVG.