KitJade
SEO & WebsiteRuns in your browserNew

robots.txt Generator

Write valid crawl rules without syntax mistakes.

Use the robots.txt Generator tool

Runs in your browserUpdated Report a problem

What this robots.txt generator does

A robots.txt file tells crawlers which paths they may request. The generator builds one group per user agent, with allow and disallow lines and an optional sitemap directive. Blank lines and comments are handled for you.

The tool also checks for the classic mistakes: a bare `Disallow: /` that blocks the entire site, a wildcard used where a literal path was meant, and rules that accidentally block CSS or JavaScript needed for rendering.

Why use it

  • Create a correct robots.txt for a new site.
  • Block a staging area without blocking the whole domain.
  • Add the sitemap location in the expected format.

How to use it

  1. 1Choose the user agent or keep the wildcard group.
  2. 2Add allow and disallow paths, one per line.
  3. 3Enter the absolute sitemap URL and copy the file.

Worked examples

Block an admin area

Input

User-agent: * · Disallow: /admin/

Output

A robots.txt that excludes /admin/ and allows everything else.

Sitemap reference

Input

https://example.com/sitemap.xml

Output

Sitemap: https://example.com/sitemap.xml

Common mistakes

  • Using robots.txt to hide private pages; the file is public and does not enforce access control.
  • Blocking /wp-content/ or equivalent asset paths, which breaks rendering for crawlers.
  • Assuming every crawler honours a rule; reputable ones do, malicious scrapers do not.

Frequently asked questions

Does robots.txt remove pages from search results?
No. It controls crawling, not indexing. To keep a page out of results, use a noindex directive and allow crawling so the directive can be seen.
Where should the file live?
At the root of the host, for example https://example.com/robots.txt. Subdirectories are not honoured.
Do I need one?
No. Without a robots.txt, crawlers assume everything is allowed. Add one when you need exclusions or a sitemap reference.

Related tools

All tools

Sitemap Generator

Create a valid XML sitemap from a URL list.

Runs locallyNew

Meta Tag Analyzer

Audit the head of a page for SEO and sharing tags.

Runs locallyNew

Canonical URL Checker

Spot canonical mismatches before they split ranking signals.

Runs locallyNew

SERP Snippet Preview

Preview how a search result will look.

Runs locallyNew