Skip to main content
AmPuH Sign in
Switch toolrobots.txt builder

Tags and markup

Meta tag generator Meta tag analyser JSON-LD generator Structured data validator Social card proof Hreflang generator

Structure and content

Slug generator Heading outline checker Image alt auditor Link and anchor auditor Duplicate URL finder

Files and rules

robots.txt builder robots.txt tester Robots directive decoder Sitemap XML generator Sitemap validator Redirect rule generator Security header builder

Tool 12

robots.txt builder

Assemble crawl rules, and understand the one thing about Disallow that most guides get wrong.

These generators run in your browser, so they need JavaScript enabled. Everything explained on this page is still readable without it.

Nothing is sent anywhere. Reload to clear.

Disallow does not remove a page from Google

It asks a crawler not to fetch the page. A page nobody may fetch can still be listed from links pointing at it, and Google cannot read a noindex tag on a page it is forbidden to open, so blocking a page you want gone is what keeps it there. Use noindex for that, from the meta tag generator, and leave the page crawlable.

Serve at /robots.txt 
Add a rule or a sitemap to build the file.

Where this file goes

Your own hosting: a real file at the domain root. It has to answer at exactly /robots.txt, because a crawler looks nowhere else.

Blogger: there is no filesystem, so it goes into Settings, then Crawlers and indexing, then Custom robots.txt.

WordPress: most SEO plugins own this file and will overwrite a real one, so edit it through the plugin rather than by upload.

Also in files and rules

All 18 tools