X-Robots Header Generator
Generate an X-Robots-Tag HTTP header with noindex, nofollow, and other directives, with optional user-agent targeting.
Directives
Leave blank to apply the directives to all crawlers
Header:
X-Robots-Tag: noindex
How to Use the X-Robots Header Generator
Tick the indexing directives you want to apply — such as noindex or nofollow — and optionally target a specific crawler by name (e.g. googlebot). The tool assembles the resulting X-Robots-Tag HTTP response header for you to add to your server or CDN configuration for the resource in question.
Example
Checking noindex and nofollow with no target user-agent produces:X-Robots-Tag: noindex, nofollow
Common Use Cases
- Blocking a PDF, image, or other non-HTML file from being indexed, since it can't carry a meta robots tag.
- Applying different indexing rules to a specific crawler (e.g. blocking one bot but not others).
- Deindexing a whole class of resources (like generated thumbnails) at the server-config level instead of editing every file.
FAQs
- How is this different from the Robots.txt Generator? Robots.txt is a text file that controls CRAWLING at the site or path level, and it can only stop a crawler from fetching a URL — it can't deindex a URL search engines already know about. X-Robots-Tag is an HTTP response HEADER applied per-resource that gives page-level INDEXING control, and it's the only option for non-HTML files like PDFs or images, which can't carry a meta robots tag at all.
- What does the "none" directive do? It's shorthand equivalent to combining
noindexandnofollowin a single directive. - When should I target a specific user-agent? Use it when you want different crawlers to receive different instructions for the same resource — for example, allowing a general search engine to index a page while blocking a specific bot by name.