Robots.txt Generator
Generate a valid robots.txt file with custom allow/disallow rules, crawl delay, and sitemap URL.
Rules
Disallow
Allow
Seconds between crawl requests (leave empty for default)
How to Use the Robots.txt Generator
Enter your website URL, add allow/disallow rules for the paths you want to control, and optionally set a crawl delay and sitemap URL. Click "Generate" to create a valid robots.txt file you can copy and paste into your website root.
Example
To block search engines from crawling your admin pages but allow the rest of your site, add a "Disallow: /admin/" rule and an "Allow: /" rule. The generated output will look like a standard robots.txt file.
FAQs
- Where do I put the robots.txt file? Upload it to the root directory of your website (e.g., https://example.com/robots.txt).
- Does a robots.txt block all crawlers? No — it is a voluntary standard. Legitimate search engine bots respect it, but malicious bots may ignore it.
- What is Crawl-delay? It tells compliant crawlers how many seconds to wait between requests to your site, helping reduce server load.
Common Use Cases
- Blocking search engines from indexing admin or private areas of your site.
- Specifying the location of your sitemap for easier crawling.
- Controlling crawl rate to avoid overloading your server.