Home Tool Robots txt generator

Robots.txt Generator

Create a robots.txt file for your website to control search engine crawling.

Allow all crawlers (most common):

Block all crawlers:

Custom directives:

What is robots.txt?

A robots.txt file tells search engine crawlers which pages or sections of your site to crawl or avoid crawling. It's placed in your website's root directory (e.g., tyzo.in/robots.txt).

✅ Best for: Website owners, SEO specialists, developers.

Common templates

Allow all crawlers (default for most sites)

User-agent: *
Allow: /

Sitemap: https://yourwebsite.com/sitemap.xml

✅ Use this for public websites you want indexed.

Block all crawlers (development/staging sites)

User-agent: *
Disallow: /

⚠️ Use only for development or private sites.

Block specific directories

User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /private/
Allow: /public/

✅ Common for WordPress and admin sections.

Common mistakes to avoid

❌ Accidentally blocking important pages like /blog/ or /products/
❌ Using Disallow incorrectly (missing trailing slashes)
❌ Forgetting that robots.txt doesn't guarantee blocking (respectful crawlers only)
❌ Not adding sitemap location

Best practices

✅ Place robots.txt in your root directory (e.g., https://tyzo.in/robots.txt)
✅ Add your sitemap URL at the bottom
✅ Use Allow before Disallow if needed
✅ Test your robots.txt in Google Search Console

Frequently asked questions

Does robots.txt remove pages from Google index?

No. It only prevents crawling. Pages already indexed may stay. Use noindex meta tags to remove from index.

What's the difference between Disallow and noindex?

Disallow = don't crawl the page. noindex = don't show in search results. For sensitive pages, use both or password protection.

Do I need a robots.txt file?

Not necessarily. If you don't have one, Google will crawl everything. It's only needed if you want to block specific sections.