Free tools that run locally in your browser with zero data storage.
Tyzo
Browse Tools

Robots.txt Tester & Validator

Test, validate, and debug your robots.txt file. Check which URLs are allowed or blocked for different search engine bots.

Google

https://www.google.com/robots.txt

Facebook

https://www.facebook.com/robots.txt

Amazon

https://www.amazon.com/robots.txt

Twitter

https://twitter.com/robots.txt

Click any sample to see how major websites configure their robots.txt

Enter your website domain (e.g., https://example.com) - we'll automatically check for robots.txt

Use this if your robots.txt is at a custom location

Perfect for testing changes before uploading or for local development

No robots.txt file found?

If a website returns a 404 error for robots.txt, that means no file exists. This is completely normal! Search engines will crawl all publicly accessible pages by default. Use our Robots.txt Generator to create one if you need to block specific content.

How to Use This Robots.txt Tester

1. Test Your Own Site

Enter your domain (e.g., https://yourdomain.com) and click "Fetch robots.txt" - we'll automatically check for your robots.txt file.

2. Test Changes Before Uploading

Copy and paste your robots.txt content directly to test changes before deploying to production.

3. Test Specific URLs

After parsing, enter any URL to see if it's allowed or blocked by your robots.txt rules.

4. Test Different Bots

Select different user agents (Googlebot, Bingbot, etc.) to see how each bot interprets your rules.

Understanding Your Test Results

ALLOWED (Green)

The URL can be crawled by the selected bot. No matching Disallow rule was found, or an Allow rule overrides a Disallow.

BLOCKED (Red)

The URL cannot be crawled. A matching Disallow rule prevents the bot from accessing this URL.

NOT FOUND (Yellow)

No robots.txt file exists. Search engines will crawl all pages by default. This is normal for many websites.

Robots.txt Syntax Quick Reference

User-agent 
User-agent: *

Applies to all bots

User-agent: Googlebot

Applies only to Google

Disallow 
Disallow: /admin/

Blocks entire /admin/ folder

Disallow: /*?*

Blocks all URLs with parameters

Allow 
Allow: /admin/public/

Allows subdirectory within blocked folder

Sitemap 
Sitemap: https://example.com/sitemap.xml

Specifies sitemap location

Common Robots.txt Issues to Watch For

Disallow: / on Live Site

This blocks ALL search engines from your entire site. Your pages will disappear from Google search results.

Missing robots.txt File

No file = no restrictions. All pages can be crawled. This is fine for most sites but you miss crawl budget optimization.

Blocked CSS/JS Files

Blocking CSS or JavaScript prevents Google from rendering your pages correctly, harming mobile SEO.

No Sitemap Reference

Without a sitemap directive, search engines may miss important pages not linked internally.

You Might Also Like

Robots.txt Generator Meta Title Checker Meta Description Checker Redirect Checker Canonical Tag Checker Complete Robots.txt Guide

Frequently Asked Questions

Why does my website show "No robots.txt file found"?
A 404 error means no robots.txt file exists at that location. This is completely normal and not an error! Search engines will crawl all publicly accessible pages by default. Only create a robots.txt file if you need to block specific content (like admin areas or duplicate pages).
How accurate is this tester?
This tester follows the Google Robots.txt specification including wildcard (*) matching. It's highly accurate for Googlebot, Bingbot, and other major crawlers. However, always verify critical changes using Google Search Console's official robots.txt tester.
Can I test local development sites?
For local development, use "Option 3" and paste your robots.txt content directly into the textarea. The fetch option only works with publicly accessible URLs.
What wildcards does this tester support?
This tester supports the asterisk (*) wildcard which matches any sequence of characters. For example, Disallow: /*?* matches any URL containing a question mark (parameter URLs).
Does every website need a robots.txt file?
No! Many websites don't need a robots.txt file. Search engines will crawl all pages by default. Only create one if you need to: block specific directories, prevent crawling of duplicate content, or reference your sitemap.
Why do Googlebot and Bingbot show different results?
You can set different rules for different user agents. For example, you might allow Googlebot to crawl JavaScript but block it for other bots. The tester applies the correct rules based on the selected user agent.