What Is robots.txt?
Understand robots.txt before it accidentally blocks your site from Google—or opens the wrong doors to crawlers.
- No signup required
- Free robots.txt test
- Instant results
What Is robots.txt? (Quick Answer)
- A plain-text file at yourdomain.com/robots.txt
- Tells crawlers which paths they may or may not request
- Uses User-agent, Allow, Disallow, and Sitemap lines
- Does not secure private pages—only guides polite crawlers
- Test yours with the Robots.txt Tester
Example Report Preview
Preview crawl health scores, risky rules, and sitemap checks for any domain.
What a robots.txt Review Covers
A robots.txt review checks how crawlers are allowed to access your site before they waste budget or miss key pages.
-
User-agent groups and wildcard rules
On-page signal reviewed during analysis.
-
Allow and Disallow path patterns
On-page signal reviewed during analysis.
-
Sitemap declarations
On-page signal reviewed during analysis.
-
Rules that block CSS, JS, or entire sections
On-page signal reviewed during analysis.
-
Staging or accidental Disallow: / on production
On-page signal reviewed during analysis.
Why robots.txt Matters for SEO
- Disallow: / for all Disallow: / for all user-agents blocks your whole site from crawling.
- Blocking CSS or JS can Prevent Google from rendering pages correctly.
- Missing Sitemap references Slow down discovery of new content.
- AI crawlers increasingly respect AI crawlers increasingly respect robots.txt as a public access policy.
Key Elements Inside robots.txt
-
User-agent
Identifies which crawler a group of rules applies to (* = all bots).
-
Allow / Disallow
Specify path patterns bots may or may not crawl.
-
Crawl-delay
Optional hint for request frequency (not supported by Google).
-
Sitemap
Points crawlers to your XML sitemap for faster indexing.
Common robots.txt Mistakes
-
Disallow: / on production
Why it hurts Entire site becomes uncrawlable—organic traffic stops.
How to fix it Remove staging rules before launch; test after every deploy.
-
Blocking rendering resources
Why it hurts Google cannot see the page as users do.
How to fix it Allow CSS, JS, and image paths needed for rendering.
See How Crawlers Read Your robots.txt
Enter your domain and visualize what crawlers are allowed to access.
- Fetch live file
- Parse rules per bot
- Flag risky patterns
- Sitemap check
Frequently Asked Questions
Can robots.txt block my site from Google?
Yes. A broad Disallow directive can prevent Googlebot from crawling URLs, which stops new indexing and updates. Note that robots.txt controls crawling, not necessarily removal of already-indexed pages. Staging environments often ship with Disallow: / and accidentally go live that way. Test robots.txt after every migration and CMS change. If you need a page out of search results, use noindex—not robots.txt alone.
Is robots.txt the same as noindex?
No. robots.txt tells crawlers not to fetch certain URLs; noindex (via meta robots or HTTP header) tells search engines not to index a page that was crawled. A page blocked by robots.txt may still appear in results with a snippet from external links if Google knows the URL. Use noindex when you want a page excluded from search but still need crawlers to see other signals. Use robots.txt to manage crawl budget on low-value paths like filtered faceted URLs.
Do AI crawlers follow robots.txt?
Many AI crawlers state that they respect robots.txt as a site-wide access policy, though policies vary by provider. If you want to allow search indexing but limit AI training crawlers, you may need provider-specific user-agent rules. robots.txt is public—anyone can read it—so do not treat it as security. Review rules whenever you add new bot user-agents to your policy.
Is Your robots.txt Blocking Traffic?
Test your crawl rules in seconds and fix issues before they cost you rankings.
Test robots.txt Free