User-agent
Identifies which crawler a group of rules applies to.
What Is robots.txt?
Understand how the robots.txt file works, what it can and cannot do, and why it matters for search engines and AI crawlers.
Test robots.txt with Robots.txt Tester
No sign-up required. Runs directly in your browser.
Quick summary
Allow / Disallow
Specify paths that bots may or may not crawl.
Crawl-delay
Optional pacing hint for how frequently bots should request pages.
robots.txt in simple terms
robots.txt is a public text file that lives at the root of your domain, such as https://example.com/robots.txt. It tells crawlers which paths they are allowed to request and which ones they should avoid. It does not secure private data, but it acts as a first gate for crawlers discovering your site.
Key elements inside robots.txt
-
User-agent
Identifies which crawler a group of rules applies to.
-
Allow / Disallow
Specify paths that bots may or may not crawl.
-
Crawl-delay
Optional pacing hint for how frequently bots should request pages.
-
Sitemap
Points crawlers to one or more XML sitemaps for faster discovery.
Common robots.txt Mistakes
- Using Disallow: / for a broad user-agent and unintentionally blocking the entire site.
- Blocking CSS or JavaScript resources that search engines need to render pages correctly.
- Forgetting to reference your XML sitemap, making content discovery slower.
Frequently Asked Questions
Can robots.txt block my site from Google?
Yes. A broad Disallow rule can prevent crawling. Testing helps prevent accidental lockouts.
Is robots.txt the same as noindex?
No. robots.txt controls crawling, while noindex (meta robots) controls whether a page is indexed.
Do AI crawlers follow robots.txt?
Many AI crawlers respect robots.txt as a public access policy, so it is a good place to define preferences.
See how crawlers read your robots.txt
Use the free Robots.txt Tester to fetch, parse and review crawl rules for your domain.
Test robots.txt with Robots.txt Tester