Robots.txt Tester

Inspect how madcurlsailingcharters.com controls crawler access, blocked paths, sitemap references and AI crawler rules.

Robots.txt Status Present Score 92 /100 · Strong

Domain madcurlsailingcharters.com

robots.txt URL https://madcurlsailingcharters.com/robots.txt

Last analyzed April 5, 2026

View Full Robots.txt

Open robots.txt in a new tab

Download robots.txt

Robots.txt Content Preview

# START YOAST BLOCK
# ---------------------------
User-agent: *
Disallow: /?s=
Disallow: /page/*/?s=
Disallow: /search/
Disallow: /wp-json/
Disallow: /?rest_route=

User-agent: AdsBot
Disallow: /

Sitemap: https://madcurlsailingcharters.com/sitemap_index.xml
# ---------------------------
# END YOAST BLOCK

User-agent Rules

User-agent(s)	Allowed paths	Disallowed paths
*	No explicit Allow rules.	/?s= /page/*/?s= /search/ /wp-json/ /?rest_route=
adsbot	No explicit Allow rules.	/

Blocked and Allowed Paths

Blocked paths	/?s= /page/*/?s= /search/ /wp-json/ /?rest_route= /
Allowed paths	No explicit Allow paths detected.
Crawl-delay	No Crawl-delay directive detected.

Sitemaps Detected

https://madcurlsailingcharters.com/sitemap_index.xml

AI Crawler Policy

No explicit blocks were detected for common AI crawlers (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended).

Issues Found

At least one user-agent has Disallow: / which blocks the entire site.

Recommendations

Avoid blocking the entire site (Disallow: /); restrict only sensitive or low-value paths instead.
Document your AI crawler policy explicitly in robots.txt so future bots know how to treat your content.
Consider adding explicit Allow rules for important sections to clarify crawling intent for complex setups.
Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.