Robots.txt Tester

Inspect how schutzit.com controls crawler access, blocked paths, sitemap references and AI crawler rules.

Preview

Score: 100

Document your AI crawler policy explicitly in robots.txt so future bots know how to treat your content.
Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.

Get your full report + exact fixes

See what’s hurting your SEO and how to fix it step by step.

Full breakdown
Actionable fixes
Prioritized next steps

Robots.txt Status Present Score 100 /100 · Strong

Domain schutzit.com

robots.txt URL https://schutzit.com/robots.txt

Last analyzed June 2, 2026

View Full Robots.txt

Open robots.txt in a new tab

Download robots.txt

Robots.txt Content Preview

# robots.txt for Schutz IT
# Main domain: https://schutzit.com

User-agent: Googlebot
Allow: /
Disallow: /admin/
Disallow: /private/
Allow: /pki
Allow: /pki$
Allow: /pki/*
Allow: /ciam
Allow: /ciam$
Crawl-delay: 1
Sitemap: https://schutzit.com/sitemap.xml
Sitemap: https://schutzit.com/sitemap-index.xml

User-agent: Bingbot
Allow: /
Crawl-delay: 1
Sitemap: https://schutzit.com/sitemap.xml
Sitemap: https://schutzit.com/sitemap-index.xml

User-agent: Twitterbot
Allow: /
Sitemap: https://schutzit.com/sitemap.xml

User-agent: facebookexternalhit
Allow: /
Sitemap: https://schutzit.com/sitemap.xml

# AI crawlers / LLM training & retrieval bots
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Bytespider
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: DuckAssistBot
Allow: /


User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Sitemap: https://schutzit.com/sitemap.xml
Sitemap: https://schutzit.com/sitemap-index.xml

User-agent Rules

User-agent(s)	Allowed paths	Disallowed paths
googlebot	/ /pki /pki$ /pki/* /ciam /ciam$	/admin/ /private/
bingbot	/	No explicit Disallow rules.
twitterbot	/	No explicit Disallow rules.
facebookexternalhit	/	No explicit Disallow rules.
gptbot	/	No explicit Disallow rules.
oai-searchbot	/	No explicit Disallow rules.
chatgpt-user	/	No explicit Disallow rules.
claudebot	/	No explicit Disallow rules.
claude-web	/	No explicit Disallow rules.
anthropic-ai	/	No explicit Disallow rules.
perplexitybot	/	No explicit Disallow rules.
perplexity-user	/	No explicit Disallow rules.
google-extended	/	No explicit Disallow rules.
ccbot	/	No explicit Disallow rules.
applebot	/	No explicit Disallow rules.
applebot-extended	/	No explicit Disallow rules.
bytespider	/	No explicit Disallow rules.
meta-externalagent	/	No explicit Disallow rules.
cohere-ai	/	No explicit Disallow rules.
duckassistbot	/	No explicit Disallow rules.
*	/	/admin/ /private/

Blocked and Allowed Paths

Blocked paths	/admin/ /private/
Allowed paths	/ /pki /pki$ /pki/* /ciam /ciam$
Crawl-delay	1.0 seconds

Sitemaps Detected

AI Crawler Policy

No explicit blocks were detected for common AI crawlers (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended).

Recommendations

Document your AI crawler policy explicitly in robots.txt so future bots know how to treat your content.
Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.