Robots.txt Audit Complete
Your Robots.txt Analysis is Complete
We've analyzed avlonshikshaniketan.com for crawl rules, blocked paths, sitemap references and AI crawler policies.
Preview
- At least one user-agent has Disallow: / which blocks the entire site.
- Sitemap URL appears to be unreachable: https://avlonshikshaniketan.com/sitemap.xml
Get your full report + exact fixes
See what's hurting your SEO and how to fix it step by step.
Robots.txt Status
Robots.txt Status
Present
Score
84
/100
ยท Strong
View Full Robots.txt
Robots.txt Content Preview
User-Agent: * Allow: / Disallow: /admin/ Disallow: /api/ User-Agent: GPTBot User-Agent: ChatGPT-User User-Agent: Google-Extended User-Agent: CCBot User-Agent: anthropic-ai User-Agent: ClaudeBot User-Agent: Bytespider User-Agent: Diffbot User-Agent: FacebookBot User-Agent: Applebot-Extended User-Agent: PerplexityBot User-Agent: Omgilibot User-Agent: Amazonbot Disallow: / Sitemap: https://avlonshikshaniketan.com/sitemap.xml
User-agent Rules
| User-agent(s) | Allowed paths | Disallowed paths |
|---|---|---|
| * |
|
|
| gptbot | No explicit Allow rules. | No explicit Disallow rules. |
| chatgpt-user | No explicit Allow rules. | No explicit Disallow rules. |
| google-extended | No explicit Allow rules. | No explicit Disallow rules. |
| ccbot | No explicit Allow rules. | No explicit Disallow rules. |
| anthropic-ai | No explicit Allow rules. | No explicit Disallow rules. |
| claudebot | No explicit Allow rules. | No explicit Disallow rules. |
| bytespider | No explicit Allow rules. | No explicit Disallow rules. |
| diffbot | No explicit Allow rules. | No explicit Disallow rules. |
| facebookbot | No explicit Allow rules. | No explicit Disallow rules. |
| applebot-extended | No explicit Allow rules. | No explicit Disallow rules. |
| perplexitybot | No explicit Allow rules. | No explicit Disallow rules. |
| omgilibot | No explicit Allow rules. | No explicit Disallow rules. |
| amazonbot | No explicit Allow rules. |
|
Blocked and Allowed Paths
| Blocked paths |
|
|---|---|
| Allowed paths |
|
| Crawl-delay | No Crawl-delay directive detected. |
Sitemaps Detected
AI Crawler Policy
No explicit blocks were detected for common AI crawlers (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended).
Issues Found
- At least one user-agent has Disallow: / which blocks the entire site.
- Sitemap URL appears to be unreachable: https://avlonshikshaniketan.com/sitemap.xml
Recommendations
- Avoid blocking the entire site (Disallow: /); restrict only sensitive or low-value paths instead.
- Document your AI crawler policy explicitly in robots.txt so future bots know how to treat your content.
- Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.
Recommended Next Steps
Continue analyzing avlonshikshaniketan.com with related SEO tools and uncover additional opportunities.