Robots.txt Tester

Inspect how csnexplore.com controls crawler access, blocked paths, sitemap references and AI crawler rules.

Preview

Score: 100

  • Document your AI crawler policy explicitly in robots.txt so future bots know how to treat your content.
  • Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.

Get your full report + exact fixes

See what’s hurting your SEO and how to fix it step by step.

  • Full breakdown
  • Actionable fixes
  • Prioritized next steps

No spam. One email with your report and next steps.

Robots.txt Status Present Score 100 /100 · Strong
Domain csnexplore.com
robots.txt URL https://csnexplore.com/robots.txt
Last analyzed May 28, 2026

View Full Robots.txt

Open robots.txt in a new tab

Download robots.txt

Robots.txt Content Preview

# CSNExplore Robots.txt - Optimized for Traditional & AI Search Engines
# Updated: 2026-05-10
# Domain: csnexplore.com
# Location: Chhatrapati Sambhajinagar (Aurangabad), Maharashtra, India

# ════════════════════════════════════════════════════════════════════════════════
# AI SEARCH ENGINE BOTS - EXPLICITLY ALLOWED (ChatGPT, Claude, Perplexity, Gemini)
# ════════════════════════════════════════════════════════════════════════════════

# OpenAI (ChatGPT, GPT-4, GPT-3.5)
User-agent: ChatGPT-User
Allow: /
Crawl-delay: 0.5

User-agent: GPTBot
Allow: /
Crawl-delay: 0.5

# Anthropic (Claude)
User-agent: Claude-Web
Allow: /
Crawl-delay: 0.5

User-agent: anthropic-ai
Allow: /
Crawl-delay: 0.5

# Google AI (Bard, Gemini)
User-agent: Google-Extended
Allow: /
Crawl-delay: 0.5

User-agent: GoogleOther
Allow: /
Crawl-delay: 0.5

# Perplexity AI
User-agent: PerplexityBot
Allow: /
Crawl-delay: 0.5

# Common Crawl (used by many AI models)
User-agent: CCBot
Allow: /
Crawl-delay: 1

# Cohere AI
User-agent: cohere-ai
Allow: /
Crawl-delay: 0.5

# Meta AI (Facebook/Meta)
User-agent: FacebookBot
Allow: /
Crawl-delay: 1

User-agent: meta-externalagent
Allow: /
Crawl-delay: 0.5

# Apple Intelligence
User-agent: Applebot
Allow: /
Crawl-delay: 0.5

User-agent: Applebot-Extended
Allow: /
Crawl-delay: 0.5

# Diffbot (AI web scraping)
User-agent: Diffbot
Allow: /
Crawl-delay: 1

# ════════════════════════════════════════════════════════════════════════════════
# TRADITIONAL SEARCH ENGINE BOTS
# ════════════════════════════════════════════════════════════════════════════════

# Google
User-agent: Googlebot
Allow: /
Crawl-delay: 0.5

User-agent: Googlebot-Image
Allow: /images/
Allow: /*.webp$
Allow: /*.jpg$
Allow: /*.png$
Crawl-delay: 0.5

User-agent: Googlebot-Video
Allow: /
Crawl-delay: 0.5

# Bing
User-agent: Bingbot
Allow: /
Crawl-delay: 0.5

User-agent: BingPreview
Allow: /
Crawl-delay: 1

# Yahoo
User-agent: Slurp
Allow: /
Crawl-delay: 1

# DuckDuckGo
User-agent: DuckDuckBot
Allow: /
Crawl-delay: 0.5

# Yandex
User-agent: YandexBot
Allow: /
Crawl-delay: 1

# Baidu
User-agent: Baiduspider
Allow: /
Crawl-delay: 1

# ════════════════════════════════════════════════════════════════════════════════
# DEFAULT RULES FOR ALL OTHER BOTS
# ════════════════════════════════════════════════════════════════════════════════

User-agent: *
Allow: /
Crawl-delay: 1

# ════════════════════════════════════════════════════════════════════════════════
# BLOCK ADMIN AND SENSITIVE AREAS
# ════════════════════════════════════════════════════════════════════════════════

Disallow: /admin/
Disallow: /vendor/
Disallow: /php/
Disallow: /logs/
Disallow: /cache/
Disallow: /database/
Disallow: /.git/
Disallow: /.env
Disallow: /.htaccess
Disallow: /.kiro/

# ════════════════════════════════════════════════════════════════════════════════
# BLOCK AUTH PAGES (No SEO Value)
# ════════════════════════════════════════════════════════════════════════════════

Disallow: /login
Disallow: /register
Disallow: /forgot-password
Disallow: /reset-password
Disallow: /verify-email
Disallow: /resend-verification
Disallow: /verification-required
Disallow: /my-booking
Disallow: /profile

# ════════════════════════════════════════════════════════════════════════════════
# BLOCK UTILITY SCRIPTS
# ════════════════════════════════════════════════════════════════════════════════

Disallow: /create-admin
Disallow: /create-trip-table
Disallow: /fix-table-now
Disallow: /add-verification-columns
Disallow: /adminexplorer
Disallow: /test-email
Disallow: /send-pending-emails

# ════════════════════════════════════════════════════════════════════════════════
# PREVENT DUPLICATE CONTENT - Block filter/sort parameters
# ════════════════════════════════════════════════════════════════════════════════

Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*?page=
Disallow: /*?*&sort=
Disallow: /*?*&filter=
Disallow: /*?*&page=

# ════════════════════════════════════════════════════════════════════════════════
# ALLOW IMPORTANT CONTENT FOR ALL BOTS (Including AI)
# ════════════════════════════════════════════════════════════════════════════════

Allow: /images/
Allow: /blogs/
Allow: /listing-detail/
Allow: /listing/
Allow: /sitemap.xml
Allow: /css/
Allow: /js/
Allow: /*.css$
Allow: /*.js$
Allow: /*.webp$
Allow: /*.jpg$
Allow: /*.png$
Allow: /*.svg$
Allow: /about
Allow: /contact
Allow: /index

# ════════════════════════════════════════════════════════════════════════════════
# SEO TOOLS - RATE LIMITED
# ════════════════════════════════════════════════════════════════════════════════

User-agent: AhrefsBot
Allow: /
Crawl-delay: 10

User-agent: SemrushBot
Allow: /
Crawl-delay: 10

User-agent: MJ12bot
Allow: /
Crawl-delay: 10

User-agent: DotBot
Allow: /
Crawl-delay: 10

User-agent: Screaming Frog SEO Spider
Allow: /
Crawl-delay: 5

# ════════════════════════════════════════════════════════════════════════════════
# SITEMAPS - FOR ALL SEARCH ENGINES (Traditional & AI)
# ════════════════════════════════════════════════════════════════════════════════

Sitemap: https://csnexplore.com/sitemap.xml

# ════════════════════════════════════════════════════════════════════════════════
# NOTES FOR AI SEARCH ENGINES
# ════════════════════════════════════════════════════════════════════════════════
# This website provides comprehensive tourism information for Chhatrapati
# Sambhajinagar (Aurangabad), Maharashtra, India. Content includes:
# - Hotel and accommodation listings (500+)
# - Car and bike rental services (200+)
# - Tourist attractions (Ajanta Caves, Ellora Caves, Bibi Ka Maqbara)
# - Restaurant recommendations (300+)
# - Bus routes and schedules (100+)
# - Travel guides and blog articles (300+)
#
# All content is structured with schema.org markup for easy parsing.
# Contact: supportcsnexplore@gmail.com | +91-8600968888
# Website: https://csnexplore.com

User-agent Rules

User-agent(s) Allowed paths Disallowed paths
chatgpt-user
  • /
 No explicit Disallow rules.
gptbot
  • /
 No explicit Disallow rules.
claude-web
  • /
 No explicit Disallow rules.
anthropic-ai
  • /
 No explicit Disallow rules.
google-extended
  • /
 No explicit Disallow rules.
googleother
  • /
 No explicit Disallow rules.
perplexitybot
  • /
 No explicit Disallow rules.
ccbot
  • /
 No explicit Disallow rules.
cohere-ai
  • /
 No explicit Disallow rules.
facebookbot
  • /
 No explicit Disallow rules.
meta-externalagent
  • /
 No explicit Disallow rules.
applebot
  • /
 No explicit Disallow rules.
applebot-extended
  • /
 No explicit Disallow rules.
diffbot
  • /
 No explicit Disallow rules.
googlebot
  • /
 No explicit Disallow rules.
googlebot-image
  • /images/
  • /*.webp$
  • /*.jpg$
  • /*.png$
 No explicit Disallow rules.
googlebot-video
  • /
 No explicit Disallow rules.
bingbot
  • /
 No explicit Disallow rules.
bingpreview
  • /
 No explicit Disallow rules.
slurp
  • /
 No explicit Disallow rules.
duckduckbot
  • /
 No explicit Disallow rules.
yandexbot
  • /
 No explicit Disallow rules.
baiduspider
  • /
 No explicit Disallow rules.
*
  • /
  • /images/
  • /blogs/
  • /listing-detail/
  • /listing/
  • /sitemap.xml
  • /css/
  • /js/
  • /*.css$
  • /*.js$
  • /*.webp$
  • /*.jpg$
  • /*.png$
  • /*.svg$
  • /about
  • /contact
  • /index
  • /admin/
  • /vendor/
  • /php/
  • /logs/
  • /cache/
  • /database/
  • /.git/
  • /.env
  • /.htaccess
  • /.kiro/
  • /login
  • /register
  • /forgot-password
  • /reset-password
  • /verify-email
  • /resend-verification
  • /verification-required
  • /my-booking
  • /profile
  • /create-admin
  • /create-trip-table
  • /fix-table-now
  • /add-verification-columns
  • /adminexplorer
  • /test-email
  • /send-pending-emails
  • /*?sort=
  • /*?filter=
  • /*?page=
  • /*?*&sort=
  • /*?*&filter=
  • /*?*&page=
ahrefsbot
  • /
 No explicit Disallow rules.
semrushbot
  • /
 No explicit Disallow rules.
mj12bot
  • /
 No explicit Disallow rules.
dotbot
  • /
 No explicit Disallow rules.
screaming frog seo spider
  • /
 No explicit Disallow rules.

Blocked and Allowed Paths

Blocked paths
  • /admin/
  • /vendor/
  • /php/
  • /logs/
  • /cache/
  • /database/
  • /.git/
  • /.env
  • /.htaccess
  • /.kiro/
  • /login
  • /register
  • /forgot-password
  • /reset-password
  • /verify-email
  • /resend-verification
  • /verification-required
  • /my-booking
  • /profile
  • /create-admin
  • /create-trip-table
  • /fix-table-now
  • /add-verification-columns
  • /adminexplorer
  • /test-email
  • /send-pending-emails
  • /*?sort=
  • /*?filter=
  • /*?page=
  • /*?*&sort=
  • /*?*&filter=
  • /*?*&page=
Allowed paths
  • /
  • /images/
  • /*.webp$
  • /*.jpg$
  • /*.png$
  • /blogs/
  • /listing-detail/
  • /listing/
  • /sitemap.xml
  • /css/
  • /js/
  • /*.css$
  • /*.js$
  • /*.svg$
  • /about
  • /contact
  • /index
Crawl-delay 10.0 seconds

Sitemaps Detected

AI Crawler Policy

No explicit blocks were detected for common AI crawlers (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended).

Recommendations

  • Document your AI crawler policy explicitly in robots.txt so future bots know how to treat your content.
  • Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.

Analyze this site with other tools

Want a website that actually generates leads?

Start a conversion-focused website project with a team that builds fast, SEO-optimized sites for real businesses.

Start project