Robots.txt Tester

Inspect how indiamart.com controls crawler access, blocked paths, sitemap references and AI crawler rules.

Preview

Score: 84

At least one user-agent has Disallow: / which blocks the entire site.
robots.txt does not reference any sitemap URLs.

Get your full report + exact fixes

See what’s hurting your SEO and how to fix it step by step.

Full breakdown
Actionable fixes
Prioritized next steps

Robots.txt Status Present Score 84 /100 · Strong

Domain indiamart.com

robots.txt URL https://indiamart.com/robots.txt

Last analyzed May 31, 2026

View Full Robots.txt

Open robots.txt in a new tab

Download robots.txt

Robots.txt Content Preview

User-agent: *

#Disallowing paths
Disallow: /pd/    # For preview snippet generation WhatsApp bot does not honor robots.txt
Disallow: /stats/
Disallow: /cgi/
Disallow: /temp/
Disallow: /company1/
Disallow: /proddetail1
Disallow: /proddetail1/
Disallow: /proddetail2
Disallow: /proddetail2/
Disallow: /mdc/
Disallow: /company/A/
Disallow: /company/B/
Disallow: /company/C/
Disallow: /company/D/
Disallow: /company/E/
Disallow: /company/N/
Disallow: /company/VFCP/
Disallow: /company/EFCP/
Disallow: /company/O/
Disallow: /company/G0/
Disallow: /company/G2/
Disallow: /company/purl/
Disallow: /company/view-catalog.html
Disallow: /easybuy/cmp/
Disallow: /enquiry.html
Disallow: /prod-fcp/cgi/model/
Disallow: /company/bl_overlay.pl
Disallow: /TDWIM/
Disallow: /CWSIM/
Disallow: /eyeblaster/
Disallow: /*/search.html

#Disallowing AI/ML Bots
User-agent: AdIdxbot		#AdIdxbot
Disallow: /
User-agent: Facebookbot		#Facebook bot
Disallow: /
User-agent: Facebot		#Facebook bot
Disallow: /
User-agent: ClaudeBot		#Claude Bot
Disallow: /
User-agent: CriteoBot/0.1	#Criteo Bot
Disallow: /
User-agent: alexabot		#Alexabot
Disallow: /
User-agent: ia_archiver		#Internet Archive
Disallow: /
User-agent: alexa site audit 	#Alexa audit crawler
Disallow: /
User-agent: Bytespider		#ByteDance
Disallow: /
User-agent: omgili		#Webz.io
Disallow: /

#Disallowing Other Bots
User-agent: Dataprovider.com
Disallow: /
User-agent: dcrawl
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: HTTrack 3.0
Disallow: /
User-agent: MetaInspector
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: 008
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: SearchmetricsBot
Disallow: /
User-agent: Feedonomics
Disallow: /
User-agent: EtaoSpider
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: trendictionbot
Disallow: /
User-agent: EyeMonIT Uptime Bot
Disallow: /
User-agent: AhrefsSiteAudit
Disallow: /
User-agent: Mail.RU_Bot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: SputnikBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: FeedBurner
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: proximic
Disallow: /
User-agent: Scrapy
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: coccocbot
Disallow: /
User-agent: IAS Crawler
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: ltx71
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Pingdom.com_bot
Disallow: /
User-agent: GetIntent Crawler
Disallow: /
User-agent: expo9
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: bingbot
Disallow: /api/ajax-services/supplierrating/

#Allowing
User-agent: Mediapartners-Google
Allow: /
User-agent: Adsbot-Google
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: meta-externalagent
Allow: /

User-agent Rules

User-agent(s)	Allowed paths	Disallowed paths
*	No explicit Allow rules.	/pd/ # for preview snippet generation whatsapp bot does not honor robots.txt /stats/ /cgi/ /temp/ /company1/ /proddetail1 /proddetail1/ /proddetail2 /proddetail2/ /mdc/ /company/a/ /company/b/ /company/c/ /company/d/ /company/e/ /company/n/ /company/vfcp/ /company/efcp/ /company/o/ /company/g0/ /company/g2/ /company/purl/ /company/view-catalog.html /easybuy/cmp/ /enquiry.html /prod-fcp/cgi/model/ /company/bl_overlay.pl /tdwim/ /cwsim/ /eyeblaster/ /*/search.html
adidxbot #adidxbot	No explicit Allow rules.	/
facebookbot #facebook bot	No explicit Allow rules.	/
facebot #facebook bot	No explicit Allow rules.	/
claudebot #claude bot	No explicit Allow rules.	/
criteobot/0.1 #criteo bot	No explicit Allow rules.	/
alexabot #alexabot	No explicit Allow rules.	/
ia_archiver #internet archive	No explicit Allow rules.	/
alexa site audit #alexa audit crawler	No explicit Allow rules.	/
bytespider #bytedance	No explicit Allow rules.	/
omgili #webz.io	No explicit Allow rules.	/
dataprovider.com	No explicit Allow rules.	/
dcrawl	No explicit Allow rules.	/
nutch	No explicit Allow rules.	/
httrack	No explicit Allow rules.	/
httrack 3.0	No explicit Allow rules.	/
metainspector	No explicit Allow rules.	/
offline explorer	No explicit Allow rules.	/
008	No explicit Allow rules.	/
slurp	No explicit Allow rules.	/
seznambot	No explicit Allow rules.	/
searchmetricsbot	No explicit Allow rules.	/
feedonomics	No explicit Allow rules.	/
etaospider	No explicit Allow rules.	/
blexbot	No explicit Allow rules.	/
imagesiftbot	No explicit Allow rules.	/
trendictionbot	No explicit Allow rules.	/
eyemonit uptime bot	No explicit Allow rules.	/
ahrefssiteaudit	No explicit Allow rules.	/
mail.ru_bot	No explicit Allow rules.	/
ahrefsbot	No explicit Allow rules.	/
baiduspider	No explicit Allow rules.	/
sputnikbot	No explicit Allow rules.	/
mj12bot	No explicit Allow rules.	/
feedburner	No explicit Allow rules.	/
exabot	No explicit Allow rules.	/
proximic	No explicit Allow rules.	/
scrapy	No explicit Allow rules.	/
semrushbot	No explicit Allow rules.	/
coccocbot	No explicit Allow rules.	/
ias crawler	No explicit Allow rules.	/
dotbot	No explicit Allow rules.	/
ltx71	No explicit Allow rules.	/
sogou web spider	No explicit Allow rules.	/
pingdom.com_bot	No explicit Allow rules.	/
getintent crawler	No explicit Allow rules.	/
expo9	No explicit Allow rules.	/
petalbot	No explicit Allow rules.	/
yandex	No explicit Allow rules.	/
bingbot	No explicit Allow rules.	/api/ajax-services/supplierrating/
mediapartners-google	/	No explicit Disallow rules.
adsbot-google	/	No explicit Disallow rules.
oai-searchbot	/	No explicit Disallow rules.
gptbot	/	No explicit Disallow rules.
chatgpt-user	/	No explicit Disallow rules.
google-extended	/	No explicit Disallow rules.
meta-externalagent	/	No explicit Disallow rules.

Blocked and Allowed Paths

Blocked paths	/pd/ # for preview snippet generation whatsapp bot does not honor robots.txt /stats/ /cgi/ /temp/ /company1/ /proddetail1 /proddetail1/ /proddetail2 /proddetail2/ /mdc/ /company/a/ /company/b/ /company/c/ /company/d/ /company/e/ /company/n/ /company/vfcp/ /company/efcp/ /company/o/ /company/g0/ /company/g2/ /company/purl/ /company/view-catalog.html /easybuy/cmp/ /enquiry.html /prod-fcp/cgi/model/ /company/bl_overlay.pl /tdwim/ /cwsim/ /eyeblaster/ /*/search.html / /api/ajax-services/supplierrating/
Allowed paths	/
Crawl-delay	No Crawl-delay directive detected.

Sitemaps Detected

No Sitemap directives found in robots.txt.

AI Crawler Policy

No explicit blocks were detected for common AI crawlers (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended).

Issues Found

At least one user-agent has Disallow: / which blocks the entire site.
robots.txt does not reference any sitemap URLs.

Recommendations

Add a Sitemap directive in robots.txt pointing to your primary XML sitemap.
Avoid blocking the entire site (Disallow: /); restrict only sensitive or low-value paths instead.
Document your AI crawler policy explicitly in robots.txt so future bots know how to treat your content.
Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.