Robots.txt Tester

Inspect how web.facebook.com controls crawler access, blocked paths, sitemap references and AI crawler rules.

Preview

Score: 84

  • At least one user-agent has Disallow: / which blocks the entire site.
  • Sitemap URL appears to be unreachable: https://web.facebook.com/sitemap/cpg_offplat_future_public_events_sitemap.xml.gz

Get your full report + exact fixes

See what’s hurting your SEO and how to fix it step by step.

  • Full breakdown
  • Actionable fixes
  • Prioritized next steps

No spam. One email with your report and next steps.

Robots.txt Status Present Score 84 /100 · Strong
Domain web.facebook.com
robots.txt URL https://web.facebook.com/robots.txt
Last analyzed June 18, 2026

View Full Robots.txt

Open robots.txt in a new tab

Download robots.txt

Robots.txt Content Preview

# Notice: Collection of data on Facebook through automated means is
# prohibited unless you have express written permission from Facebook
# and may only be conducted for the limited purpose contained in said
# permission.
# All authorized user-agents listed on this page must comply with Meta’s
# Automated Data Collection Terms available at:
# https://www.facebook.com/legal/automated_data_collection_terms
User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Brightbot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: uptimerobot
Disallow: /

User-agent: viberbot
Disallow: /

User-agent: YaK
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: Yeti
Disallow: /

User-agent: Applebot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Bingbot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Discordbot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: DuckDuckBot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: facebookexternalhit
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Googlebot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Googlebot-Image
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /map_tile.php
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /static_map.php
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Googlebot-Video
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Googlebot-News
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Google-InspectionTool
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: LinkedInBot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: msnbot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Pinterestbot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Screaming Frog SEO Spider
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: seznambot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Slurp
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: teoma
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: TelegramBot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: Twitterbot
Disallow: /*/plugins/*
Disallow: /?*next=
Disallow: /a/bz?
Disallow: /ajax/
Disallow: /album.php
Disallow: /business/*&categories
Disallow: /business/*?categories
Disallow: /checkpoint/
Disallow: /contact_importer/
Disallow: /dialog/
Disallow: /fbml/ajax/dialog/
Disallow: /feeds/
Disallow: /file_download.php
Disallow: /job_application/
Disallow: /l.php
Disallow: /login.php*&next=
Disallow: /login.php/?next=
Disallow: /login.php?next=
Disallow: /login/*&next=
Disallow: /login/?next=
Disallow: /login/device-based/regular/login/*&next=
Disallow: /login/device-based/regular/login/?next=
Disallow: /meta-ai-answers/
Disallow: /moments_app/
Disallow: /p.php
Disallow: /photos.php
Disallow: /plugins/
Disallow: /share.php
Disallow: /share/
Disallow: /sharer.php
Disallow: /sharer/
Disallow: /tr/
Disallow: /tr?
Disallow: /ufi/reaction/profile/browser/
Disallow: /x/oauth/
Allow: /ajax/bootloader-endpoint/
Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet
Allow: /careers/
Allow: /safetycheck/

User-agent: *
Disallow: /

Sitemap: https://web.facebook.com/sitemap/cpg_offplat_future_public_events_sitemap.xml.gz
Sitemap: https://web.facebook.com/sitemap/public_active_groups_sitemap.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_popular_posts_sitemap_qe.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_latest_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_latest_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_latest_engaged_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_latest_engaged_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_gqs_rand_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_gqs_select_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_gd_latest_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_gqs_keyword_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_phq_bf_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_phq_bf_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_llm_bf_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_qa_page_bf_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_matcha_keyword_bf_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_phq_med_bf_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_phq_med_bf_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_llm_title_bf_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_p_green_a_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_p_green_b_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_p_green_c_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_p_green_c_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_phq_10_a_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_phq_10_a_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_phq_10_b_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_phq_10_b_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pvd_bf_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pvd_bf_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pvd_bf_posts_2.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pvd_bf_posts_3.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_gqs_inc_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_p_qa_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_popular_posts_sitemap.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_ifr_groups_sitemap.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_ifr_video_posts_sitemap.xml.gz
Sitemap: https://web.facebook.com/sitemap/profiles_2500_followers.xml.gz
Sitemap: https://web.facebook.com/sitemap/profiles_150_followers.xml.gz
Sitemap: https://web.facebook.com/sitemap/profiles_no_vanity_2500_followers.xml.gz
Sitemap: https://web.facebook.com/sitemap/profiles_no_vanity_150_followers.xml.gz
Sitemap: https://web.facebook.com/sitemap/local_business_profiles_150_followers.xml.gz
Sitemap: https://web.facebook.com/sitemap/profiles_no_vanity_p_2500_followers.xml.gz
Sitemap: https://web.facebook.com/sitemap/profiles_no_vanity_p_150_followers.xml.gz
Sitemap: https://web.facebook.com/sitemap/group_aggregation_pages_sitemap.xml.gz
Sitemap: https://web.facebook.com/sitemap/www_facebook_com_business_sitemap.xml.gz
Sitemap: https://web.facebook.com/sitemap/www_facebook_com_help_sitemap.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_p_qa_10_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_p_green_a_2_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_p_green_a_2_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_p_green_b_2_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_p_green_b_2_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_p_green_c_2_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_e_ya_visits_2_a_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_e_ya_visits_2_b_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_e_ya_visits_2_c_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_target_boosting_regular_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_target_boosting_topical_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_target_boosting_with_cp_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_target_boosting_with_firehose_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_target_boosting_with_firehose_and_cp_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_tg_bst_1_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_tg_bst_pq_3_plus_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_tg_bst_pq_3_plus_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_tg_bst_pq_3_plus_posts_2.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_tg_bst_pq_3_plus_posts_3.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_tg_bst_pq_3_plus_posts_4.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_tg_bst_pq_3_plus_posts_5.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_e_ya_visits_3_a_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_e_ya_visits_3_b_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_e_ya_visits_3_c_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_e_ya_visits_3_d_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_e_ya_visits_3_e_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_e_ya_visits_3_f_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_e_ya_visits_3_g_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_e_ya_visits_3_a_2_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_e_ya_visits_3_b_2_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_e_ya_visits_3_c_2_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_cp_replacement_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_p_qa_10_eng_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_query_boost_tpc_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_query_boost_pst_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_new_posts_pq_3_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_new_posts_pq_3_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_new_posts_pq_3_posts_2.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_new_posts_pq_3_posts_3.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_new_posts_pq_3_posts_4.xml.gz
Sitemap: https://web.facebook.com/sitemap/fb_main_profiles_url_sitemap.xml.gz
Sitemap: https://web.facebook.com/sitemap/fb_promode_profiles_url_sitemap.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_hvc_inventory_pq4_plus_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pq3_plus_spanish_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_fb_answer_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_post_pq2_intent_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pq2_humor_intent_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pplus_pquestion_cqm_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pplus_pquestion_cqm_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_fbpost_p_green_a_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_fbpost_p_green_b_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_fbpost_p_green_c_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_fbpost_p_green_a_2_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_comment_summary_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_fbpost_pq_3_plus_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_i18n_pq_3_plus_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_i18n_pq_3_plus_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_group_pq3_inc_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_group_pq3_inc_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_2.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_3.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_4.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_5.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_6.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_7.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_8.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_9.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_10.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_11.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_12.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_13.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_14.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_15.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_inc_posts_16.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pqv3_groups_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pqv3_profiles_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_boosting_sitemap_exp_1_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_boosting_sitemap_exp_2_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_ya_visits_5_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_ya_visits_6_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_v2_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_v2_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_v2_posts_2.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_v2_posts_3.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_v2_posts_4.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_v2_posts_5.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_v2_posts_6.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_v2_posts_7.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_v2_posts_8.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pq3_v2_posts_9.xml.gz
Sitemap: https://web.facebook.com/sitemap/fb_seo_sitemap_videos_results.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pq2_info_opinion_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_i18n_pq3_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_i18n_pq3_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_eya_media2_a_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_eya_media2_b_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_eya_media2_c_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_groups_llm_title_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_public_pqv3_v1_0_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pq2_exp_sharing_2026_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pqv3_groups_launch_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pqv3_profiles_launch_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pqv3_profiles_launch_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pqv3_profiles_launch_posts_2.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pq3_plus_inc_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pq3_plus_inc_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pq3_plus_inc_posts_2.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pq3_plus_inc_posts_3.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_pq3_plus_inc_posts_4.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_deindexed_public_p1_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_deindexed_public_p1_posts_1.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_deindexed_public_p2_posts_0.xml.gz
Sitemap: https://web.facebook.com/sitemap/groups_deindexed_public_p2_posts_1.xml.gz

User-agent Rules

User-agent(s) Allowed paths Disallowed paths
amazonbot No explicit Allow rules.
  • /
applebot-extended No explicit Allow rules.
  • /
brightbot No explicit Allow rules.
  • /
claudebot No explicit Allow rules.
  • /
google-extended No explicit Allow rules.
  • /
gptbot No explicit Allow rules.
  • /
perplexitybot No explicit Allow rules.
  • /
petalbot No explicit Allow rules.
  • /
scrapy No explicit Allow rules.
  • /
uptimerobot No explicit Allow rules.
  • /
viberbot No explicit Allow rules.
  • /
yak No explicit Allow rules.
  • /
yandex No explicit Allow rules.
  • /
yeti No explicit Allow rules.
  • /
applebot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
bingbot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
discordbot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
duckduckbot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
facebookexternalhit
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
googlebot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
googlebot-image
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /map_tile.php
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /static_map.php
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
googlebot-video
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
googlebot-news
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
google-inspectiontool
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
linkedinbot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
msnbot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
pinterestbot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
screaming frog seo spider
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
seznambot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
slurp
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
teoma
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
telegrambot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
twitterbot
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
* No explicit Allow rules.
  • /

Blocked and Allowed Paths

Blocked paths
  • /
  • /*/plugins/*
  • /?*next=
  • /a/bz?
  • /ajax/
  • /album.php
  • /business/*&categories
  • /business/*?categories
  • /checkpoint/
  • /contact_importer/
  • /dialog/
  • /fbml/ajax/dialog/
  • /feeds/
  • /file_download.php
  • /job_application/
  • /l.php
  • /login.php*&next=
  • /login.php/?next=
  • /login.php?next=
  • /login/*&next=
  • /login/?next=
  • /login/device-based/regular/login/*&next=
  • /login/device-based/regular/login/?next=
  • /meta-ai-answers/
  • /moments_app/
  • /p.php
  • /photos.php
  • /plugins/
  • /share.php
  • /share/
  • /sharer.php
  • /sharer/
  • /tr/
  • /tr?
  • /ufi/reaction/profile/browser/
  • /x/oauth/
  • /map_tile.php
  • /static_map.php
Allowed paths
  • /ajax/bootloader-endpoint/
  • /ajax/pagelet/generic.php/pagepostssectionpagelet
  • /careers/
  • /safetycheck/
Crawl-delay No Crawl-delay directive detected.

Sitemaps Detected

AI Crawler Policy

At least one AI crawler (such as GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot or Google-Extended) appears to be blocked by robots.txt.

Issues Found

  • At least one user-agent has Disallow: / which blocks the entire site.
  • Sitemap URL appears to be unreachable: https://web.facebook.com/sitemap/cpg_offplat_future_public_events_sitemap.xml.gz

Recommendations

  • Avoid blocking the entire site (Disallow: /); restrict only sensitive or low-value paths instead.
  • Review your AI crawler policy for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended to ensure it matches your content strategy.
  • Ensure important pages, CSS and JavaScript assets are crawlable so search engines can fully render your site.

Analyze this site with other tools

Want a website that actually generates leads?

Start a conversion-focused website project with a team that builds fast, SEO-optimized sites for real businesses.

Start project