# https://www.robotstxt.org/robotstxt.html

# Allow all crawlers — including AI / answer-engine bots such as GPTBot,
# OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended.
User-agent: *
Allow: /

# 🔴 The private / authenticated areas are DELIBERATELY crawlable.
#
# They used to be listed here as `Disallow`, and Google indexed /dashboard and
# /settings anyway — Search Console reported them under "Indexed, though blocked
# by robots.txt". That is not a contradiction: robots.txt controls CRAWLING, not
# INDEXING. Google can index a URL it has never fetched (from links or
# references alone), and because it was forbidden from fetching the page it
# could never see a `noindex` telling it not to.
#
# Disallowing them was therefore the thing PREVENTING their removal. Each of
# these pages now emits `<meta name="robots" content="noindex, nofollow">` at
# runtime via <SEO noindex /> (see src/pages/{Dashboard,Settings,Profile,Feedback}Page.tsx),
# and they must stay crawlable for Google to read it and drop them.
#
# Do not re-add a Disallow for these routes. They are auth-gated, so a crawler
# sees only the signed-out shell, and the noindex tag is what keeps them out of
# the index.
#   /dashboard  /settings  /profile  /feedback

# Individual family pages. The trailing slash matters: it disallows
# /family/<id>, /family/new and /family/mine while leaving the public /family
# marketing page crawlable. Each family page also emits a runtime noindex —
# NOINDEX_ROUTES in scripts/seo-routes.mjs feeds only the prerender validator
# and emits no tag, so it would not have covered this on its own.
#
# ⚠️ These are left disallowed ON PURPOSE, unlike the four routes above: a
# family page URL contains a user-chosen id and may be shared privately, so
# keeping crawlers out entirely is worth more than being able to deindex one
# that somehow got referenced. Revisit only if GSC starts reporting them as
# "Indexed, though blocked by robots.txt" too.
Disallow: /family/

# Sitemap
Sitemap: https://jagannathahora.com/sitemap.xml
Sitemap: https://jagannathahora.com/sitemap-astrologers.xml