# FOTOhub — robots.txt
# https://fotohub.app
#
# ONE rule group, deliberately.
#
# This file used to carry ~45 named groups (Googlebot, Bingbot, GPTBot, ClaudeBot,
# PerplexityBot, …), each of them "User-agent: X / Allow: /". A crawler that finds its
# own name obeys THAT group and ignores "User-agent: *" completely — so every one of
# those 45 bots was being told it may crawl /dashboard, /admin, /auth and
# /reset-password, which are exactly the paths the * group was trying to protect. The
# named groups were the leak, not the protection.
#
# Naming them is also unnecessary. Every crawler is allowed by default; a token only
# has to appear here if its rules should DIFFER from everyone else's, and none of
# these should. Google-Extended and Applebot-Extended in particular are opt-OUT
# controls — absent means allowed, so an "Allow: /" entry for them was a no-op that
# cost us the disallow list.
#
# Explicitly welcome, and covered by the group below: Googlebot, Bingbot, Slurp,
# DuckDuckBot, YandexBot, Baiduspider, Naverbot, PetalBot; GPTBot, OAI-SearchBot,
# ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot,
# Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, Applebot,
# Applebot-Extended, DuckAssistBot, MistralAI-User, Meta-ExternalAgent,
# meta-externalfetcher, Amazonbot, Bytespider, YouBot, cohere-ai, AI2Bot, Diffbot,
# Timpibot, Webzio-Extended, omgili, CCBot; and the preview bots (Twitterbot,
# facebookexternalhit, LinkedInBot, Slackbot, Discordbot, TelegramBot, WhatsApp).
#
# No Crawl-delay. Google ignores it, and the "Crawl-delay: 2" that used to be here
# capped Bing at one request every two seconds — Bing being the index behind several
# assistant search products, that was a self-inflicted throttle on both search and
# AI discovery.
#
# /generate/* is NOT disallowed on purpose. Those routes are login-gated and are
# marked "noindex" by an X-Robots-Tag header instead (see
# autom-aws/frontend/nginx/conf.d/seo-noindex-map.conf). A crawler that is not
# allowed to fetch a URL never sees its noindex, so blocking them here would freeze
# the ones Google has already indexed in place forever.

User-agent: *
Allow: /

# Login-gated or operational — the same prefix list the X-Robots-Tag map uses.
# No trailing slash where the prefix is unambiguous: "Disallow: /dashboard/" leaves
# /dashboard itself crawlable, which is the URL that actually gets linked.
Disallow: /dashboard
Disallow: /settings
Disallow: /admin
Disallow: /modera
Disallow: /my-photos
Disallow: /fh/
Disallow: /auth
Disallow: /callback
Disallow: /reset-password
Disallow: /private
Disallow: /_internal
# /api/ keeps its slash: /api-docs is a public page and must stay crawlable.
Disallow: /api/
# $ anchors the match, so /gallery is blocked and /gallery-apps is not.
Disallow: /gallery$
Disallow: /albums/
# Share links. Both routes carry an unguessable token and render their content only
# after JavaScript runs, so a crawler that executes JS already sees the client-side
# noindex and one that does not would index an empty shell. Neither is a disaster —
# but a share link is something a user pastes into a chat or a ticket, and the moment
# one of those is public the URL is crawlable. Blocking the prefixes costs nothing and
# removes the case where a token reaches an index through someone else's carelessness.
# Only the two SHARE prefixes, not /public/ as a whole, so a future genuinely-public
# page under /public/ stays crawlable by default.
Disallow: /public/video-project/
Disallow: /public/audiocore/

Sitemap: https://fotohub.app/sitemap.xml

# Written for assistants that read the site, and generated from the same registry as
# the sitemap by scripts/generate-llms-txt.mjs:
#   https://fotohub.app/llms.txt        — every public URL, with a description
#   https://fotohub.app/llms-full.txt   — the full copy of the pages that have it