User-agent: *
Allow: /

# /chat is the signed-out app route, already noindex. It is disallowed because
# it accepts a pre-composed question as `?q=` — every library page links to it
# that way — and answering one costs a real model call. A crawler following
# those links is indistinguishable from a visitor asking, which is how ~45,000
# pages each became a link that spends money. Turnstile is what actually stops
# it (a crawler spoofing a browser UA ignores this file); this is the polite
# half, for the crawlers that do read it.
Disallow: /chat
Disallow: /library/*/e/2
Disallow: /es/library/*/e/2

# Retired editions (/library/<country>/<region>/<doc>/e/<edition>/…, 2026-10-18,
# operator decision): readers open them from the "Earlier editions" links;
# crawlers have no reason to. They are noindex and in no sitemap (0 Google
# impressions in 90 days), but their `follow` sidebars led bots through
# ~633,000 old pages, each a Vercel render and database reads (2026-10-12:
# load1 41). `*` matches "/" here, so the rule is "/e/" + a year: every retired
# edition starts with "2" and no current page path has "e/<digit>" (checked
# 2026-10-18; 52 current paths do use an `e` segment, e.g. IRS p4118sp, so a
# bare "/e/" rule would block real pages). Every group below repeats it.

# ⚠️⚠️ A NAMED GROUP REPLACES THE `*` GROUP FOR THAT BOT — IT DOES NOT ADD TO IT.
# robots.txt matching picks the ONE most specific group, so a bot listed below
# stops seeing everything above, `Disallow: /chat` included. That is why each of
# these repeats it rather than saying only `Allow: /`. Omitting it would hand the
# crawlers that follow links most aggressively the ~45,000 `?q=` links that each
# cost a real model call — the exact incident the rule above exists to prevent.
#
# These three are already allowed by the `*` group; they are named so that the
# allowance is explicit and survives a future tightening of `*`. OpenAI's own
# publisher guidance is that a site opted out of OAI-SearchBot cannot appear in
# ChatGPT search at all, and measured 2026-09-20, visitors arriving from an LLM
# assistant sign up at roughly 9x the rate of Google organic — so being readable
# by these is worth stating on purpose rather than inheriting by accident.

# Powers ChatGPT search results. Blocking it removes us from ChatGPT entirely.
User-agent: OAI-SearchBot
Allow: /
Disallow: /chat
Disallow: /library/*/e/2
Disallow: /es/library/*/e/2

# Fetches a page when a ChatGPT user asks it to open one.
User-agent: ChatGPT-User
Allow: /
Disallow: /chat
Disallow: /library/*/e/2
Disallow: /es/library/*/e/2

# Model training. Allowed deliberately: the corpus is public law, and a model
# that has read it is likelier to cite us for it.
User-agent: GPTBot
Allow: /
Disallow: /chat
Disallow: /library/*/e/2
Disallow: /es/library/*/e/2

# Keep search engines and user-requested AI fetchers readable. This explicit
# group protects that policy if the default group becomes stricter later.
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
Disallow: /chat
Disallow: /library/*/e/2
Disallow: /es/library/*/e/2

# SEO-tool crawlers: shut out entirely (2026-09-30). They build backlink
# indexes for Ahrefs/Semrush/SE Ranking customers and send no visitors. Every
# page they fetch that is not in Vercel's ISR cache is a fresh render and a
# billed ISR write, and the cache is wiped on every production deploy. Measured
# 09-27..09-30: AhrefsBot alone caused 385,468 cache misses (26% of all misses,
# up from ~0 the week before) while ISR writes ran 8-12M units/day (~$35-50).
# All of these state that they obey robots.txt. One group, several User-agent
# lines (RFC 9309) — it replaces `*` for these bots, so Disallow: / covers /chat.
User-agent: AhrefsBot
User-agent: AhrefsSiteAudit
User-agent: SemrushBot
User-agent: SemrushBot-SA
User-agent: SERankingBacklinksBot
User-agent: MJ12bot
User-agent: DotBot
User-agent: DataForSeoBot
User-agent: BLEXBot
User-agent: Barkrowler
Disallow: /

# These AI crawlers caused sustained library SSR requests on 2026-10-12.
# The Worker also enforces this block before an R2 miss reaches Vercel.
User-agent: ShapBot
User-agent: Reflectionbot
Disallow: /

Sitemap: https://gocodebook.com/sitemap-index.xml
Sitemap: https://gocodebook.com/blog-sitemap.xml
Sitemap: https://gocodebook.com/library/sitemap-index.xml