# As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a Content-Signal = yes, you may collect content for the corresponding # use. # (b) If a Content-Signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a Content-Signal for a # corresponding use, the website operator neither grants nor restricts # permission via Content-Signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search does not # include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of content for # generative AI search answers). # ai-train: training or fine-tuning AI models. # use: how AI systems may consume the content (immediate, reference, or full). # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # BEGIN Cloudflare Managed content User-agent: * Content-Signal: search=yes,ai-train=no,use=reference Allow: / # Training crawlers User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: omgili Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: MistralAI-Training Disallow: / User-agent: meta-externalagent Disallow: / User-agent: GoogleOther Disallow: / User-agent: Baiduspider Disallow: / User-agent: PetalBot Disallow: / User-agent: AwarioSmartBot User-agent: AwarioRssBot Disallow: / User-agent: Google-CloudVertexBot Disallow: / User-agent: QualifiedBot Disallow: / User-agent: Cotoyogi Disallow: / User-agent: ICC-Crawler Disallow: / User-agent: atlassian-bot Disallow: / User-agent: FishBot Disallow: / User-agent: BorderxBot Disallow: / User-agent: NavuBot Disallow: / User-agent: SemrushBot-SWA Disallow: / User-agent: WARDBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: KimiBot Disallow: / User-agent: CitibotSiteCrawler Disallow: / # END Cloudflare Managed Content User-agent: * Allow: / Disallow: /admin # Matchup Lab deep links are a combinatorial space: every head-to-head row on # every athlete page renders a "Compare" link, which is 261,053 reachable # /matchup/-vs- URLs against ~10,300 real content URLs in the sitemap. # None of them are declared, so Googlebot was spending its crawl budget on a # 25:1 majority of pages we never asked it to index -- and the sitemap's own # athlete pages sat in "Discovered - currently not indexed" behind them. # The pattern requires "-vs-", so /matchup itself (the pair-picker) stays # crawlable. Human visitors are unaffected; the links still work. Disallow: /matchup/*-vs-* # /embed/athletes/ is a framable rendering of a profile that is ALREADY in # the sitemap -- ~9,100 duplicate URLs carrying no fact the canonical page lacks. # Each one serves noindex + rel=canonical, but Googlebot has to spend a crawl to # READ that noindex, which is the same crawl-budget problem the matchup rule # above exists to solve (8,215 athlete pages sat "Discovered - currently not # indexed" behind exactly that kind of majority). Disallowing costs nothing here: # nothing links to these but the copy-paste snippet on the profile itself, and # they hold no content of their own to lose. Framing is unaffected -- robots.txt # governs crawlers, not browsers. Disallow: /embed/ # The three parameterised families below were 699 URLs in "Alternate page with # proper canonical tag" on 2026-08-17, and that bucket is not a defect: they DO # self-canonicalise to their bare path, correctly. Canonicalising correctly and # wasting crawl budget are separate questions, and the 2026-08-08 read only # answered the first -- so the leak survived the matchup fix and put the ratio of # undeclared to real URLs back above 1:1 (~13,000 vs ~10,300). None of the three # is in the sitemap, and none renders a fact its bare path doesn't already carry, # which is the test each had to pass before being blocked. # reportLink() (functions/_shared.js) renders a crawlable "Report an error" link # on EVERY athlete and EVERY contest page: 9,128 + 1,177 = ~10,305 URLs, 1:1 with # the entire real content set, all canonicalising to /report. 584 of the 699 above # (84%) were these, and only 584 of the 10,305 had been crawled -- so the bucket # was still growing toward five figures while 7,630 real athlete pages sat in # "Discovered - currently not indexed", never fetched. # The trailing "?" is load-bearing: bare /report is the actual form page and stays # crawlable, exactly as /matchup survives the -vs- rule above. Reporting is # unaffected -- every link still works, and /report remains the route for a # visitor to send us a correction, which the project requires. Disallow: /report? # ?division= on a contest page selects which division shows an "In focus" chip. # It reveals NOTHING: renderContestPage maps over every division regardless, so # the bare /contests/ already carries every division's full field. ~2,601 # contest-division links means ~2,601 of these against 1,177 real contest URLs. # Same shape as the old GROUP_VISIBLE bug -- a variant that changed a page's # appearance and nothing a crawler can see. Disallow: /contests/*?division= # ?cell=- opens a drill-down panel on the /scorecards coverage # matrix, ~154 cells. The panel only LISTS shows, and every contest it lists is # already in the sitemap at its own /contests/, so blocking this costs no # reachability. Deliberately narrow: /scorecards?contest=&division= is a real # server-rendered card and one of ~300 genuine content URLs, so it must stay # crawlable, and this pattern cannot match it. Disallow: /scorecards?cell= # CSV exports, blocked 2026-09-26. This is the SAME leak the /report? rule above # documents, missed at the time: renderCiteTools() (functions/_shared.js) emits a # real crawlable anchor -- -- inside the # "Cite / download" panel on EVERY athlete page and every contest page that has # results. That is 39,925 + 1,404 = ~41,300 URLs against ~10,300 real content URLs, # a 4:1 majority, none of them in the sitemap. Each one is a Pages Function run # plus Supabase queries that return, in CSV form, exactly the table the canonical # page already rendered -- no fact a crawler cannot already read at /athletes/ # or /contests/. # Downloading is UNAFFECTED. robots.txt governs crawlers, not people: the button # still works, the file still generates, and /api/v1/* (the other machine-readable # route) is untouched and stays open. This only stops crawlers spending fetches on # a second copy of pages they have already read. Disallow: /athletes/*/career.csv Disallow: /contests/*/results.csv # Presentation view, blocked 2026-09-26. Same shape as /embed/ above, but weaker # on its own behalf: every contest page with a readable card renders an "Open # presentation view" link (the present-cta block in functions/contests/[slug].js), # giving 2,980 reachable /present// URLs -- one per # contest_division holding two or more scored placings. It is a full-screen, # large-type re-rendering of the scorecard already shown on /contests/, built # for screen-sharing and screenshots. # Unlike /embed/, which serves noindex + rel=canonical via _shared.embedDoc and was # STILL disallowed because reading that noindex costs a crawl, /present/ declares # NEITHER -- so to a crawler it looks like original content rather than a duplicate. # That makes it the stronger candidate of the two, not the weaker one. # Human use is unaffected: the link still opens, the view still works. Disallow: /present/ # SEO-tool crawlers, blocked outright 2026-09-21. They index the site for # third-party backlink/keyword products and send no readers back, but every page # here is server-rendered with ~0% edge cache, so each hit is a Pages Function # run plus Supabase queries. On 2026-09-20 AhrefsBot (9,761), SemrushBot (4,332) # and DotBot (829) were ~40% of all requests, against ~230 real browser page # loads. Search engines and AI assistants (Googlebot, bingbot, GPTBot, etc.) are # NOT in this list -- they do send readers, and they stay under the * rules above. # A bot named here ignores the * group entirely, which is fine: it's a full block. User-agent: AhrefsBot User-agent: AhrefsSiteAudit User-agent: SemrushBot User-agent: SiteAuditBot User-agent: DotBot User-agent: rogerbot User-agent: MJ12bot User-agent: BLEXBot User-agent: DataForSeoBot User-agent: serpstatbot Disallow: / Sitemap: https://calloutiq.com/sitemap.xml