# vike.io — robots.txt # # ALLOW-LIST, not a block-list: the homepage and /learn/ are crawlable and # nothing else is. This inverts the previous policy, and it was measured rather # than argued (2026-08-23, GA4 + nginx access log + ClickHouse query_log): # # 50,000 consecutive requests meta-externalagent 23,834 (48%) # ClaudeBot 14,504 (29%) # other crawlers ~4,700 # HUMANS 7,501 (15%) # # 30 days of page views /polymarket/wallet/* 50,817 views 0 users 17,308 URLs # /polymarket/market/* 33,295 views 0 users 21,235 URLs # /learn/ 3,214 views 4 users 71 URLs # # 5 months of sessions google/organic: 14 sessions in TOTAL, never # more than 6 in any month. # # So 38,543 per-entity URLs were absorbing the crawl budget and returning no # humans, while the 72 pages that ARE the search play — /learn/, every one # published between 2026-08-05 and 2026-08-17 — waited behind them. Crawl budget # is finite and allocated per site; this spends it on the pages written to earn it. # # ⚠ THE COUNTER-ARGUMENT, ANSWERED. "Disallowing pages kills SEO" is true of # pages that rank. These do not and structurally cannot: they are per-entity and # unbounded (one URL per wallet and per market, growing daily), they are absent # from sitemap.xml, and they carry zero organic entries across the whole of GA4's # history here. What they DO carry is cost — each URL is unique, so the CDN gives # no relief and every crawl hit reaches ClickHouse live. Measured that morning: # ~10 cores of ClickHouse CPU, continuously, serving crawlers. # # ⚠ THE APP IS NOT THE SEARCH SURFACE, deliberately rather than by oversight. # /chart, /perps/*, /spot/*, /polymarket carry this site's actual humans (20, 16, # 9 and 9 users over 60 days) and they arrive DIRECTLY, not from search. # Disallowing them costs no traffic that exists. If that changes, add one Allow # line here and one sitemap entry — do not reopen the tree. # # ⚠ THIS IS NOT `noindex`. robots.txt governs CRAWLING; a URL already in an index # can linger as a bare link for a while. The per-entity pages age out as recrawls # stop. Removing them FASTER needs an `X-Robots-Tag: noindex` header on those # routes, which requires them to stay CRAWLABLE — the opposite of this file. # Do not attempt both at once; they cancel. User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: /$ Allow: /learn/ # ⚠ /data/ and /trader/ are OPENED, and this file's own argument is what makes # that safe: the allow-list exists to stop PER-ENTITY fan-out — 38,543 unbounded # URLs, one per wallet and per market, each a live ClickHouse query. These two # prefixes are the opposite shape. They are a bounded, enumerable set of static # server-rendered pages built from a committed registry, with no database on the # request path, and every one of them is declared in sitemap.xml and /llms.txt. # They are the search play /learn/ is, not the app the `Disallow` is aimed at. # # ⚠ ONE /data/ path does read the database: /data/showcase/.json, the # feed the Free Data API pages' live widgets fetch (2026-10-12). It is NOT # disallowed here, on purpose. The set is bounded (one URL per panel, about a # dozen, unknown names 404 without a read), each answer is one cached local row # (`Cache-Control: public, max-age=60`), a rendering crawler needs it to draw # the widgets it indexes the page with, and it carries `X-Robots-Tag: noindex` # — which a crawler can only read on a URL it is allowed to fetch. A Disallow # here would cancel that header, exactly as the note at the top of this file # warns. Allow: /data/ Allow: /trader/ # ⚠ /docs/ — the DOCUMENTATION site, opened 2026-09-04, and it had been shipping # in the broken state this file already describes twice. It is a separate static # export served under /docs/, so `Allow: /trader/` never reached it: a prefix # match starts at the path root, and /docs/trader/... starts with /docs. Every # one of its pages emits index,follow, it publishes its own machine index at # /docs/llms.txt — and that index was itself unfetchable, the same defect the # note below records for the site's own /llms.txt in August. # # It is the same SHAPE as /data/ and /trader/ and safe for the same reason: a # bounded, enumerable set of static pages built at deploy time from committed # markdown, with no database on the request path. There is no per-entity fan-out # here at all — the whole tree is a few hundred URLs that change when someone # writes one. Allow: /docs/ # ⚠ /polymarket — the ONE app page opened, 2026-10-17, by the rule the note above # gives ("add one Allow line here and one sitemap entry — do not reopen the # tree"). It targets "polymarket leaderboard" / "polymarket analytics". `$` makes # it exactly the dashboard: /polymarket/wallet/* and /polymarket/market/* stay # shut. The leaderboard itself is in the first HTML; the `?partial=1` line is the # rest of the dashboard (its htmx body), which a rendering crawler fetches. It # answers `X-Robots-Tag: noindex`, so it is fetched # for rendering and never indexed as a page of its own. # Since 2026-10-25 the page is `/polymarket/` (every sitemap URL ends in `/`); the # bare `/polymarket` stays allowed because it is the 301 a crawler holding the old # URL has to be able to follow. Allow: /polymarket/$ Allow: /polymarket/?partial=1 Allow: /polymarket$ # ⚠ /polymarket/unusual/ — the SECOND app page opened, 2026-10-26, by the same rule and in the same # three-line shape. It targets "polymarket unusual bets" / "polymarket insider trading tracker". Its body # is one cached read and is in the first HTML; `?partial=1` is the lazy fallback after a failed read # (`X-Robots-Tag: noindex`), and the bare `/polymarket/unusual` is its 301. Allow: /polymarket/unusual/$ Allow: /polymarket/unusual/?partial=1 Allow: /polymarket/unusual$ Allow: /static/ Allow: /sitemap.xml # ⚠ /llms.txt was NOT in this list when the allow-list shipped on 2026-08-23, so # `Disallow: /` blocked it — the one file whose entire job is telling an agent # what to read was the file an agent was told not to read. It is generated from # the same pages sitemap.xml lists (app/routers/learn.py), so the two can never # disagree about what exists. Allow: /llms.txt Allow: /llms-full.txt # The IndexNow key file — public by protocol (app/routers/learn.py, INDEXNOW_KEY). Allow: /84800c6a8497e791f88fd32085148081.txt Allow: /favicon.ico Disallow: / # ── AI-training crawlers ───────────────────────────────────────────────────── # ⚠ A bot obeys ONLY its most-specific User-agent group and ignores `*`, so each # group here is self-contained and repeats nothing from above by accident. # # ⚠ The user-facing SEARCH bots are deliberately absent and must stay absent — # Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User. Those cite vike.io # to somebody who can click. (2026-10-22: OAI-SearchBot, ChatGPT-User and the # training crawlers named next now have the group near the end of this file.) # ⚠ So are the index bots — Googlebot, bingbot, Baiduspider, YandexBot. # Those ARE the SEO: they inherit `*` above and keep the homepage and /learn/. # # ⚠ 2026-10-17: the TRAINING crawlers of the assistants people actually use — # GPTBot (OpenAI), ClaudeBot (Anthropic), Meta-ExternalAgent (Meta AI) — no longer # have a group. From 08-23 they had `Disallow: /`; they now inherit `*`, which is # the bounded static set only (/, /learn/, /data/, /trader/, /docs/, /polymarket/), # never the per-entity pages whose crawl cost ~10 ClickHouse cores, and nginx # still holds every one of them to `vike_bulk_crawlers` (6r/m). The trade is # deliberate: what those models train on is what the assistant knows about Vike # when it does NOT search, and ChatGPT is a measured signup channel. Gemini's # and Apple's training tokens (Google-Extended, Applebot-Extended) were never # named here, so they always inherited `*`. Scrapers with no assistant a reader # uses keep their Disallow-all below. # ── 2026-08-24: the token rotation, and why this group exists ──────────────── # The Meta-ExternalAgent Disallow-all group (08-23 to 10-17) WORKED — that token # made ZERO requests on 08-24. The load came back the next morning under Meta's THIRD token, # `meta-webindexer/1.1`, which walked /polymarket/wallet/* (8,927 hits) and # /polymarket/market/* (6,189) at ~129 req/min — 6,986 requests in the 09h hour # alone, ~8 cores of ClickHouse — and fetched this file ZERO times. Meta runs # three crawlers, each with its own token: meta-externalagent (training), # meta-externalfetcher (a fetch a reader triggered), meta-webindexer (the search # index). Naming one names none of the others. # # ⚠ It is NOT in the Disallow-all list below, and that is the same judgement the # search siblings get: meta-webindexer feeds Meta AI's SEARCH index, so it cites # vike.io to somebody who can click. It gets exactly what `*` gets. The group is # spelled out anyway, self-contained, because of the warning above — a bot that # reads a `Meta-WebIndexer` group ignores `*` ENTIRELY, so an inherited-looking # group with the Allow lines left off would cost it /learn/, the 72 pages that # ARE the search play. meta-externalfetcher is deliberately left to inherit `*`. # # ⚠ AND ROBOTS.TXT IS NOT THE ENFORCEMENT — this one never read it. What stopped # it is /etc/nginx/conf.d/vike-crawler-limit.conf's `$vike_entity_crawler` map # plus the per-entity `location` in sites-enabled/vike, which answer 403 by PATH # FAMILY rather than by token. That is the only shape that survives the next # rotation, and the rotation is no longer hypothetical. User-agent: Meta-WebIndexer Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: /$ Allow: /learn/ # ⚠ Same three prefixes as the `*` group above, repeated because this group # inherits nothing. If one list gains a prefix and the other does not, this bot # loses it — which is the exact failure the warning above describes, and which # `tests/unit/test_robots_and_sitemap_agree.py` now holds shut in both # directions rather than leaving to whoever edits one list. Allow: /data/ Allow: /trader/ Allow: /docs/ Allow: /polymarket/$ Allow: /polymarket/?partial=1 Allow: /polymarket$ Allow: /polymarket/unusual/$ Allow: /polymarket/unusual/?partial=1 Allow: /polymarket/unusual$ Allow: /static/ Allow: /sitemap.xml Allow: /llms.txt Allow: /llms-full.txt Allow: /84800c6a8497e791f88fd32085148081.txt Allow: /favicon.ico Disallow: / # ── Content-Signal (every group that allows crawling) ──────────────────────────────────── # A declaration of how content may be used AFTER it is fetched, per contentsignals.org: # search and ai-input (answering and citing, which is the visibility this allow-list # exists to earn) are yes; ai-train is no. It is a stated preference, not enforcement — # nginx still is, per the notes above — and a parser that does not know the line ignores it. # Change the value here and in ALL THREE groups together (`*`, Meta-WebIndexer, the opened AI agents below). # ── 2026-10-22: the opened AI agents — ONE group (RFC 9309 allows several User-agent lines) ────────────────── # /polymarket/wallet/* and /polymarket/market/* are crawlable for exactly these agents, and for nobody else. This # file is only the request: nginx serves those pages to one of them only when Cloudflare itself verified the request # as that bot (vike-crawler-limit.conf `$vike_ai_open_ua` + `$vike_entity_block`: Cloudflare's own verification, or the # vendor's published addresses for PerplexityBot, Perplexity-User and Claude-User), at a per-operator quota, because a # User-agent string is spoofable (298 of 367 "GPTBot" requests in 4 days were not GPTBot). Review 2026-10-29: to cut a # bot, delete its line here AND its name in that nginx map; tests/unit/test_robots_and_sitemap_agree.py fails if the # two lists differ. Deliberately NOT here: cohere-ai (no published addresses to verify against), Googlebot and bingbot # (search engines, not AI), Meta-WebIndexer and meta-externalfetcher (their own tokens), every unnamed agent. # ⚠ A group inherits nothing from `*`: these Allow lines are `*`'s plus the two entity prefixes, kept equal by the same test. User-agent: GPTBot User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: Meta-ExternalAgent User-agent: Applebot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Claude-User Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: /$ Allow: /learn/ Allow: /data/ Allow: /trader/ Allow: /docs/ Allow: /polymarket/$ Allow: /polymarket/?partial=1 Allow: /polymarket$ Allow: /polymarket/unusual/$ Allow: /polymarket/unusual/?partial=1 Allow: /polymarket/unusual$ Allow: /static/ Allow: /sitemap.xml Allow: /llms.txt Allow: /llms-full.txt Allow: /84800c6a8497e791f88fd32085148081.txt Allow: /favicon.ico Allow: /polymarket/wallet/ Allow: /polymarket/market/ Disallow: / # ── Training-only scrapers with no user-facing product ─────────────────────── User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Omgilibot Disallow: / User-agent: FriendlyCrawler Disallow: / Sitemap: https://vike.io/sitemap.xml # ⚠ A SECOND sitemap, and the split is deliberate rather than untidy. The file above is generated # by this site's own application (app/routers/learn.py) and lists the three docs TREE ROOTS, # because that application cannot see the documentation content directory — it is a separate static # export built from markdown. So for months a few hundred published pages were declared to nobody: # crawlable since 2026-09-04, internally linked, and in no sitemap at all. The format allows any # number of Sitemap lines; each is then owned by the system that actually knows what it publishes. # ⚠ Since 2026-10-19 the second line is the DATA API docs only (/docs/data/ + /docs/ai/data/), # not /docs/sitemap.xml: that file lists every docs page, Trader included, and the Trader docs are # not ready to be pushed to search engines. Add /docs/sitemap-trader.xml here when they are. Sitemap: https://vike.io/docs/sitemap-data-api.xml