User-agent: * Disallow: /*? Disallow: /s/ # # ── STAGED, do not uncomment yet ───────────────────────────────────────────── # Read the full rationale and the 🔴 sequencing gate at the FOOT of this file # before touching these. They live inside this `User-agent: *` group on purpose: # a second `User-agent: *` group further down would be merged by Google but is # SILENTLY DISCARDED by other parsers (CPython's urllib.robotparser keeps only # the first — "the first default entry wins", verified 2026-07-31). Keep them # here; do not move them next to their explanation. # # Disallow: /actualitate/ # Disallow: /*/actualitate/ # Disallow: /actualite/ # Disallow: /*/actualite/ # Disallow: /advertentie/ # Disallow: /*/advertentie/ # Disallow: /annuaire-professionnel/ # Disallow: /*/annuaire-professionnel/ # Disallow: /annunci-classificati/ # Disallow: /*/annunci-classificati/ # Disallow: /anuario-de-empresas/ # Disallow: /*/anuario-de-empresas/ # Disallow: /anuncios-classificados/ # Disallow: /*/anuncios-classificados/ # Disallow: /anunturi-clasificate/ # Disallow: /*/anunturi-clasificate/ # Disallow: /avisos-clasificados/ # Disallow: /*/avisos-clasificados/ # Disallow: /bedrijvengids/ # Disallow: /*/bedrijvengids/ # Disallow: /branchenverzeichnis/ # Disallow: /*/branchenverzeichnis/ # Disallow: /business-directory/ # Disallow: /*/business-directory/ # Disallow: /classified-ads/ # Disallow: /*/classified-ads/ # Disallow: /directorio-empresa/ # Disallow: /*/directorio-empresa/ # Disallow: /directorul-companiei/ # Disallow: /*/directorul-companiei/ # Disallow: /elenco-aziende/ # Disallow: /*/elenco-aziende/ # Disallow: /evenements/ # Disallow: /*/evenements/ # Disallow: /evenimente/ # Disallow: /*/evenimente/ # Disallow: /eventi/ # Disallow: /*/eventi/ # Disallow: /events/ # Disallow: /*/events/ # Disallow: /katalog-firm/ # Disallow: /*/katalog-firm/ # Disallow: /kleinanzeigen/ # Disallow: /*/kleinanzeigen/ # Disallow: /lieux/ # Disallow: /*/lieux/ # Disallow: /news/ # Disallow: /*/news/ # Disallow: /notizia/ # Disallow: /*/notizia/ # Disallow: /ogloszenia-niejawne/ # Disallow: /*/ogloszenia-niejawne/ # Disallow: /petites-annonces/ # Disallow: /*/petites-annonces/ # Disallow: /places/ # Disallow: /*/places/ # ───────────────────────────────────────────────────────────────────────────── # ───────────────────────────────────────────────────────────────────────────── # Blocked crawlers — added 2026-07-26. # # Measured from one hour of Cloud Run request logs (2026-07-26 21:00-22:00 UTC): # of 6,860 HTML page requests — each of which costs an SSR render — 71% came from # self-declared crawlers. The agents below are blocked because they consume # rendering budget and return nothing: no search traffic, no citations, no # referrals. Pages/hour at the time of measurement: # # Amazonbot 1,784 AhrefsBot 515 SemrushBot 315 SleepBot 219 # PetalBot 147 Bytespider 89 DotBot 83 SpiderLing 77 # YouBot 15 (+706 asset requests/h) Scrapy 22 ShapBot 10 # VelenPublicWebCrawler 7 AffsignalCrawler 5 # ABEvalBot 2 HanaleiBot 1 # # Amazonbot alone was ~43k pages/day — five times Googlebot's 360/h. # # Deliberately NOT blocked: # Googlebot (360/h), bingbot (216/h) search revenue. # Applebot (677/h) heavy, but feeds Siri/Spotlight and # Safari suggestions: real discovery. # ClaudeBot (88/h), ChatGPT-User (85/h), PerplexityBot (39/h), OAI-SearchBot, # Qwantbot (27/h), YandexBot (9/h) answer engines that cite sources and # can refer traffic. # CriteoBot, peer39_crawler ad-tech brand-safety verification, # ~1/h each; blocking them risks # degrading our AdSense inventory. # TikTokSpider 1/h, and it backs link previews. # # Bytespider IS blocked despite being AI: it trains only and never cites. # # The last eight agents below were not seen in the sampled hour — they are # pre-emptive, and all well-known freeloaders. # # To re-measure, pull an hour of request logs and group by user agent: # gcloud logging read 'resource.type="cloud_run_revision" # AND resource.labels.service_name="alltrippers"' --project=alltrippers-eu # ───────────────────────────────────────────────────────────────────────────── # UPDATE 2026-07-31 — Amazon re-tokenised around the block. # # Re-measured over 2026-07-31 07:06-09:00 UTC (20,000 requests, 9,451 of them # HTML page renders). Self-declared crawlers are now 61% of all page renders. # # Amazonbot obeyed the 07-26 block and went to zero — and the same crawl volume # reappeared under a token this file did not name. Requests/day: # # Amazonbot Amzn-SearchBot # 07-25 29,637 112 # 07-26 38,259 39 <- block added # 07-27 6,754 8,064 # 07-28 0 8,542 # 07-29 0 11,159 # 07-30 0 18,997 # # Amzn-SearchBot is now the single largest SSR consumer on the site (~20k # pages/day), ahead of Googlebot (~9.4k/day). Blocking it is therefore recurring # maintenance, not a one-off: expect Amazon to appear under a third token. # # meta-externalagent (~9k pages/day) was never in this file — it simply was not # present in the single hour sampled on 07-26. It is Llama training crawl: no # citations, no referrals. # # Deliberately still NOT blocked, on top of the list above: # facebookexternalhit (~330/day) Facebook/Instagram LINK PREVIEWS. This is a # different token from meta-externalagent and # must stay allowed or shared links lose their # preview cards. Do not merge the two. # meta-externalfetcher user-initiated link fetches, same rationale # as ChatGPT-User. Not seen in the sample. # GoogleOther (~7.9k/day) does not feed Search, so blocking is # documented-safe — but the risk is asymmetric # (Google-branded agent vs. ~6% of renders) # and it honours path rules, so it will follow # the staged Disallow at the foot of this file # instead. Revisit only if that lands and it # is still heavy. # OAI-SearchBot (~1.9k/day) answer engine that cites. # # Ignoring this file entirely as of 07-31 (blocked above, still crawling): # SleepBot ~4,055/day SpiderLing ~2,312/day ShapBot ~354/day # Scrapy ~240/day # robots.txt cannot reach these — they are already listed above and simply do # not honour it. The only lever left is a UA check in server.ts. NOT Cloud # Armor: alltrippers.com and www are custom-domain-mapped straight to Cloud Run # with no load balancer in front (verified 2026-07-31 — zero backend-services, # zero security-policies in the project), and Cloud Armor requires an external # HTTPS LB. Adding one to block ~7k renders/day is not worth it. # ───────────────────────────────────────────────────────────────────────────── # UPDATE 2026-08-06 — Meta re-tokenised around the block, exactly as predicted. # # Every agent blocked on 07-26/07-31 now measures ZERO: Amazonbot 0, # Amzn-SearchBot 0, meta-externalagent 0. The blocks work. What changed is that # the volume came back under names this file did not list. Sample of 2,000 # requests, 2026-08-05 09:00-09:30 UTC: # # meta-webindexer 423 <- NEW, unblocked # GeedoShopProductFinder 391 <- NEW, unblocked # Applebot 86 # bingbot 52 # Googlebot 6 # # Two unblocked agents were ~41% of requests against Googlebot's 6. Cloud Run # requests went ~190k/day -> 689k (08-04) -> 773k (08-05); request CPU # GBP 1.34 -> 2.34/day and intercontinental egress 0.16 -> 0.97/day. # # ⚠️ THEY COST COMPLETELY DIFFERENT THINGS — do not reason about them together: # meta-webindexer 500/500 sampled requests were HTML = SSR RENDERS. # This is the CPU line. ~20k renders/day. # GeedoShopProductFinder 494/500 were static assets (JS chunks, images). # Near-zero CPU; this is the EGRESS line. # # WHY meta-webindexer IS BLOCKED DESPITE BEING A CITING ANSWER ENGINE. # It feeds Meta AI search and CAN cite/link sources, so by the rule used above # (cites = allowed, trains-only = blocked) it belongs in the allowed group. It # is blocked anyway because the citation channel measurably pays us nothing. # GA4, 90 days to 2026-08-05, bot-filtered, by sessionSource: # # chatgpt.com 446 perplexity 84 gemini 12 copilot 11 claude.ai 7 # meta.ai 0 <- zero, while meta-externalagent crawled ~9k pages/day # right up to the 07-31 block # # Allowing citing crawlers genuinely pays — ChatGPT sends 446 sessions. It just # does not pay for Meta. And 79% of meta-webindexer's crawl is # /fr/*/annuaire-professionnel/ — the business directory this site already # NOINDEXED on 07-29, so even a future citation would surface a page we have # decided should not appear in search at all. # # 🔁 REVERSE THIS IF: `meta.ai` ever appears in # npm run ga -- report --days=90 --dimensions=sessionSource --metrics=sessions # Worth re-checking quarterly. The honest caveat is that meta-webindexer only # began crawling us ~08-04, so its citations had not had time to appear. # # 🔴 DO NOT ADD facebookexternalhit. It is a DIFFERENT agent and it is what the # 58 Facebook/Instagram sessions in that same 90 days actually come from — # l.facebook.com and lm.facebook.com are its share-link redirectors. Blocking it # kills link previews on every shared URL. Same for meta-externalfetcher # (user-initiated). The warning above about not merging the Meta tokens is now # backed by measurement, not just caution. # # GeedoShopProductFinder is geedo.com, a PRODUCT search engine indexing online # stores (documented IPs 83.99.206.0/24, reverse-DNS *.geedo.com). We are a # travel affiliate site, not a shop: it is crawling JS bundles and images # hunting for products that do not exist. No search value, no referral value. # # LESSON, now confirmed twice: blocking by token is RECURRING MAINTENANCE. # Amazonbot -> Amzn-SearchBot (07-27), meta-externalagent -> meta-webindexer # (~08-04). Expect a third token each time. The durable fix is the staged # path-level Disallow at the foot of this file, which stops the FETCH for every # compliant agent at once — but it is gated on deindexation completing. # ───────────────────────────────────────────────────────────────────────────── User-agent: Amazonbot User-agent: Amzn-SearchBot User-agent: meta-externalagent User-agent: meta-webindexer User-agent: GeedoShopProductFinder User-agent: AhrefsBot User-agent: SemrushBot User-agent: SleepBot User-agent: PetalBot User-agent: Bytespider User-agent: DotBot User-agent: SpiderLing User-agent: YouBot User-agent: Scrapy User-agent: ShapBot User-agent: VelenPublicWebCrawler User-agent: AffsignalCrawler User-agent: ABEvalBot User-agent: HanaleiBot User-agent: MJ12bot User-agent: BLEXBot User-agent: Barkrowler User-agent: DataForSeoBot User-agent: ImagesiftBot User-agent: serpstatbot User-agent: Diffbot User-agent: SeekportBot Disallow: / # ───────────────────────────────────────────────────────────────────────────── # STAGED — do not uncomment yet. Gated on deindexation completing. # # Why this beats blocking crawlers one token at a time: 76% of Amzn-SearchBot's # crawl on 2026-07-31 was the country x entity crossproduct, which returns 200 # for every combination: # # /en/mongolia/places/the-british-museum -> 200 # /en/guam/business-directory/oakwood-theme-park/avis -> 200 # /en/tunisia/places/national-gallery -> 200 # # ~200 countries x N entities x sub-pages (/avis, /a-propos, /evenements, # /annonces, /codes-promo) x 9 locales is an unbounded URL space. Amazon's # repeat rate across 1,617 requests was 0% — it is not re-crawling, it is # walking a maze with no end, which is why its volume climbed 8k -> 19k/day. # # These pages already serve `X-Robots-Tag: noindex, follow` (shipped 01466-lag, # 2026-07-29), so they will not be indexed — but noindex still costs a full SSR # render every time, because the crawler must fetch the page to discover it. # `follow` also means crawlers keep walking outward from them. Only a path-level # Disallow stops the fetch, and it applies to every compliant agent — Googlebot # included, which is the real prize: crawl budget currently spent here is budget # not spent on the affiliate money pages. # # 🔴 SEQUENCING — why this is commented out: # Disallow prevents Google from RE-CRAWLING, and therefore from ever seeing the # noindex header. URLs already in the index at the time you block can get stuck # there indefinitely. The deindex only shipped 2026-07-29. Wait until GSC shows # these URLs dropped out of the index, THEN uncomment. Blocking too early # freezes them in the index permanently — the exact opposite of the goal. # # This is the same conclusion already recorded at server.ts:966 ("Add the # Disallow only once Search Console shows these deindexed"). That comment is the # decision; this block is the prepared payload. Do not enable one without the # other. # # ⚠️ SOURCE OF TRUTH — do not hand-edit this list. # The segments below are the 28 distinct non-empty values of the five # NOINDEXED_FUNCTIONALITY_KEYS (server.ts:495 — classifiedAds, placemarks, # companies, events, news) resolved through FUNCTIONALITY_URIS in Enums.ts. They # must stay in lockstep with NOINDEXED_PATH_RE, or robots.txt will block a path # set that differs from the one being noindexed. Regenerate rather than patch: # # node -e "const {FUNCTIONALITY_URIS}=require('./src/app/misc/Enums.ts'); \ # console.log([...new Set(['classifiedAds','placemarks','companies', \ # 'events','news'].flatMap(k=>Object.values(FUNCTIONALITY_URIS[k]) \ # .filter(v=>typeof v==='string'&&v&&!v.startsWith('_'))))].sort().join('\n'))" # # ⚠️ Empty-segment hazard: placemarks, events and news carry an EMPTY STRING for # several locales in FUNCTIONALITY_URIS and fall back to `_default_` — which is # why `places` covers de/es/it/nl/pl/pt/ro while only fr has its own (`lieux`). # Verified against live traffic 2026-07-31: /de/.../places/, /ro/.../places/ all # appear. The same empty values are what makes the `v.length > 0` filter at # server.ts:510 load-bearing; the note there explains what an empty alternative # does to the regex. Never emit a bare `Disallow: //`. # # ⚠️ Two forms per segment on purpose: the country-prefixed one and a root-level # one, because a handful of URLs are served with no locale prefix (/places/…, # /lieux/… both seen in logs). Do NOT collapse to `Disallow: /*places/` — a # bare-substring pattern also swallows unrelated paths. # # ⚠️ The rules themselves are NOT here — they sit commented inside the # `User-agent: *` group at the TOP of this file, because a second `User-agent: *` # group down here would be silently dropped by strict parsers even though Google # merges it. To enable: uncomment the 56 `# Disallow:` lines at the top. Nothing # needs to change in this note. # ───────────────────────────────────────────────────────────────────────────── Sitemap: https://alltrippers.com/sitemap/sitemap-index.xml