# ========================================================================= # Translatorus.com — robots.txt # ========================================================================= # ---- Default policy: allow everything except specific paths ---- User-agent: * Allow: / # Block conversion / admin / internal pages Disallow: /admin/ Disallow: /*/admin/ Disallow: /*/order/confirmation Disallow: /test-order Disallow: /*/test-order Disallow: /*/commande-bis Disallow: /*/checkout Disallow: /*/payment # Block tracking-parameter URLs to avoid duplicate indexing Disallow: /*?gclid= Disallow: /*?wbraid= Disallow: /*?gbraid= Disallow: /*?utm_ Disallow: /*?fbclid= Disallow: /*&gclid= Disallow: /*&utm_ # ---- Crawl-delay for aggressive bots ---- User-agent: Bingbot Crawl-delay: 2 User-agent: Yandex Crawl-delay: 5 User-agent: AhrefsBot Crawl-delay: 5 # Audits de site (18/09/2026) : les deux crawls du matin tombaient en meme # temps sur l'hebergement mutualise (Ahrefs : 226 delais depasses, 2 a 12 s # de premier octet). Un espacement leger suffit a lisser la charge. User-agent: SiteAuditBot Crawl-delay: 1 User-agent: AhrefsSiteAudit Crawl-delay: 1 User-agent: SemrushBot Crawl-delay: 5 User-agent: MJ12bot Crawl-delay: 10 User-agent: DotBot Crawl-delay: 10 # ---- AI crawlers: allow indexing for citation in AI answers (AEO) ---- User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: anthropic-ai Allow: / User-agent: Google-Extended Allow: / User-agent: PerplexityBot Allow: / User-agent: CCBot Allow: / # ---- Sitemaps ---- Sitemap: https://translatorus.com/sitemap.xml