User-agent: * Allow: / # OG image endpoints serve real PNG content referenced by og:image meta tags. # Google needs to fetch these to validate social previews; leaving them under # the /api/ Disallow caused 46+ "Blocked by robots.txt" warnings in GSC and # stops OG previews showing in Google Discover / SERP image cards. Allow: /api/og/ # The agent API (read-only catalogue JSON for AI tools, attribution required; see /for-ai). Its # responses carry X-Robots-Tag: noindex, so allowing the fetch does not index the JSON. Allow: /api/agent/ Disallow: /api/ Disallow: /admin Disallow: /admin/ Disallow: /sign-in Disallow: /sign-up Disallow: /sso-callback # Counted outbound hops (/go/driva, /go/offer/): a 302 to a partner or a maker, never content. # The responses also carry X-Robots-Tag: noindex, nofollow. Disallow: /go # AI answer/search crawlers: explicitly welcomed (CarSorted wants AI-search # citation), granted the same access as everyone else. Listed by name so a future # "block the AI scrapers" edit can't silently catch the citation-critical ones. # Reuse terms (who may crawl, cite, infer and train, with attribution): https://www.carsorted.ai/terms#data-reuse User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: Meta-ExternalAgent User-agent: Meta-WebIndexer Allow: /api/og/ Allow: /api/agent/ Disallow: /api/ Disallow: /admin Disallow: /admin/ Disallow: /sign-in Disallow: /sign-up Disallow: /sso-callback # Counted outbound hops (/go/driva, /go/offer/): a 302 to a partner or a maker, never content. # The responses also carry X-Robots-Tag: noindex, nofollow. Disallow: /go Allow: / Sitemap: https://www.carsorted.ai/sitemap.xml