User-agent: * Disallow: /accounts/ Disallow: /atom_img/ Disallow: /boond_link_img/ Disallow: /boonds_api/ Disallow: /private_api/ Disallow: /jsi18n/ Disallow: /promoted_users/ Disallow: /*.pdf$ Disallow: /register/* # URLs with ?utm_ and tracking parameters are NOT blocked on purpose (2026-10-20): # blocking them stopped Google from reading their canonical/noindex tags, so it indexed # hundreds of tagged copies blind (and noindex user pages leaked in through them). Allow: /register/$ User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BlexBot Disallow: / User-agent: Gigabot Disallow: / User-agent: rogerbot Disallow: / User-agent: Yandex Disallow: / User-agent: Baidu Disallow: / User-agent: Sogou Disallow: / User-agent: Exabot Disallow: / User-agent: AddThis Disallow: / User-agent: SiteBot Disallow: / # CCBot (Common Crawl) allowed since 2026-11-09: its crawl feeds the datasets AI assistants learn from. User-agent: Mail.Ru Disallow: / User-agent: YandexBot Disallow: / User-agent: Uptimebot Disallow: / User-agent: Pingdom Disallow: / User-agent: Updown Disallow: / Sitemap: https://www.atom.bio/sitemap.xml Sitemap: https://d209nv3nexrgbg.cloudfront.net/sitemaps/profile-directory/sitemap-directory-index.xml