User-agent: * Allow: / # /listings/*/apply and /claim/* are intentionally NOT disallowed here. # Both send "X-Robots-Tag: noindex, nofollow", and a crawler has to be able to # fetch a URL to see that header. Blocking them in robots.txt would hide the # noindex and leave the URLs sitting in the index as bare results instead. Sitemap: https://solajobs.com/sitemap.xml