A crawler icon beside a branching sitemap structure.

robots.txt and sitemaps answer different questions

robots.txt is a crawler-facing preference; a sitemap is a URL inventory; neither is an authorization layer.

August 22, 2026 · 2 min · XB Field Notes