A service registry is only as useful as its honesty. Consul keeps a catalog of registered instances and attaches health checks to them, so a caller asking where a healthy instance of a service lives gets an answer that changes as reality changes. Registration is trivial; the check is the whole product, because a registry that lists dead instances is worse than a static list — it actively routes traffic into a hole. ...
A sitemap gives a crawler an explicit list of URLs to consider. It is useful when new or deeply linked documents are difficult to discover through ordinary navigation. The list does not bypass authentication, grant crawl permission, guarantee indexing, or promise a revisit date. Treat it as a published inventory of intended public documents, and keep that inventory aligned with the site’s canonical URLs and actual release state. ...
A robots.txt file communicates crawling rules to automated clients that implement the Robots Exclusion Protocol. It can reduce unwanted traversal of expensive or unhelpful paths, but the server does not automatically enforce its contents against incoming requests. RFC 9309 explicitly separates these rules from access authorization. Treat robots.txt as a public interface for cooperating crawlers, while authentication and authorization decide who may actually retrieve protected resources. ...
robots.txt is a crawler-facing preference; a sitemap is a URL inventory; neither is an authorization layer.