Two small files are often treated as if they were a security boundary. They are not. A robots.txt file describes crawler preferences, while a sitemap gives search systems a list of URLs that belong to the site.

What robots.txt can and cannot do

The rules are published at the origin’s /robots.txt path and are read by cooperating crawlers. They do not protect a private directory, remove a URL from the network, or replace authentication. A route that must not be public needs server authorization or a network boundary.

Keep the file boring and test its actual location:

User-agent: *
Disallow:
Sitemap: https://www.example.test/sitemap.xml

The file should be UTF-8. If a deployment redirects it, verify the crawler behavior you intend rather than assuming every client follows the same chain.

What a sitemap adds

A sitemap is an inventory, not a permission list. Use absolute URLs from one host, encode XML values correctly, and include only canonical pages that you are prepared to publish. Do not list a URL merely because it exists in a build directory.

The protocol permits up to 50,000 URLs and 50 MB per sitemap file. A small site does not need an index file; one ordinary sitemap is easier to inspect. Validate it as XML and request it through the same HTTPS edge as a normal page.

curl -fsS https://www.example.test/robots.txt
curl -fsS https://www.example.test/sitemap.xml | xmllint --noout -

If xmllint is not installed, parse the XML in the same CI environment that publishes it. The important part is that the check runs before the file is replaced at the edge.

Sources