A sitemap gives a crawler an explicit list of URLs to consider. It is useful when new or deeply linked documents are difficult to discover through ordinary navigation. The list does not bypass authentication, grant crawl permission, guarantee indexing, or promise a revisit date. Treat it as a published inventory of intended public documents, and keep that inventory aligned with the site’s canonical URLs and actual release state.

Generate from the publication inventory

Use the same authoritative content selection that produces the public pages. Exclude drafts, private documents, and redundant tracking variants. A filename present in a source checkout is not necessarily a published URL. Conversely, pagination, translated documents, or generated routes may exist without a one-to-one source filename. Check the generator’s actual output rather than guessing the inventory from a directory listing.

The XML sitemap format uses a declared namespace and a urlset containing URL entries. Each loc supplies a fully qualified URL. XML escaping and URL encoding solve different problems: an ampersand separating query parameters must be escaped in the XML text, while characters inside the URL must also have the appropriate URI representation. Parse the generated file back to verify that the intended URL survives both layers.

Respect the format’s scope and size requirements. A sitemap normally lists URLs for the relevant host and permitted location scope; supported cross-site submission arrangements have additional rules. Each sitemap is limited to fifty thousand URLs and 52,428,800 uncompressed bytes. Gzip reduces transfer size without removing the uncompressed limit. Split larger inventories and publish an appropriate sitemap index rather than silently truncating the list.

When it breaks

A valid XML file can still list redirects, missing pages, or content that should not be indexed. Validate destinations as well as syntax, and align their canonical declarations with the intended inventory. An HTTP 200 response containing a login screen or generic error page is not proof that the listed document is available. Sitemap delivery and document usefulness are separate checks.

Modification dates can also become misleading. If lastmod is supplied, derive it from a significant change to the page rather than replacing every timestamp on each build. Google documents that it uses consistently accurate lastmod values and ignores priority and changefreq. Those fields are not a scheduling interface, and setting maximum priority everywhere does not command more crawling.

A sitemap cannot make a private page publicly retrievable or override robots exclusions. Nor does submission guarantee that a search engine will select the same canonical, fetch every entry, or include every page in results. Keep ordinary navigation usable and use observed crawling or indexing reports for those later outcomes. Do not describe a successful upload as proof that discovery and indexing have completed.

Check it quickly

Save this as sitemap.py and run python3 sitemap.py. The standard-library fixture serves a generated XML sitemap and a tiny local publication inventory. Three entries are public, including one with two query parameters; a draft is excluded. The generated URLs point only to the temporary loopback origin, and the validator checks that scope before requesting destinations.

from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from urllib.parse import urlsplit
import json
import xml.etree.ElementTree as ET

namespace = 'http://www.sitemaps.org/schemas/sitemap/0.9'
ET.register_namespace('', namespace)
pages = {'/article': True, '/zh/article': True, '/report?year=2026&region=west': True, '/draft': False}
state = {'bad_entry': False}


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/sitemap.xml':
            root = ET.Element(f'{{{namespace}}}urlset')
            paths = [path for path, public in pages.items() if public]
            if state['bad_entry']:
                paths.append('/missing')
            for path in paths:
                item = ET.SubElement(root, f'{{{namespace}}}url')
                ET.SubElement(item, f'{{{namespace}}}loc').text = f'http://127.0.0.1:{self.server.server_port}{path}'
            status, data, kind = 200, ET.tostring(root, encoding='utf-8', xml_declaration=True), 'application/xml'
        else:
            status = 200 if pages.get(self.path) else 404
            data, kind = (b'published page' if status == 200 else b'missing'), 'text/plain'
        self.send_response(status)
        self.send_header('Content-Type', kind)
        self.send_header('Content-Length', str(len(data)))
        self.end_headers()
        self.wfile.write(data)

    def log_message(self, *args):
        pass


server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
thread = Thread(target=server.serve_forever)
thread.start()
try:
    def fetch(path):
        connection = HTTPConnection('127.0.0.1', server.server_port, timeout=5)
        try:
            connection.request('GET', path)
            response = connection.getresponse()
            return response.status, response.read()
        finally:
            connection.close()

    def validate(data):
        root = ET.fromstring(data)
        assert root.tag == f'{{{namespace}}}urlset'
        locations = [item.text for item in root.findall(f'{{{namespace}}}url/{{{namespace}}}loc')]
        assert len(locations) == len(set(locations))
        for location in locations:
            parsed = urlsplit(location)
            assert (parsed.scheme, parsed.hostname, parsed.port) == ('http', '127.0.0.1', server.server_port)
            path = parsed.path + ('?' + parsed.query if parsed.query else '')
            status, _ = fetch(path)
            if status != 200:
                raise ValueError(f'listed URL returned {status}: {path}')
        return locations

    status, data = fetch('/sitemap.xml')
    assert status == 200 and b'&' in data
    locations = validate(data)
    assert len(locations) == 3 and all('/draft' not in value for value in locations)
    assert any('?year=2026&region=west' in value for value in locations)
    state['bad_entry'] = True
    _, invalid = fetch('/sitemap.xml')
    try:
        validate(invalid)
    except ValueError as error:
        rejection = str(error)
    else:
        raise AssertionError('missing destination accepted')
    print(json.dumps({'public_urls': len(locations), 'query_round_trip': True,
                      'draft_excluded': True, 'invalid_destination_rejected': rejection,
                      'indexing_tested': False}))
finally:
    server.shutdown()
    thread.join()
    server.server_close()

The first sitemap contains three unique loc values in the correct namespace. Its raw XML escapes the query separator as &, while parsing recovers the intended ampersand in the URL. All three destinations return 200, and the draft does not appear. The example omits optional modification dates because it has no publication-history source to justify them.

The fixture then adds a missing destination while continuing to serve well-formed XML with status 200. The second validation fails when that listed URL returns 404. This demonstrates why a sitemap syntax check alone cannot establish that its inventory is useful. The failure is reported instead of quietly removing the entry during validation.

No search engine is contacted, and this small validator does not implement every protocol limit or indexing rule. It verifies generation, escaping, namespace, uniqueness, local scope, and destination availability for the stated examples. Before publishing the real sitemap, compare it with the complete intended public inventory and check the released file at its advertised URL. That establishes what was published without claiming control over a crawler’s later choices.

Sources