Seo · XB Field Notes

Sitemaps guide crawler discovery

Wed, 30 Jun 2027 00:00:00 +0000

A sitemap gives a crawler an explicit list of URLs to consider. It is useful when new or deeply linked documents are difficult to discover through ordinary navigation. The list does not bypass authentication, grant crawl permission, guarantee indexing, or promise a revisit date. Treat it as a published inventory of intended public documents, and keep that inventory aligned with the site’s canonical URLs and actual release state.

Generate from the publication inventory

Use the same authoritative content selection that produces the public pages. Exclude drafts, private documents, and redundant tracking variants. A filename present in a source checkout is not necessarily a published URL. Conversely, pagination, translated documents, or generated routes may exist without a one-to-one source filename. Check the generator’s actual output rather than guessing the inventory from a directory listing.

The XML sitemap format uses a declared namespace and a urlset containing URL entries. Each loc supplies a fully qualified URL. XML escaping and URL encoding solve different problems: an ampersand separating query parameters must be escaped in the XML text, while characters inside the URL must also have the appropriate URI representation. Parse the generated file back to verify that the intended URL survives both layers.

Respect the format’s scope and size requirements. A sitemap normally lists URLs for the relevant host and permitted location scope; supported cross-site submission arrangements have additional rules. Each sitemap is limited to fifty thousand URLs and 52,428,800 uncompressed bytes. Gzip reduces transfer size without removing the uncompressed limit. Split larger inventories and publish an appropriate sitemap index rather than silently truncating the list.

When it breaks

A valid XML file can still list redirects, missing pages, or content that should not be indexed. Validate destinations as well as syntax, and align their canonical declarations with the intended inventory. An HTTP 200 response containing a login screen or generic error page is not proof that the listed document is available. Sitemap delivery and document usefulness are separate checks.

Modification dates can also become misleading. If lastmod is supplied, derive it from a significant change to the page rather than replacing every timestamp on each build. Google documents that it uses consistently accurate lastmod values and ignores priority and changefreq. Those fields are not a scheduling interface, and setting maximum priority everywhere does not command more crawling.

A sitemap cannot make a private page publicly retrievable or override robots exclusions. Nor does submission guarantee that a search engine will select the same canonical, fetch every entry, or include every page in results. Keep ordinary navigation usable and use observed crawling or indexing reports for those later outcomes. Do not describe a successful upload as proof that discovery and indexing have completed.

Check it quickly

Save this as sitemap.py and run python3 sitemap.py. The standard-library fixture serves a generated XML sitemap and a tiny local publication inventory. Three entries are public, including one with two query parameters; a draft is excluded. The generated URLs point only to the temporary loopback origin, and the validator checks that scope before requesting destinations.

from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from urllib.parse import urlsplit
import json
import xml.etree.ElementTree as ET

namespace = 'http://www.sitemaps.org/schemas/sitemap/0.9'
ET.register_namespace('', namespace)
pages = {'/article': True, '/zh/article': True, '/report?year=2026&region=west': True, '/draft': False}
state = {'bad_entry': False}


class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 if self.path == '/sitemap.xml':
 root = ET.Element(f'{{{namespace}}}urlset')
 paths = [path for path, public in pages.items() if public]
 if state['bad_entry']:
 paths.append('/missing')
 for path in paths:
 item = ET.SubElement(root, f'{{{namespace}}}url')
 ET.SubElement(item, f'{{{namespace}}}loc').text = f'http://127.0.0.1:{self.server.server_port}{path}'
 status, data, kind = 200, ET.tostring(root, encoding='utf-8', xml_declaration=True), 'application/xml'
 else:
 status = 200 if pages.get(self.path) else 404
 data, kind = (b'published page' if status == 200 else b'missing'), 'text/plain'
 self.send_response(status)
 self.send_header('Content-Type', kind)
 self.send_header('Content-Length', str(len(data)))
 self.end_headers()
 self.wfile.write(data)

 def log_message(self, *args):
 pass


server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
thread = Thread(target=server.serve_forever)
thread.start()
try:
 def fetch(path):
 connection = HTTPConnection('127.0.0.1', server.server_port, timeout=5)
 try:
 connection.request('GET', path)
 response = connection.getresponse()
 return response.status, response.read()
 finally:
 connection.close()

 def validate(data):
 root = ET.fromstring(data)
 assert root.tag == f'{{{namespace}}}urlset'
 locations = [item.text for item in root.findall(f'{{{namespace}}}url/{{{namespace}}}loc')]
 assert len(locations) == len(set(locations))
 for location in locations:
 parsed = urlsplit(location)
 assert (parsed.scheme, parsed.hostname, parsed.port) == ('http', '127.0.0.1', server.server_port)
 path = parsed.path + ('?' + parsed.query if parsed.query else '')
 status, _ = fetch(path)
 if status != 200:
 raise ValueError(f'listed URL returned {status}: {path}')
 return locations

 status, data = fetch('/sitemap.xml')
 assert status == 200 and b'&amp;' in data
 locations = validate(data)
 assert len(locations) == 3 and all('/draft' not in value for value in locations)
 assert any('?year=2026&region=west' in value for value in locations)
 state['bad_entry'] = True
 _, invalid = fetch('/sitemap.xml')
 try:
 validate(invalid)
 except ValueError as error:
 rejection = str(error)
 else:
 raise AssertionError('missing destination accepted')
 print(json.dumps({'public_urls': len(locations), 'query_round_trip': True,
 'draft_excluded': True, 'invalid_destination_rejected': rejection,
 'indexing_tested': False}))
finally:
 server.shutdown()
 thread.join()
 server.server_close()

The first sitemap contains three unique loc values in the correct namespace. Its raw XML escapes the query separator as &, while parsing recovers the intended ampersand in the URL. All three destinations return 200, and the draft does not appear. The example omits optional modification dates because it has no publication-history source to justify them.

The fixture then adds a missing destination while continuing to serve well-formed XML with status 200. The second validation fails when that listed URL returns 404. This demonstrates why a sitemap syntax check alone cannot establish that its inventory is useful. The failure is reported instead of quietly removing the entry during validation.

No search engine is contacted, and this small validator does not implement every protocol limit or indexing rule. It verifies generation, escaping, namespace, uniqueness, local scope, and destination availability for the stated examples. Before publishing the real sitemap, compare it with the complete intended public inventory and check the released file at its advertised URL. That establishes what was published without claiming control over a crawler’s later choices.

Sources

Sitemaps.org protocol, XML structure, URL encoding, scope, and size limits, checked 2026-09-13.
Google: build and submit a sitemap, canonical URLs, meaningful lastmod, and unsupported hints, checked 2026-09-13.

Canonical URLs name a preferred representation

Sat, 26 Jun 2027 00:00:00 +0000

Several URLs can deliver the same document: a tracking parameter, alternate host, or trailing-slash variant may leave the meaningful content unchanged. A canonical link identifies the preferred representation among suitable duplicates. It helps consumers interpret that relationship, but it does not redirect an HTTP request or force a search engine to select the declared URL. The declaration is useful only when its target and surrounding signals describe the actual content relationship.

Choose identity before writing the tag

Decide which public URL should represent the document, then make internal links and relevant sitemap entries consistent with that choice. Do not strip every query parameter automatically. A tracking parameter may be redundant, while a language, page number, or product selection can materially change the representation. Collapsing those URLs without comparing their meaning can point distinct documents at the wrong preferred page.

In HTML, put a link element with rel="canonical" in a valid head and use a clear target URL. An HTTP Link header can express the relation for other representations, such as a PDF. Prefer an absolute public URL to avoid ambiguous resolution through a base URL or deployment prefix. Generate it from the site’s intended public configuration, rather than accidentally publishing an internal proxy host.

Canonicalization is not limited to a single origin by the link relation itself. Cross-domain declarations need careful content and ownership review; a canonical value does not prove control of the target. For translated pages, normally select an appropriate canonical in the same language when one exists. Language alternatives and canonical duplicates describe different relationships, so sending every translation to the English homepage erases meaningful distinctions.

When it breaks

A target can return an error, redirect somewhere else, or declare another canonical. Multiple conflicting declarations can also appear when a template, plugin, and response header each choose independently. Review the complete response rather than looking for the mere presence of one tag. Keep the preferred target usable and the declarations consistent with the actual representation.

An ordinary link relation leaves the original URL accessible unless routing separately changes it. If an old route should move for all clients, configure an appropriate HTTP redirect and verify its method behavior. If variants remain useful, a canonical declaration may coexist with successful responses at them. Do not equate the metadata signal with a redirect, an authorization rule, or an instruction that a crawler must obey.

Search engines combine signals and may choose another canonical. A correct tag therefore does not prove that indexing has converged. Distinguish what the origin serves from what a search service reports. Check for inconsistent internal links, sitemap entries, redirects, or language metadata before assuming that adding more copies of the same tag will solve disagreement.

Check it quickly

Save this as canonical.py and run python3 canonical.py. The script creates a temporary loopback HTTP origin and reads actual HTML responses with Python’s HTML parser. It serves an English document, a tracking variant, a Chinese document, and two intentionally inconsistent metadata cases. The local URLs are demonstration identities, not search-engine submissions.

from html.parser import HTMLParser
from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
import json


class Canonical(HTMLParser):
 def __init__(self):
 super().__init__()
 self.urls = []

 def handle_starttag(self, tag, attrs):
 values = dict(attrs)
 if tag == 'link' and 'canonical' in values.get('rel', '').split():
 self.urls.append(values.get('href'))


class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 origin = f'http://127.0.0.1:{self.server.server_port}'
 chinese = self.path.startswith('/zh/')
 target = '/zh/article' if chinese else '/article'
 if self.path == '/zh/wrong':
 target = '/article'
 link = f'<link rel="canonical" href="{origin}{target}">'
 if self.path == '/conflict':
 link += f'<link rel="canonical" href="{origin}/another">'
 title = '中文文章' if chinese else 'English article'
 data = f'<!doctype html><html lang="{"zh" if chinese else "en"}"><head><meta charset="utf-8"><title>{title}</title>{link}</head><body><h1>{title}</h1></body></html>'.encode()
 self.send_response(200)
 self.send_header('Content-Type', 'text/html; charset=utf-8')
 self.send_header('Content-Length', str(len(data)))
 self.end_headers()
 self.wfile.write(data)

 def log_message(self, *args):
 pass


server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
thread = Thread(target=server.serve_forever)
thread.start()
try:
 origin = f'http://127.0.0.1:{server.server_port}'

 def fetch(path):
 connection = HTTPConnection('127.0.0.1', server.server_port, timeout=5)
 try:
 connection.request('GET', path)
 response = connection.getresponse()
 text = response.read().decode()
 parsed = Canonical()
 parsed.feed(text)
 return response.status, response.getheader('Location'), text, parsed.urls
 finally:
 connection.close()

 variant = fetch('/article?utm_source=example')
 primary = fetch('/article')
 translated = fetch('/zh/article')
 assert variant[:2] == (200, None) and variant[2] == primary[2]
 assert variant[3] == [origin + '/article']
 assert translated[3] == [origin + '/zh/article']
 wrong = fetch('/zh/wrong')
 conflict = fetch('/conflict')
 assert wrong[3] != [origin + '/zh/article']
 assert len(conflict[3]) == 2
 print(json.dumps({'variant_status': variant[0], 'variant_redirect': variant[1],
 'same_representation': variant[2] == primary[2],
 'english_target': variant[3], 'chinese_target': translated[3],
 'wrong_language_detected': True, 'conflicting_targets': len(conflict[3]),
 'search_engine_selection_tested': False}))
finally:
 server.shutdown()
 thread.join()
 server.server_close()

The tracking variant returns 200 without a Location header. Its bytes match the English primary page and its canonical points to /article. The Chinese page instead points to /zh/article. This verifies both the duplicate relationship in the fixture and the separation between the language representations.

The wrong-language case points a Chinese response at the English target, while the conflicting case emits two different canonical targets. Both still return 200, so a status-only check would miss the inconsistency. The assertions identify those specific errors by inspecting the metadata and the intended language mapping.

No crawler participates, and the small parser is not a complete canonicalization validator. The experiment establishes served declarations and response behavior, not Google’s selected canonical or ranking. Before changing production identity, review equivalent content, public targets, incoming routes, and language relationships together. Preserve existing URLs or their deliberate redirects so a metadata cleanup does not quietly break users’ saved links.

Sources

RFC 6596: canonical link relation, preferred representations and the scope of the link relation, checked 2026-09-13.
Google: consolidate duplicate URLs, canonical signals, consistent targets, and language handling, checked 2026-09-13.

Redirect chains add requests

Sun, 04 Oct 2026 00:00:00 +0000

A followed redirect makes the client resolve a Location value and issue another request before it reaches the desired representation. Several individually reasonable rules can form a chain: normalize a hostname, add HTTPS, fix a trailing slash, and move an old article. Review the complete path seen by a visitor, because each intermediary becomes another dependency.

Requests and connections are different counts

A fresh redirect response normally adds a sequential request-response exchange. The next destination is learned from that response, so useful work waits for the earlier step. However, another request does not necessarily require another TCP connection. Same-origin HTTP/1.1 can reuse a persistent connection, while HTTP/2 and HTTP/3 have their own connection and stream behavior. Moving to another host can add name resolution or connection establishment, depending on what is already available.

Cached redirects and browser transport upgrades can alter the network sequence. HSTS may upgrade an HTTP URL internally, without contacting its plaintext endpoint. A permanent redirect may already be cached. Consequently, neither a fixed number of milliseconds per hop nor a claim that three redirects triple total page load time is defensible without measurement. Redirect traversal is only part of loading a document and its subresources.

Choose status codes for their meaning. With 301 and 302, historical handling permits a POST to become GET; 307 and 308 preserve the method, while 303 directs retrieval of another resource. Permanence and cacheability also matter. Replacing every rule with a permanent redirect merely to improve a trace can change client behavior long after a mistaken release is corrected.

Location can contain a relative reference. Resolve it against the current request URL before deciding its origin and destination. Recording only the raw header can hide how an apparently identical value leads to different paths when returned from different directories.

When it breaks

Two rules can send a request back and forth, such as opposite preferences for a trailing slash. A successful response at each individual endpoint does not prove termination. Clients enforce redirect limits, but reaching that limit is a failure rather than successful protection of the user’s task. Trace visited destinations and test a representative set of old URLs after changing shared rules.

A middle destination can also become unavailable while the final page remains healthy. Update internal links to the final canonical destination and, where semantics allow, point historical addresses there directly. Preserve meaningful query parameters and encoding during this change. Avoid unrestricted user-supplied redirect targets, and do not blindly forward credentials to a different origin.

Crawler traversal consumes requests too, but a chain does not imply a predictable search ranking penalty. Search engines have their own processing and limits. Report observable outcomes such as hop counts, loops, final status, and target consistency; do not turn a shorter local trace into an unsupported SEO claim.

Check it quickly

Save the following as redirect-chain.py and run python3 redirect-chain.py. Python 3 creates an HTTP/1.1 loopback server and a single HTTPConnection. Responses contain explicit lengths so the client can finish reading each response and reuse its connection. Cache-Control is no-store to keep the experiment about actual exchanges rather than cached redirects.

from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from urllib.parse import urljoin, urlsplit
import json

events = []
routes = {'/long': '/middle', '/middle': '/final', '/short': '/final',
 '/loop-a': '/loop-b', '/loop-b': '/loop-a'}


class Handler(BaseHTTPRequestHandler):
 protocol_version = 'HTTP/1.1'

 def do_GET(self):
 events.append({'path': self.path, 'peer_port': self.client_address[1]})
 status = 302 if self.path in routes else 200 if self.path == '/final' else 404
 data = b'final representation' if status == 200 else b''
 self.send_response(status)
 if status == 302:
 self.send_header('Location', routes[self.path])
 self.send_header('Cache-Control', 'no-store')
 self.send_header('Content-Length', str(len(data)))
 self.end_headers()
 self.wfile.write(data)

 def log_message(self, *args):
 pass


server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
thread = Thread(target=server.serve_forever)
thread.start()
connection = HTTPConnection('127.0.0.1', server.server_port, timeout=5)
try:
 origin = f'http://127.0.0.1:{server.server_port}'

 def follow(start):
 path, seen, trail = start, set(), []
 for _ in range(5):
 if path in seen:
 raise ValueError('redirect loop')
 seen.add(path)
 connection.request('GET', path)
 response = connection.getresponse()
 body = response.read()
 trail.append([path, response.status])
 if response.status == 200:
 assert body == b'final representation'
 return trail
 assert response.status in (301, 302, 303, 307, 308)
 target = urlsplit(urljoin(origin + path, response.getheader('Location')))
 assert (target.scheme, target.netloc) == ('http', urlsplit(origin).netloc)
 path = target.path + ('?' + target.query if target.query else '')
 raise ValueError('redirect budget exhausted')

 long_path = follow('/long')
 short_path = follow('/short')
 assert len(long_path) == 3 and len(short_path) == 2
 try:
 follow('/loop-a')
 except ValueError as error:
 assert str(error) == 'redirect loop'
 else:
 raise AssertionError('loop accepted')
 connections = len({event['peer_port'] for event in events})
 assert connections == 1 and len(events) == 7
 print(json.dumps({'long_path': long_path, 'short_path': short_path,
 'loop_rejected': True, 'requests': len(events),
 'observed_tcp_connections': connections}))
finally:
 connection.close()
 server.shutdown()
 thread.join()
 server.server_close()

Expect /long → /middle → /final to contain three requests and /short → /final two. The loop begins at /loop-a, visits /loop-b, and is rejected before requesting the first destination again. The server observes seven requests in total and exactly one peer connection. These assertions ran successfully with Python 3.14.4; the program then closes its connection and server.

This is a count experiment using GET and temporary 302 responses. It does not benchmark latency, exercise POST rewriting, or test TLS and cross-origin credential behavior. The client intentionally accepts only same-origin targets and a small hop budget; it is not a general redirect library. In a real deployment, combine a full request trace with tests for method, query, host, and old-address preservation. A direct route is useful only if it still reaches the intended representation with the required semantics.

Sources

MDN: Redirections in HTTP, redirect chains, loops and code selection, checked 2026-09-13.
RFC 9110: HTTP Semantics, Location and redirection method semantics, checked 2026-09-13.