A robots.txt file communicates crawling rules to automated clients that implement the Robots Exclusion Protocol. It can reduce unwanted traversal of expensive or unhelpful paths, but the server does not automatically enforce its contents against incoming requests. RFC 9309 explicitly separates these rules from access authorization. Treat robots.txt as a public interface for cooperating crawlers, while authentication and authorization decide who may actually retrieve protected resources.

Match the intended crawler and path

The standard file lives at /robots.txt for the relevant scheme and authority. A file under an application’s subdirectory does not automatically set the origin’s policy. User-agent groups identify the crawler products to which rules apply, with a wildcard group covering other agents according to the protocol. Path rules are evaluated against request targets; capitalization can change the match. Review concrete example URLs whenever routing changes.

Keep the file readable as UTF-8 plain text and make its intended response explicit. A broad static-site fallback might return a home page with status 200 at the robots path. That proves HTTP delivery, not that the crawler received the intended policy. Complicated Allow and Disallow combinations, escaping, and wildcard patterns need checks against the protocol and the actual crawler implementation. A simple parser demonstration does not certify every edge case.

When it breaks

Listing a private path in robots.txt does not conceal it. The file itself can advertise that path, and a client can request it without consulting the policy. Even a cooperating crawler’s decision does not configure an access rule in the origin server. If content must be restricted, enforce that restriction where the request is authorized; a Disallow line is not a replacement for the permission check.

Crawling and indexing are also different. Google documents that a disallowed URL can still appear in search results when other sources link to it. If the crawler cannot fetch the page, it may not see an intended noindex directive there. Choose the appropriate indexing control for public content, and use real access restrictions for private content. Do not assume that blocking retrieval automatically removes every existing index entry.

Availability failures have distinct semantics. RFC 9309 allows a crawler to access resources when the robots file is unavailable, for example under HTTP 4xx responses. When it is unreachable because of server or network errors, such as 5xx, the crawler must initially assume complete disallow. The specification also addresses prolonged failures and cached copies. Treat a broken deployment of the policy as a behavioral change, not merely a missing decorative file.

Crawler caching introduces publication delay. A successful update at the origin does not prove that every crawler has fetched the new rules. Keep the served response, intended rules, and observed crawler behavior as separate evidence. A sitemap pointer can aid discovery, but it does not grant access or guarantee crawling and indexing of every listed URL.

Check it quickly

Save this as robots.py and run python3 robots.py. The fixture starts a loopback HTTP server serving a short robots file and harmless demonstration content. It fetches the actual policy and passes those bytes to Python’s RobotFileParser. The selected paths use a simple Disallow prefix without conflicting rules or advanced wildcard behavior.

from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from urllib.robotparser import RobotFileParser
import json


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/robots.txt':
            status, data = 200, b'User-agent: *\nDisallow: /private/\n'
        elif self.path in ('/private/report', '/public'):
            status, data = 200, b'public demonstration data\n'
        else:
            status, data = 404, b'missing\n'
        self.send_response(status)
        self.send_header('Content-Type', 'text/plain; charset=utf-8')
        self.send_header('Content-Length', str(len(data)))
        self.end_headers()
        self.wfile.write(data)

    def log_message(self, *args):
        pass


server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
thread = Thread(target=server.serve_forever)
thread.start()
try:
    origin = f'http://127.0.0.1:{server.server_port}'

    def fetch(path):
        connection = HTTPConnection('127.0.0.1', server.server_port, timeout=5)
        try:
            connection.request('GET', path)
            response = connection.getresponse()
            return response.status, response.read().decode()
        finally:
            connection.close()

    status, rules = fetch('/robots.txt')
    assert status == 200
    parser = RobotFileParser(origin + '/robots.txt')
    parser.parse(rules.splitlines())
    decisions = {path: parser.can_fetch('ExampleBot', origin + path)
                 for path in ('/private/report', '/public', '/Private/report')}
    assert decisions == {'/private/report': False, '/public': True, '/Private/report': True}
    direct_status, direct_body = fetch('/private/report')
    assert direct_status == 200 and direct_body == 'public demonstration data\n'
    print(json.dumps({'parser_decisions': decisions, 'direct_http_status': direct_status,
                      'direct_http_body': direct_body, 'search_index_tested': False}))
finally:
    server.shutdown()
    thread.join()
    server.server_close()

The parser rejects /private/report, permits /public, and permits the differently capitalized /Private/report. Those are decisions returned to the caller. The script then deliberately performs an ordinary HTTP GET for /private/report without applying the parser’s decision. The server returns 200 and the public demonstration text, proving that the policy did not install an access barrier.

No search engine or indexing service participates in this experiment. It verifies one parser’s selected matching behavior and the distinction between that decision and server enforcement. It does not establish full RFC conformance of the library or predict every crawler’s behavior. Before publishing a policy, check representative allowed and disallowed URLs, verify the actual response, and keep the resource’s authorization requirements independently satisfied.

Sources