A DNS TTL expresses how long a resource record may ordinarily remain in a cache before refresh is needed. Changing the authoritative record changes future answers, but it does not send an invalidation message to every resolver that previously stored the data. A migration therefore has both an authoritative update and a period during which clients can legitimately discover different addresses.

When it breaks

Lowering TTL at the moment of an address change does not shorten the lifetime of earlier cached answers. A resolver that stored yesterday’s longer TTL has not learned today’s shorter one. Removing the old service immediately can break those clients even when every authoritative query already returns the new address. The failure is a mismatch between service overlap and outstanding discovery state.

Rollback has the same asymmetry. Changing the zone back does not retract the new address from caches that already learned it. A destination that worked briefly may continue receiving requests after the rollback command completes. Keep the rollback target and the recently published target usable for the relevant overlap, or define how clients using either address will receive an acceptable result.

Do not calculate propagation by adding every cache layer’s original TTL. Ordinary forwarding caches return remaining lifetime, and downstream caches age that received value. Independent resolver populations may have filled at different times, however. Application-specific caching, long-lived connections, authoritative synchronization delays, and configured serve-stale behavior also prevent a single TTL from becoming a universal end-to-end completion guarantee.

Check it quickly

Save this as check-dns-move.py and run python3 check-dns-move.py with Docker. The pinned environment uses BIND 9.18.49 as an authority and dnsmasq 2.90 as one forwarding cache. Both run on the isolated container’s loopback interface. The example changes only its disposable zone, increments its serial, and confirms each authoritative reload before observing the cache:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends bind9=1:9.18.49-1~deb12u2 bind9-dnsutils=1:9.18.49-1~deb12u2 dnsmasq-base=2.90-4~deb12u2 python3-dnspython=2.3.0-1 && rm -rf /var/lib/apt/lists/*\nCMD ["/usr/bin/python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,re,signal,subprocess,time
import dns.exception,dns.message,dns.query,dns.rcode,dns.rdatatype

def run(*args,check=True):
    result=subprocess.run(args,capture_output=True,text=True,timeout=15)
    if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
    return result
def query(name,kind='A',port=1053):
    return dns.query.udp(dns.message.make_query(name,kind),'127.0.0.1',port=port,timeout=1)
def values(response,kind='A'):
    wanted=dns.rdatatype.from_text(kind)
    return [item.to_text() for rrset in response.answer if rrset.rdtype==wanted for item in rrset]
def show(response):
    def section(records):return [{'name':str(rr.name),'type':dns.rdatatype.to_text(rr.rdtype),'ttl':rr.ttl,'data':[item.to_text() for item in rr]} for rr in records]
    return {'rcode':dns.rcode.to_text(response.rcode()),'answer':section(response.answer),'authority':section(response.authority)}
def wait(predicate,timeout=15):
    deadline=time.monotonic()+timeout
    while time.monotonic()<deadline:
        try:
            if predicate():return
        except (OSError,dns.exception.Timeout):pass
        time.sleep(.05)
    raise AssertionError('DNS fixture condition did not become true')
def stop(process):
    if process.poll() is None:
        process.terminate()
        try:process.wait(timeout=5)
        except subprocess.TimeoutExpired:process.kill();process.wait(timeout=5)

@contextmanager
def lab(records,soa_ttl=8,minimum=4,extra=''):
    with TemporaryDirectory(prefix='xbcatm-dns-') as directory,ExitStack() as stack:
        root=Path(directory);zone=root/'zone';conf=root/'named.conf';log=root/'named.log';serial=1
        def save(body):
            zone.write_text(f'$ORIGIN lab.test.\n$TTL 30\n@ {soa_ttl} IN SOA ns.lab.test. hostmaster.lab.test. ({serial} 60 60 3600 {minimum})\n@ IN NS ns.lab.test.\nns IN A 127.0.0.1\n'+body+'\n')
            run('named-checkzone','lab.test',str(zone))
        save(records)
        conf.write_text(f'''options {{ directory "{root}"; listen-on port 1053 {{127.0.0.1;}}; listen-on-v6 {{none;}};
recursion no; dnssec-validation no; notify no; querylog yes; max-cache-size 8m; pid-file "{root}/named.pid"; session-keyfile none;
{extra} }};
controls {{}};
zone "lab.test" {{ type primary; file "{zone}"; }};
''')
        run('named-checkconf',str(conf))
        output=stack.enter_context(log.open('w'))
        authority=subprocess.Popen(['named','-g','-c',str(conf),'-n','1','-U','1'],stdout=output,stderr=subprocess.STDOUT)
        stack.callback(stop,authority)
        def ready():
            assert authority.poll() is None,log.read_text()
            response=query('lab.test.','SOA');return response.rcode()==0 and response.answer[0][0].serial==serial
        wait(ready)
        def update(body):
            nonlocal serial
            serial+=1;save(body);authority.send_signal(signal.SIGHUP);wait(ready)
        def cache(port=1054,upstream=1053,size=150,options=()):
            logfile=root/('cache-'+str(port)+'.log');stream=stack.enter_context(logfile.open('w'))
            command=['dnsmasq','--no-daemon','--conf-file=/dev/null','--no-resolv','--no-hosts','--bind-interfaces',
                '--listen-address=127.0.0.1','--port='+str(port),'--server=127.0.0.1#'+str(upstream),
                '--cache-size='+str(size),'--log-queries','--log-facility=-',*options]
            process=subprocess.Popen(command,stdout=stream,stderr=subprocess.STDOUT);stack.callback(stop,process)
            def cache_ready():
                assert process.poll() is None,logfile.read_text()
                return query('lab.test.','SOA',port).rcode()==0
            wait(cache_ready);return process,logfile
        def queries(name,kind='A'):
            return sum(bool(re.search(r'query: '+re.escape(name.rstrip('.'))+r' IN '+kind+r'\b',line)) for line in log.read_text().splitlines())
        yield root,update,cache,queries
def record(address,ttl):return f'move {ttl} IN A {address}'
with lab(record('127.0.0.2',8)) as (_,update,cache,queries):
    cache();before=queries('move.lab.test');started=time.monotonic();initial=query('move.lab.test.',port=1054)
    wait(lambda: queries('move.lab.test')==before+1)
    assert values(initial)==['127.0.0.2'] and initial.answer[0].ttl==8
    update(record('127.0.0.2',3));lowered=query('move.lab.test.');cached_after_lower=query('move.lab.test.',port=1054)
    assert lowered.answer[0].ttl==3 and cached_after_lower.answer[0].ttl>3
    update(record('127.0.0.3',3));new_authority=query('move.lab.test.');old_cache=query('move.lab.test.',port=1054)
    assert values(new_authority)==['127.0.0.3'] and values(old_cache)==['127.0.0.2']
    time.sleep(3.3);after_new_ttl=query('move.lab.test.',port=1054)
    assert values(after_new_ttl)==['127.0.0.2']
    observations=[]
    def changed():
        answer=query('move.lab.test.',port=1054);observations.append(show(answer));return values(answer)==['127.0.0.3']
    wait(changed,timeout=8);migration_delay=round(time.monotonic()-started,3)
    update(record('127.0.0.2',3));rollback_authority=query('move.lab.test.');rollback_cache=query('move.lab.test.',port=1054)
    assert values(rollback_authority)==['127.0.0.2'] and values(rollback_cache)==['127.0.0.3']
    wait(lambda: values(query('move.lab.test.',port=1054))==['127.0.0.2'],timeout=6)
    print(json.dumps({'initial':show(initial),'lowered_authority':show(lowered),'cache_after_lower':show(cached_after_lower),
        'after_address_change':{'authority':show(new_authority),'cache':show(old_cache)},'after_new_TTL_elapsed':show(after_new_ttl),
        'new_address_observed_after_seconds':migration_delay,'rollback':{'authority':show(rollback_authority),'cache':show(rollback_cache)},
        'rollback_eventually_observed':True},sort_keys=True))
"""
name='xbcatm-dns-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
    with TemporaryDirectory(prefix=name+'-') as directory:
        root=Path(directory);(root/'Dockerfile').write_text(recipe)
        built=docker('build','-t',name,str(root));assert built.returncode==0,built.stderr
        result=docker('run','--rm','-i','--name',name,'--network','none','--memory','384m','--pids-limit','128',name,
            '/usr/bin/python3','-',input=program)
        assert result.returncode==0,result.stderr
        print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
    docker('rm','-f',name);docker('image','rm',name)

First, move.lab.test points to 127.0.0.2 with TTL 8. Querying through dnsmasq stores that answer, and the authoritative log confirms one upstream query. The script then republishes the same address with TTL 3. Direct authority lookup sees 3, while the previously populated cache still reports a remaining TTL greater than 3. Lowering the source value has not rewritten existing cache state.

The authority next changes the address to 127.0.0.3, still with TTL 3. Direct queries see the new address immediately; the cache continues returning 127.0.0.2. After another 3.3 seconds, the old address remains cached. This deliberately waits longer than the newly configured TTL and demonstrates why that new number cannot be used as the expiry time for an earlier answer.

The final run first returns 127.0.0.3 about 7.35 seconds after its initial fill. Integer-second expiry and process scheduling affect this interval; it is not a portable eight-second stopwatch. Crucially, the existing copy survives the new three-second interval. The script then rolls the authority back to 127.0.0.2. An immediate cache lookup still returns 127.0.0.3, and a later lookup returns the restored address after that cached generation expires.

Prepare the overlap before changing discovery

Lower TTL while the old destination is still healthy, confirm that all relevant authoritative servers have the intended data, and allow the previous longer cache lifetime to drain before the planned move. Measure the resolvers and client populations that matter to the service. One query to one public resolver samples one view; it cannot prove that every user now holds the same record.

At cutover, verify the new endpoint through the real hostname and required protocols. Keep the old endpoint serving or safely forwarding traffic for the planned overlap. Address discovery, TLS identity, application state, and connection draining are separate responsibilities. DNS changes cannot repair a new server that lacks the correct certificate or move an established connection away from an old process.

Maintain a usable rollback window after the first successful test. Record when the new address became authoritative, its TTL, and which endpoints must remain available. If a client still reaches the old address, establish whether it performed a new lookup, used a cached answer, or reused an existing connection. Those observations lead to different fixes and different timing expectations.

RFC 8767 describes serving expired data under controlled conditions when refresh fails. This example does not enable that feature; its older answers are still within their ordinary cache lifetimes. Inspect the actual resolver policy before interpreting an expired answer. Planning with explicit cache generations and endpoint overlap is more reliable than treating the zone editor’s save time as global completion.

Further reading

Sources