XB Field Notes · XB Field Notes

InnoDB deadlocks roll back the whole victim transaction

Thu, 20 Apr 2028 00:00:00 +0000

An InnoDB deadlock occurs when transactions form a cycle of incompatible lock requests. With deadlock detection enabled, InnoDB chooses a victim and rolls back that entire transaction so another participant can proceed. The application receives error 1213 with SQLSTATE 40001. Retrying only the statement that reported the error omits earlier work that was rolled back with it.

When it breaks

Do not assign one rollback rule to every lock error. Under the default innodb_rollback_on_timeout=OFF, a lock-wait timeout rolls back the waiting statement while earlier statements in the transaction remain pending. Enabling that server option changes timeout rollback to the whole transaction. A deadlock already rolls back the whole victim transaction regardless of that default timeout behavior.

Victim selection is not a contract that a particular session always loses. Applications must inspect the actual error and retry the complete logical transaction with bounded attempts. Keep irreversible external effects out of an automatically replayed transaction unless they have a separate deduplication contract. A disconnected commit response is also not equivalent to a confirmed deadlock rollback.

Check it quickly

Use three fresh administrative MySQL 8.4 client sessions, A, B, and observer O, against a disposable instance. Start the clients in autocommit mode. Disable client automatic reconnection so session identity remains stable. The recorded run used MySQL 8.4.11. In O, create an unused fixture schema and verify detection is enabled and timeout rollback uses its default statement scope:

CREATE DATABASE xbcatm_deadlock_lab;
USE xbcatm_deadlock_lab;
CREATE TABLE accounts (id INT PRIMARY KEY, balance INT NOT NULL) ENGINE=InnoDB;
INSERT INTO accounts VALUES (1,100),(2,100);
SELECT @@innodb_deadlock_detect,@@innodb_rollback_on_timeout;

The settings should be one and zero. In A, start an operation adding ten to both accounts, initially locking account one:

USE xbcatm_deadlock_lab;
SET SESSION innodb_lock_wait_timeout=30;
SET @increment=10;
START TRANSACTION;
UPDATE accounts SET balance=balance+@increment WHERE id=1;

In B, start an operation adding one hundred, initially locking account two:

USE xbcatm_deadlock_lab;
SET SESSION innodb_lock_wait_timeout=30;
SET @increment=100;
START TRANSACTION;
UPDATE accounts SET balance=balance+@increment WHERE id=2;

In A, request the row B holds. This statement waits:

UPDATE accounts SET balance=balance+@increment WHERE id=2;

Before A’s thirty-second lock timeout, run the opposing request in B:

UPDATE accounts SET balance=balance+@increment WHERE id=1;

One session receives error 1213; identify it as the victim. The other update completes. Run this only in the surviving session:

COMMIT;

In O, inspect the committed state:

SELECT id,balance FROM accounts ORDER BY id;

Both balances are 110 if A survived, or 200 if B survived. The victim’s earlier update is absent too. The recorded run selected B, leaving 110 in both rows. In the victim’s existing session, its session variable still records the original increment. Retry the whole operation, now acquiring rows in consistent order:

SELECT @increment;
START TRANSACTION;
UPDATE accounts SET balance=balance+@increment WHERE id=1;
UPDATE accounts SET balance=balance+@increment WHERE id=2;
COMMIT;

In O, verify the combined outcome:

SELECT id,balance FROM accounts ORDER BY id;

Both balances must now be 210. Next, contrast a timeout. In O, reset the fixture:

UPDATE accounts SET balance=100;

In A, hold account one:

START TRANSACTION;
UPDATE accounts SET balance=balance+1 WHERE id=1;

In B, change account two and set a short wait:

SET SESSION innodb_lock_wait_timeout=1;
START TRANSACTION;
UPDATE accounts SET balance=balance+5 WHERE id=2;

Still in B, request A’s row:

UPDATE accounts SET balance=balance+5 WHERE id=1;

This produces error 1205 with SQLSTATE HY000. In B, inspect its own pending work:

SELECT id,balance FROM accounts ORDER BY id;

B reads 100 and 105. In O, committed readers still see 100 and 100:

SELECT id,balance FROM accounts ORDER BY id;

The timeout did not undo B’s earlier five-unit change. Explicitly discard the remaining transactions, first in B:

ROLLBACK;

Then in A:

ROLLBACK;

Finally, in O, confirm both original balances and remove the fixture:

SELECT id,balance FROM accounts ORDER BY id;
DROP DATABASE xbcatm_deadlock_lab;

The check distinguishes rollback scope through actual values, rather than relying only on an error message. Stop on unexpected results and roll back open fixture transactions before cleanup.

Reduce cycles without assuming they disappear

Acquire shared business resources in consistent order, keep transactions short, and use appropriate indexes to avoid locking more records than the operation needs. Range operations and secondary-index maintenance can add lock interactions beyond the two primary-key rows shown here. Consistent application ordering reduces a known cycle; it is not proof that every possible deadlock has been removed.

Use SHOW ENGINE INNODB STATUS to examine the latest detected deadlock, and capture broader history when incident frequency warrants it. Correlate the involved statements and lock order with the application transaction, not just the final failed statement. Retry complete work within a deadline, distinguish timeout semantics, and verify the business result before treating recovery as successful.

Sources

MySQL 8.4 InnoDB error handling, complete deadlock rollback and configurable timeout rollback scope, checked 2026-09-12.
MySQL 8.4 deadlock example, reproducing a cycle and examining the selected victim, checked 2026-09-12.
MySQL 8.4 InnoDB locking, record and range locks involved in concurrent operations, checked 2026-09-12.

dnsmasq separates forwarded cache entries from configured local names

Sat, 15 Apr 2028 00:00:00 +0000

dnsmasq can forward DNS questions upstream, retain supported answers in a bounded cache, and serve names configured locally. These are different sources of data. A repeated answer may be a cache hit or a local configuration result. Establishing which source supplied it is essential when a hostname disagrees with the authority or continues resolving after a cache flush.

When it breaks

Starting dnsmasq does not automatically make every application use it. Clients must direct their DNS queries through the intended listener, and some applications use other resolvers. Likewise, a correct local listener can forward to the wrong upstream or read unexpected host configuration. Diagnose the actual query path before enlarging the cache or restarting a daemon that never received the failing query.

A local name can deliberately override upstream data. Clearing the forwarding cache does not delete that configured name, so repeated flushing cannot make the upstream address win. Treat a local record as configuration with an owner and purpose. An answer with local TTL zero can still be returned repeatedly because dnsmasq generates it from configuration rather than retaining a forwarded response.

Cache capacity and freshness solve different problems. A larger cache can retain more eligible entries, but it does not extend their valid TTLs or make every possible query answerable during an upstream outage. New names and expired entries still need the appropriate resolution path. Cache-size tuning also does not directly specify the maximum number of simultaneous upstream queries or reserve application-level availability.

Check it quickly

Save the script as check-dnsmasq.py and run python3 check-dnsmasq.py with Docker. The pinned environment uses dnsmasq 2.90 and BIND 9.18.49. The test forwarders ignore ambient resolver and hosts files, bind explicit loopback listeners, and forward only to the disposable authority. They publish no host ports and use no DHCP configuration:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends bind9=1:9.18.49-1~deb12u2 bind9-dnsutils=1:9.18.49-1~deb12u2 dnsmasq-base=2.90-4~deb12u2 python3-dnspython=2.3.0-1 && rm -rf /var/lib/apt/lists/*\nCMD ["/usr/bin/python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,re,signal,subprocess,time
import dns.exception,dns.message,dns.query,dns.rcode,dns.rdatatype

def run(*args,check=True):
 result=subprocess.run(args,capture_output=True,text=True,timeout=15)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def query(name,kind='A',port=1053):
 return dns.query.udp(dns.message.make_query(name,kind),'127.0.0.1',port=port,timeout=1)
def values(response,kind='A'):
 wanted=dns.rdatatype.from_text(kind)
 return [item.to_text() for rrset in response.answer if rrset.rdtype==wanted for item in rrset]
def show(response):
 def section(records):return [{'name':str(rr.name),'type':dns.rdatatype.to_text(rr.rdtype),'ttl':rr.ttl,'data':[item.to_text() for item in rr]} for rr in records]
 return {'rcode':dns.rcode.to_text(response.rcode()),'answer':section(response.answer),'authority':section(response.authority)}
def wait(predicate,timeout=15):
 deadline=time.monotonic()+timeout
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,dns.exception.Timeout):pass
 time.sleep(.05)
 raise AssertionError('DNS fixture condition did not become true')
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=5)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=5)

@contextmanager
def lab(records,soa_ttl=8,minimum=4,extra=''):
 with TemporaryDirectory(prefix='xbcatm-dns-') as directory,ExitStack() as stack:
 root=Path(directory);zone=root/'zone';conf=root/'named.conf';log=root/'named.log';serial=1
 def save(body):
 zone.write_text(f'$ORIGIN lab.test.\n$TTL 30\n@ {soa_ttl} IN SOA ns.lab.test. hostmaster.lab.test. ({serial} 60 60 3600 {minimum})\n@ IN NS ns.lab.test.\nns IN A 127.0.0.1\n'+body+'\n')
 run('named-checkzone','lab.test',str(zone))
 save(records)
 conf.write_text(f'''options {{ directory "{root}"; listen-on port 1053 {{127.0.0.1;}}; listen-on-v6 {{none;}};
recursion no; dnssec-validation no; notify no; querylog yes; max-cache-size 8m; pid-file "{root}/named.pid"; session-keyfile none;
{extra} }};
controls {{}};
zone "lab.test" {{ type primary; file "{zone}"; }};
''')
 run('named-checkconf',str(conf))
 output=stack.enter_context(log.open('w'))
 authority=subprocess.Popen(['named','-g','-c',str(conf),'-n','1','-U','1'],stdout=output,stderr=subprocess.STDOUT)
 stack.callback(stop,authority)
 def ready():
 assert authority.poll() is None,log.read_text()
 response=query('lab.test.','SOA');return response.rcode()==0 and response.answer[0][0].serial==serial
 wait(ready)
 def update(body):
 nonlocal serial
 serial+=1;save(body);authority.send_signal(signal.SIGHUP);wait(ready)
 def cache(port=1054,upstream=1053,size=150,options=()):
 logfile=root/('cache-'+str(port)+'.log');stream=stack.enter_context(logfile.open('w'))
 command=['dnsmasq','--no-daemon','--conf-file=/dev/null','--no-resolv','--no-hosts','--bind-interfaces',
 '--listen-address=127.0.0.1','--port='+str(port),'--server=127.0.0.1#'+str(upstream),
 '--cache-size='+str(size),'--log-queries','--log-facility=-',*options]
 process=subprocess.Popen(command,stdout=stream,stderr=subprocess.STDOUT);stack.callback(stop,process)
 def cache_ready():
 assert process.poll() is None,logfile.read_text()
 return query('lab.test.','SOA',port).rcode()==0
 wait(cache_ready);return process,logfile
 def queries(name,kind='A'):
 return sum(bool(re.search(r'query: '+re.escape(name.rstrip('.'))+r' IN '+kind+r'\b',line)) for line in log.read_text().splitlines())
 yield root,update,cache,queries
with lab('cached 20 IN A 127.0.0.2\nuncached 20 IN A 127.0.0.3\nlocal 20 IN A 127.0.0.4') as (_,update,cache,queries):
 daemon,log=cache(options=('--host-record=local.lab.test,127.0.0.9',))
 for _ in range(3):assert values(query('cached.lab.test.',port=1054))==['127.0.0.2']
 wait(lambda: queries('cached.lab.test')==1)
 cache(port=1055,size=0)
 for _ in range(3):assert values(query('uncached.lab.test.',port=1055))==['127.0.0.3']
 wait(lambda: queries('uncached.lab.test')==3)
 local=query('local.lab.test.',port=1054);assert values(local)==['127.0.0.9'] and local.answer[0].ttl==0
 assert queries('local.lab.test')==0
 authoritative=query('local.lab.test.');assert values(authoritative)==['127.0.0.4']
 wait(lambda: queries('local.lab.test')==1)
 update('cached 20 IN A 127.0.0.8\nuncached 20 IN A 127.0.0.3\nlocal 20 IN A 127.0.0.4')
 stale=query('cached.lab.test.',port=1054);assert values(stale)==['127.0.0.2'] and queries('cached.lab.test')==1
 previous=log.read_text().count('cleared cache');daemon.send_signal(signal.SIGHUP)
 wait(lambda: log.read_text().count('cleared cache')>previous)
 refreshed=query('cached.lab.test.',port=1054);assert values(refreshed)==['127.0.0.8']
 wait(lambda: queries('cached.lab.test')==2)
 retained=query('local.lab.test.',port=1054);assert values(retained)==['127.0.0.9']
 print(json.dumps({'three_lookups_upstream_counts':{'cache_enabled':1,'cache_disabled':3},'local_record':show(local),
 'same_name_authority':show(authoritative),'before_SIGHUP':show(stale),'after_SIGHUP':show(refreshed),
 'configured_local_record_retained':show(retained)},sort_keys=True))
"""
name='xbcatm-dns-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 built=docker('build','-t',name,str(root));assert built.returncode==0,built.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--memory','384m','--pids-limit','128',name,
 '/usr/bin/python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

Three queries for cached.lab.test through the cache-enabled listener all return 127.0.0.2, while the authoritative log records one query. A second listener uses cache-size=0. Three queries for a separate name through it generate three upstream queries. This directly verifies the cache’s effect on forwarding work; the experiment does not infer hits from response speed or assert a particular latency improvement.

The first listener also has host-record=local.lab.test,127.0.0.9. Its answer is 127.0.0.9 with TTL zero, and no upstream query is logged for that lookup. A direct authoritative query for the same name returns 127.0.0.4 and increments the authority’s counter. The differing results show an explicit local override, with a positive control confirming that the query counter detects real upstream traffic.

Next, the authority changes cached.lab.test to 127.0.0.8. The cache still returns its earlier 127.0.0.2 answer within the remaining TTL. The script sends SIGHUP to this specific dnsmasq process and waits for its cache-cleared log entry. The following query reaches the authority and returns 127.0.0.8, increasing the corresponding authoritative count from one to two.

After that same signal, local.lab.test continues returning the configured 127.0.0.9 value. The signal clears forwarded cache state while leaving the explicitly configured local record available. This is why “flush DNS” is incomplete troubleshooting advice: the expected result depends on which process owns which data, and whether that data is cached, configured, or stored somewhere else entirely.

Operate the listener and its data sources deliberately

Inspect the effective configuration, listener address, upstream destinations, and client settings together. Avoid creating a forwarding loop between two local resolvers. The example removes ambient inputs to isolate behavior; a real installation may intentionally use hosts files, DHCP leases, or resolver-management software. Record those dependencies so an automatic rewrite or a local override is visible during troubleshooting.

For dnsmasq, SIGHUP clears its cache and reloads certain data sources described by the manual, but it does not generally reread the main configuration file. Use the documented service procedure for the kind of change being made. A successful cache clear cannot be treated as proof that edited listener, forwarding, or other startup configuration has been activated.

Limit listener exposure to intended clients, especially when forwarding recursion on a reachable network. DHCP, DNSSEC validation, and encrypted upstream transport require their own configuration and verification; none is demonstrated by this cache experiment. Query logging is useful during diagnosis but can reveal client naming activity, so select its scope and retention according to the environment rather than leaving detailed logs indefinitely by habit.

Finally, compare cached answers with the authority when it is the expected source. For a configured local name, compare against its configuration. Track upstream failures, hits, and entry churn separately. Identify whether the discrepancy concerns capacity, lifetime, the upstream service, or a local naming rule.

Sources

dnsmasq manual, forwarding, cache-size, host-record, local TTL, and SIGHUP behavior, checked 2026-09-12.
RFC 1035 domain names, DNS resource records and cached lifetimes, checked 2026-09-12.

LVM snapshots capture a block device in time

Mon, 10 Apr 2028 00:00:00 +0000

An LVM snapshot is neither a copy nor a reflink: it is a fixed-size copy-on-write exception store attached to an origin logical volume. lvcreate -s -L 5G -n snap vg/data allocates 5 GiB from the volume group, and the device-mapper snapshot target records the original contents of each origin chunk the first time it is written. Creation is instant whatever the origin’s size, because nothing is copied.

That size is the whole safety margin, and it is the parameter people get wrong. A store sized for a quiet week is exhausted in an hour by a database checkpoint or an rsync, and when it fills the snapshot does not degrade — it becomes invalid and returns I/O errors. The recovery point disappears exactly when you needed it, which is why the store’s free space belongs on a dashboard, not in a runbook.

An exception store, not a copy

A classic LVM snapshot uses two device-mapper targets: the origin is re-mapped as snapshot-origin, and a new snapshot device combines it with a COW device. Reads of unchanged chunks go straight to the origin; the first write to a chunk copies the old chunk into the COW device, then writes the new data. So consumption tracks the number of distinct chunks that changed, not the number of writes: the simulation writes a million times to sixteen hot chunks and consumes fifteen exceptions, about 60 KiB.

Watch it with lvs -a -o lv_name,lv_attr,origin,data_percent,lv_size. A classic snapshot’s lv_attr reads swi-a-s--- — s for snapshot, w writable, and a state character that becomes I when the store is exhausted. data_percent is the store’s fill, and that is the number to alert on, at 70 or 80 percent; 100 percent is not a warning, it is an obituary. lvextend grows the COW device while the snapshot is still valid; lvchange --monitor y lets dmeventd do it automatically.

Exhaustion is invalidation

The kernel documentation is blunt: the COW device will often be smaller than the origin, and if it fills up the snapshot becomes useless and is disabled, returning errors. LVM’s guidance is to plan for expected writes — often 20 percent of the origin — and extend before it fills. A small slice of the store holds the COW block map, so usable data capacity is a little under the -L value.

The blast radius is narrow but the timing is cruel: the origin keeps running normally, so nothing looks wrong until the backup that reads the snapshot fails or the rollback is unavailable. There is no partial mode: at 100 percent, the snapshot never existed.

Merging, and why thin changes the failure

lvconvert --merge vg/snap rolls the snapshot’s changed chunks back into the origin, and it runs in the background while the origin stays usable — writes to chunks not yet merged are deferred. The constraint is on the origin: it must be inactive when the merge is set up, so unmount it, or schedule the merge and activate the origin afterwards. Merging consumes the snapshot, so copy out anything you need first.

Thin snapshots change the failure mode rather than removing it. A thin snapshot is created when the origin is a thin LV and no size is given; it shares blocks in the thin pool instead of owning a store, so its own divergence cannot invalidate it. The pool is the shared resource now: when it runs out, the damage is not one dead snapshot but every thin volume in the pool, which LVM warns may be damaged and hard to repair. thin_pool_autoextend_threshold and dmeventd exist precisely because pool exhaustion is a pool-wide outage.

When it breaks

The classic failure is a backup that silently stops being a backup. The job mounts the snapshot, copies for longer than the store can cover, and the store reaches 100 percent; the snapshot goes invalid and the copy fails — or worse, monitoring watches only the exit code while the job reads a stale mount. Check data_percent on every snapshot LV, alert at 70 percent, and size for the busiest hour in the retention window.

The second failure is the merge that cannot happen when you need it. lvconvert --merge requires the origin closed, and teams discover that during the incident. Rehearse the merge on a disposable volume, keep a real backup independent of the snapshot, and remember that a snapshot whose store is full cannot be merged.

Check it quickly

Look at the store’s fill before trusting the snapshot:

lvs -a -o lv_name,vg_name,lv_attr,origin,data_percent,lv_size
lvs -a -o lv_name,data_percent,metadata_percent vg/thinpool
dmsetup status vg-snap
lvextend -L +2G vg/snap

data_percent is the COW store’s fill and should sit well under 100; a state character of I in lv_attr means the snapshot is already invalid. For a thin snapshot, data_percent and metadata_percent on the pool are the numbers that matter; dmsetup status shows allocated sectors against the total so you can watch a merge drain.

Sources

lvcreate(8), documents -s/--snapshot, COW versus thin snapshots and the sizing guidance, checked 2026-09-16.
lvs(8), defines the lv_attr state characters including invalid snapshot, checked 2026-09-16.
Device-mapper snapshot support, the exception store, invalidation and lvconvert --merge behaviour, checked 2026-09-16.
lvmthin(7), thin snapshots, pool exhaustion and automatic extension, checked 2026-09-16.

Consul keeps a live map of healthy services

Thu, 06 Apr 2028 00:00:00 +0000

A service registry is only as useful as its honesty. Consul keeps a catalog of registered instances and attaches health checks to them, so a caller asking where a healthy instance of a service lives gets an answer that changes as reality changes. Registration is trivial; the check is the whole product, because a registry that lists dead instances is worse than a static list — it actively routes traffic into a hole.

The freshness of that answer is a function of two choices nobody revisits after the first week: how often the check runs, and what it actually tests. A check that runs every 30 seconds is blind for 30 seconds, and a check that only proves the process exists is blind forever.

Registration, and two levels of checks

An instance registers a name, an address, a port and one or more checks, through a config file or the registration API. Checks start in a critical state by default, deliberately, so a service cannot enter the pool before its health has been verified at least once — an initial passing status is something you opt into explicitly.

Two levels of check sit on the same node. A node-level check such as serfHealth carries an empty service name and describes the machine; a service-level check such as service:web-1 is bound to one registration. They fail independently, but the healthy-only filter requires every check on the node to pass, so a critical node-level check removes all of that machine’s services at once while a critical service check removes exactly one instance.

What a caller actually receives

The catalog answers two different questions. Unfiltered, /v1/health/service/web returns every registration with each check’s status; with passing=true it returns only the instances where all checks pass. DNS and SRV always answer the second way, so a resolver never sees a name it cannot use, and the passing set is the load-balancing pool.

The catalog is also eventually consistent, and the mechanism is anti-entropy. Registration is immediate — the agent tells the leader — but repair is periodic: each agent reconciles its local state with the catalog on an interval of one minute for clusters up to 128 nodes, growing with cluster size, and each agent staggers its start so they do not all sync at once. A sync that fails is logged and retried next period, so a wrong catalog entry can survive quietly for up to one interval. The read path has its own trade: by default the agent forwards catalog reads to the leader using a leader lease that assumes leadership is briefly stable, which is fast and can occasionally return a stale result, while the consistent mode makes the leader verify with a quorum first and stale lets any server answer without checking.

When it breaks

The first failure is the interval itself. A check is only as fresh as its period, so a service that dies right after a passing probe stays in rotation until the next probe — 30 seconds for a 30-second check, 60 for a 60-second one. A TTL check is worse than a slow interval because it is passive: the instance reports in on its own schedule, so a dead instance with a 120-second TTL keeps its passing status for the full 120 seconds.

The second failure is a check that answers the wrong question. A script that runs pgrep against the process reports passing for the entire duration of a deadlock — the process exists, the port is bound, and every request routed to it hangs. The catalog cannot tell that instance from a healthy one, so the failure is silent and total for every caller in the pool. A check that exercises a dependency, even imperfectly, converges on the truth within one interval; a liveness-only check never does. The direction of the error is what makes it expensive: a check that fails when it should pass removes a healthy instance and costs capacity, while a check that passes when it should fail keeps feeding traffic to something that cannot serve it, and the caller only finds out through its own error rate.

Check it quickly

Compare the filtered and unfiltered views, and see what DNS hands out:

consul catalog services
consul health service web
curl -s 'http://127.0.0.1:8500/v1/health/service/web' | jq -r '.[] | "\(.Service.ID) \(.Checks[] | select(.Status!="passing") | .CheckID + "=" + .Status)"'
curl -s 'http://127.0.0.1:8500/v1/health/service/web?passing=true' | jq -r '.[] | "\(.Service.ID) \(.Node.Address):\(.Service.Port)"'
dig +short SRV web.service.consul

An instance that shows up unfiltered with a failing CheckID must be absent from the passing=true list and from the SRV answer; if it is still there, the check is not being evaluated at all. A node whose node-level check is critical drops out of both lists entirely, taking every service on that machine with it.

Sources

Service discovery, the catalog, registration and the role of health checks, checked 2026-09-16.
Health checks, check types, intervals and the default critical initial state, checked 2026-09-16.
Health HTTP API, /v1/health/service and the passing filter, checked 2026-09-16.
Anti-entropy, the periodic sync interval and its failure behaviour, checked 2026-09-16.

The scheduler accounts for requests, not momentary utilization

Sun, 02 Apr 2028 00:00:00 +0000

Kubernetes scheduling answers whether a Pod can fit a node under the applicable resource requests and placement constraints. It does not simply look at a current CPU utilization chart and place work wherever the line is low. Requests provide accounting inputs for future demand; actual usage is a separate observation. Diagnose both before deciding that an idle-looking cluster has capacity for a particular Pod.

Account for the admitted specification

For ordinary application containers, the scheduler accounts for their requested resources when testing node fit. Initialization requirements, Pod overhead and supported Pod-level resource configuration can also affect the effective request. The final admitted Pod is the relevant input, including defaults and mutations, rather than just the source manifest that an operator remembers submitting.

Node allocatable is also different from the hardware’s total capacity. It represents the resource amount available to Pods under the node’s configuration. Other admitted workloads consume scheduling headroom through their requests even when their processes happen to be idle. A request does not mean that physical memory has already been allocated to the application or that CPU cores are exclusively assigned.

Limits constrain running workloads through the execution stack; they are not generally substituted for an explicitly smaller request in scheduler fit checks. A container can therefore have a CPU limit larger than one node’s CPU capacity and still schedule with a small request. That accepted limit does not manufacture additional processors or promise the process can reach the declared ceiling.

When it breaks

A Pending Pod may be blocked by insufficient requested resources, but also by taints, affinity, storage, topology or other constraints. Read the PodScheduled condition and scheduling events before changing resource values. Lowering requests cannot repair an incompatible node selector or unavailable volume, and blindly lowering an honest request can create runtime contention instead of a sound placement.

Defaulting makes omission easy to misread. When a limit is specified without a request, and no admission-time mechanism supplies another request, Kubernetes copies that resource’s limit into its request. Setting only a large limit can therefore produce a large scheduling demand. Inspect both maps on the stored object, including any LimitRange effects, before interpreting a failed placement.

The API also rejects a request greater than its corresponding limit. That is an admission failure, not a scheduler decision about cluster capacity. Keeping those failure stages separate helps explain why one manifest never creates a Pod while another produces an accepted but unscheduled Pod that can be inspected.

QoS is not an immunity promise. Under conventional per-container rules, Guaranteed requires CPU and memory requests and limits to be present and equal for every relevant container. Matching only memory is insufficient. Even a Guaranteed Pod can encounter resource failure; the class does not replace priority, capacity planning, application correctness or actual node-pressure analysis.

Check it quickly

Use a disposable Kubernetes cluster with at least one Ready, untainted worker, kubectl and python:3.11-slim-bookworm on its nodes. Save this as request-placement.py and run python3 request-placement.py KUBECONFIG_PATH CONTEXT with explicit test-cluster arguments. The account needs namespace creation and deletion permissions. The recorded server was Kubernetes 1.34.11, and the selected worker reported ten allocatable CPUs.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

from decimal import Decimal, ROUND_CEILING

def cpu_quantity(value):
 return Decimal(value[:-1]) / 1000 if value.endswith('m') else Decimal(value)

namespace = 'notes-requests-' + uuid4().hex[:8]
k('create', 'namespace', namespace)
try:
 workers = [n for n in obj(namespace, 'nodes')['items'] if
 not n['spec'].get('unschedulable') and not n['spec'].get('taints')
 and any(c['type'] == 'Ready' and c['status'] == 'True' for c in n['status']['conditions'])]
 assert workers, 'a Ready worker without taints is required'
 worker = workers[0]
 allocatable = cpu_quantity(worker['status']['allocatable']['cpu'])
 too_large = str(int(allocatable.to_integral_value(rounding=ROUND_CEILING)) + 1)
 for name, request in [('burst', '10m'), ('oversized', too_large)]:
 podspec = spec()
 podspec['nodeSelector'] = {'kubernetes.io/hostname': worker['metadata']['labels']['kubernetes.io/hostname']}
 podspec['containers'][0]['resources'] = {
 'requests': {'cpu': request, 'memory': '24Mi'}, 'limits': {'cpu': too_large, 'memory': '96Mi'}}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': name}, 'spec': podspec})
 burst = wait_for(lambda: obj(namespace, 'pod', 'burst'), pod_ready)
 assert burst['spec']['nodeName'] == worker['metadata']['name']
 def insufficient(p):
 return any(c['type'] == 'PodScheduled' and c['status'] == 'False'
 and c.get('reason') == 'Unschedulable' and 'Insufficient cpu' in c.get('message', '')
 for c in p.get('status', {}).get('conditions', []))
 pending = wait_for(lambda: obj(namespace, 'pod', 'oversized'), insufficient)
 assert pending['status']['phase'] == 'Pending' and not pending['spec'].get('nodeName')
 only_limit = spec()
 only_limit['containers'][0]['resources'] = {'limits': {'cpu': '200m', 'memory': '64Mi'}}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'defaulted'}, 'spec': only_limit})
 defaulted = wait_for(lambda: obj(namespace, 'pod', 'defaulted'), pod_ready)
 resources = defaulted['spec']['containers'][0]['resources']
 assert resources['requests'] == resources['limits'] == {'cpu': '200m', 'memory': '64Mi'}
 assert defaulted['status']['qosClass'] == 'Guaranteed'
 invalid = spec()
 invalid['containers'][0]['resources'] = {'requests': {'cpu': '200m'}, 'limits': {'cpu': '100m'}}
 denied = k('create', '-n', namespace, '-f', '-', payload={
 'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'invalid'}, 'spec': invalid}, expected=None)
 assert denied.returncode != 0 and 'must be less than or equal to cpu limit' in denied.stderr
 print(json.dumps({'node_allocatable_cpu': str(allocatable), 'limit_cpu': too_large,
 'small_request_with_large_limit_scheduled': True,
 'larger_than_node_request_pending': True,
 'limit_only_defaulted_resources': resources,
 'defaulted_qos': defaulted['status']['qosClass'],
 'request_above_limit_rejected': True}))
finally:
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

The script uses nodeSelector to target that worker while leaving the scheduler responsible for assignment. Both comparison Pods have an eleven-CPU limit. The first requests only ten millicpu and becomes Ready on the selected node. The second requests eleven CPUs, remains Pending without nodeName, and reports Unschedulable with Insufficient cpu. Neither Pod tries to consume eleven CPUs; their processes are idle.

A third Pod specifies only 200m CPU and 64Mi memory limits. The stored object contains identical requests, and its observed QoS class is Guaranteed. A fourth attempted Pod requests 200m while limiting CPU to 100m; the API rejects it. All assertions passed, and the namespace was removed.

This separates successful placement, scheduler rejection, defaulting and API validation with real objects. It does not test actual peak capacity, resource-manager exclusivity or every scheduling plugin. Size requests from workload evidence and review actual admitted demand alongside usage, instead of turning an idle chart or a familiar ratio into a placement guarantee.

Sources

Kubernetes: Resource Management, request-based placement, limits and admission defaulting, checked 2026-09-13.
Kubernetes: Pod Quality of Service Classes, Guaranteed requirements and the limits of QoS protection, checked 2026-09-13.

Baggage carries context across an explicit boundary

Mon, 27 Mar 2028 00:00:00 +0000

OpenTelemetry baggage is a context-associated collection of string key-value pairs. It lets information known near a request’s entry point travel to downstream operations: a deployment region or a rollout cohort, for example. Unlike a span attribute, baggage is intended for propagation. Unlike trace context, it describes application-selected information rather than the identifiers and flags connecting spans. These mechanisms can travel together without being the same data structure.

Propagation still needs a working boundary. An outgoing client injects context into a carrier such as HTTP headers; the receiver extracts that carrier into its local context. Instrumentation can perform these steps automatically when configured, while manually instrumented code must do them explicitly. Merely creating baggage in one process does not deliver it to an unrelated process, a background task, or an external service.

Choose what crosses and what gets recorded

Baggage entries do not automatically become span, log, or metric attributes. A downstream operation must read the value and copy it deliberately, or use a configured processor implementing that policy. This distinction matters when a team expects a searchable attribute but has only enabled propagation. It also prevents the opposite mistake: copying every incoming key into telemetry without considering its meaning, size, or cardinality.

Context operations return a new context rather than changing every existing reference. Keep the returned context and pass or attach it at the intended boundary. When removing entries for an outgoing request, preserve the trace context that should accompany that request. Treat an outbound allowlist and a telemetry-attribute allowlist as explicit decisions; the values appropriate for one receiver need not be suitable for every later destination.

When it breaks

Baggage has no built-in proof that a value was issued by a trusted component. A caller can supply a header claiming an expensive account tier or a privileged role. Parsing that header successfully establishes its representation, not its authority. Authentication and authorization must use their own trusted inputs. Small, plausible-looking values deserve the same trust analysis as obviously suspicious ones.

Unbounded baggage also increases request-header sizes and can spread sensitive data to unintended recipients, including third-party APIs reached by automatic instrumentation. Limit names and values, review the destinations that receive them, and use propagators for wire encoding. A successful round trip between two copies of one SDK is narrower evidence than interoperability across every language and proxy in a production path. Test those actual boundaries when they matter.

Check it quickly

Create a virtual environment, install opentelemetry-sdk==1.35.0, save the script as baggage.py, and execute it with that environment’s Python. This exercise ran on Python 3.14.4. It uses the real W3C trace-context and baggage propagators with an in-memory span exporter. It does not start an HTTP server: the dictionary is the carrier that an HTTP client would subsequently send.

import json

from opentelemetry import baggage, trace
from opentelemetry.context import Context
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

base = Context()
raw = baggage.set_baggage("region", "eu-west", context=base)
raw = baggage.set_baggage("release", "canary/2", context=raw)
raw = baggage.set_baggage("internal_note", "synthetic-only", context=raw)
assert not baggage.get_all(base)
propagator = CompositePropagator([TraceContextTextMapPropagator(), W3CBaggagePropagator()])
provider = TracerProvider()
exporter = InMemorySpanExporter()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("baggage-lab")
with tracer.start_as_current_span("parent", context=Context()) as parent:
 outgoing = trace.set_span_in_context(parent, baggage.clear(context=raw))
 for key in ("region", "release"):
 outgoing = baggage.set_baggage(key, baggage.get_baggage(key, raw), context=outgoing)
 carrier = {}
 propagator.inject(carrier, context=outgoing)
 received = propagator.extract(carrier, context=Context())
 assert set(baggage.get_all(received)) == {"region", "release"}
 assert baggage.get_baggage("region", received) == "eu-west"
 assert baggage.get_baggage("internal_note", raw) == "synthetic-only"
 assert trace.get_current_span(received).get_span_context().trace_id == parent.get_span_context().trace_id
 with tracer.start_as_current_span("inherited", context=received):
 pass
 with tracer.start_as_current_span("copied", context=received) as span:
 span.set_attribute("deployment.region", baggage.get_baggage("region", received))
assert provider.force_flush()
spans = {span.name: span for span in exporter.get_finished_spans()}
assert not spans["inherited"].attributes
assert spans["copied"].attributes == {"deployment.region": "eu-west"}
independent = propagator.extract({"traceparent": "invalid", "baggage": "region=eu-west"},
 context=Context())
assert not trace.get_current_span(independent).get_span_context().is_valid
assert baggage.get_baggage("region", independent) == "eu-west"
untrusted = propagator.extract({"baggage": "tier=gold"}, context=Context())
assert baggage.get_baggage("tier", untrusted) == "gold"
print(json.dumps({"wire_fields": sorted(carrier),
 "wire_baggage": carrier["baggage"],
 "automatic_span_attributes": dict(spans["inherited"].attributes),
 "explicit_span_attributes": dict(spans["copied"].attributes),
 "baggage_survives_invalid_traceparent": True,
 "caller_supplied_tier_was_parsed": True}, indent=2))
provider.shutdown()

The original context contains region, release, and a synthetic internal note. The outgoing context keeps only the first two entries while retaining the parent span. The extracted region is eu-west, and the release value canary/2 round-trips through the encoded header. Assertions also confirm that constructing the filtered context did not remove the note from the original context or change the empty base context.

Two child spans demonstrate the recording boundary. inherited receives baggage but has no attributes; copied contains only the explicitly assigned deployment.region. A malformed traceparent is tested separately: its trace context is invalid, while the independently encoded baggage still extracts successfully. Finally, a caller-supplied tier=gold is accepted as data without any identity check. These results establish serialization, filtering, and explicit recording behavior. Before relying on the same policy across services, verify the real client instrumentation, receiver extraction, outbound filtering, and the attribute mapping used by the destination backend.

Sources

OpenTelemetry baggage, baggage is separate from telemetry attributes and lacks built-in integrity checks, checked 2026-09-13.
W3C Baggage, HTTP representation, value encoding, and propagation limits, checked 2026-09-13.
W3C Trace Context, trace identity travels separately from application baggage, checked 2026-09-13.

The TCP backlog holds connections that are still syncing

Fri, 24 Mar 2028 00:00:00 +0000

A listening TCP socket has two queues, not one, and the sysctl this article is named after governs the queue that almost never drops anything. The backlog passed to listen(2) is the accept queue, holding connections that finished the handshake and await the application; the SYN queue holds half-open requests. Confusing them produces the wrong fix.

That matters because the famous advice, raise tcp_max_syn_backlog, is largely obsolete: tcp_syncookies defaults to 1, and as listen(2) puts it, with syncookies enabled there is no logical maximum length and that setting is ignored. What actually bites is the accept queue and how fast the application calls accept().

The backlog argument is the accept queue

listen(2) is explicit that the behaviour changed with Linux 2.2: backlog now specifies the queue length for completely established sockets waiting to be accepted, not the number of incomplete requests. The incomplete queue is bounded separately by /proc/sys/net/ipv4/tcp_max_syn_backlog. And if the backlog argument exceeds /proc/sys/net/core/somaxconn it is silently capped to that value, 4096 since Linux 5.4 and 128 before. So listen(8192) on a default host gives you 4096, and nothing logs it.

tcp(7) describes tcp_max_syn_backlog as the maximum number of queued requests that have not received an acknowledgement, defaulting to 256, raised to 1024 when memory is at least 128 MB, cut to 128 below 32 MB. ip-sysctl adds the qualifications: it is a per-listener limit, each half-open request costs about 304 bytes, and tcp_syncookies defaults to 1, sending a cookie on overflow instead of dropping.

With syncookies on, the SYN queue stops mattering

ip-sysctl documents tcp_syncookies as a fallback that must not be used to help a loaded server stand against legal connection throughput, and says syncookies seriously violate the protocol, disallow TCP extensions, and can degrade services. listen(2) goes further: with syncookies enabled there is no logical maximum length and tcp_max_syn_backlog is ignored. tcp(7) still describes the same knob as dropping requests once exceeded, which is the syncookies-off behaviour. Both pages are right about different configurations.

The honest reading is therefore: leave syncookies at 1 and stop tuning the SYN queue. The cost is that the fallback loses TCP options such as window scaling and SACK on the connections it rescues, which is a throughput problem rather than a connectivity one. If you see SYN-flood warnings while the traffic is legitimate, ip-sysctl’s own advice is to tune the other knobs, not to disable syncookies.

The accept queue is where the pressure lands

With the SYN queue neutralised, the queue that fills is the accept queue, sized by min(listen(backlog), somaxconn). Overflow behaviour depends on tcp_abort_on_overflow, default FALSE: ip-sysctl says the default lets a connection recover if the overflow was a burst, and warns that enabling the abort option can harm clients. The mechanism is the surprise: by default the kernel silently discards the final ACK, so connect() succeeds and the failure only surfaces when the first request stalls or is reset.

A bigger backlog is a delay line, not capacity. In the model, listen(128) and listen(65535) accept the same 20,000 connections in a second; the larger backlog only postpones the overflow, because the accept rate is set by the worker count.

When it breaks

The first failure is the silent one, and it is the default. A service whose accept loop is too slow overflows the accept queue, the kernel drops the final ACKs, and clients see connections that establish and then hang. Nothing is refused, so a client connection pool fills with dead sockets and the error surfaces as timeouts somewhere unrelated. Watch TcpExtListenOverflows and TcpExtListenDrops: those counters climbing while the accept rate is flat means the handler is the limit.

The second failure is the dropped SYN, which needs syncookies off to happen. A full SYN queue then drops the request and the client retransmits on a schedule whose documented endpoints are 67 seconds to the last retransmission and 131 seconds to the final timeout, so the user-visible symptom is a connect that hangs for over two minutes and then fails. Raising tcp_max_syn_backlog is a real fix in that configuration, and turning syncookies back on is a better one, if you accept the loss of TCP extensions.

Check it quickly

ss -lnt 'sport = :8443'
nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtTCPReqQFullDrop TcpExtTCPReqQFullDoCookies
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog net.ipv4.tcp_syncookies net.ipv4.tcp_abort_on_overflow

In ss -lnt, Send-Q is the backlog the socket was created with and Recv-Q is how many established connections are queued and unaccepted, so Recv-Q sitting at Send-Q is the queue being full. The nstat counters should be zero or near it: TcpExtListenOverflows and TcpExtListenDrops are accept-queue events, while TcpExtTCPReqQFullDrop is a dropped SYN and TcpExtTCPReqQFullDoCookies is the fallback doing its job. The sysctl line shows what the kernel will honour, and if somaxconn is smaller than your listen() argument then that smaller number is your real backlog.

Sources

Kernel ip-sysctl documentation, somaxconn, tcp_max_syn_backlog, tcp_abort_on_overflow and the syncookies warning, checked 2026-09-16.
Linux listen(2), the accept queue since Linux 2.2, the silent somaxconn cap, and syncookies removing the SYN-queue bound, checked 2026-09-16.
Linux tcp(7), tcp_max_syn_backlog defaults by memory size and the syncookies caveats, checked 2026-09-16.

ZFS snapshots give the filesystem a rewind button

Tue, 21 Mar 2028 00:00:00 +0000

A ZFS snapshot is a name attached to a dataset’s block tree at one transaction group. Nothing is copied, nothing is quiesced, and the cost is a few metadata blocks: zfs snapshot tank/data@before-upgrade returns before the shell prompt redraws. Because ZFS never overwrites a live block in place, the old tree stays valid on disk until the last reference to it goes away, and the snapshot is that reference.

That is what makes a nightly wall of history practical, and it is also where this article’s title gets dangerous. Snapshots give you many versions of the data; they do not give you safety from losing the pool. Every snapshot of tank/data lives in tank, so the failure that takes the pool takes the dataset and its history in the same instant. Multiversion and multiversion-safety are different properties, and only replication supplies the second.

A name for one consistent tree

Snapshots are created at a transaction group boundary, so the tree they capture is internally consistent — a database’s files and its metadata freeze at the same instant, not microseconds apart. -r snapshots a dataset and all its descendants, which is what you want for anything holding child datasets; without it, a recursive layout gets snapshots taken at different times at different levels, and incremental replication between them becomes awkward.

Names are the retention policy’s only handle: tank/data@daily-2026-09-16 is prunable by pattern, while an autogenerated name needs bookkeeping. zfs list -t snapshot -o name,used,refer,creation -s creation shows the set with the space each one pins.

Where the space is, in zfs list -o space

ZFS reports space in a column set that answers the only question that matters: whose data is it? zfs list -o space tank/data gives USED, USEDSNAP, USEDDS, USEDREFRESERV, USEDCHILD, REFER and AVAIL. REFER is what the dataset alone needs, USEDSNAP is what its snapshots pin, and USED is the sum. A rising USEDSNAP with a flat REFER means the snapshots, not the data, are eating the pool.

The fixture builds a wall of thirty daily snapshots over a 64 MiB dataset with 2 MiB of churn per day. Creation costs 0 KiB every single time — the cost lands on the write that follows, 2048 KiB for 2 MiB rewritten — and the whole wall costs 60 MiB against 1920 MiB for thirty full copies. That ratio is the entire argument for snapshots, and why nobody notices the pool filling until it is nearly full.

Holds, diff, and comparing versions

A hold is a named reference that makes destruction fail with EBUSY: zfs hold keep tank/data@daily-2026-09-16 protects a snapshot, zfs holds lists the holders, and zfs release keep clears it. Backup software holds the snapshot it is sending so retention cannot delete it; the cost is that pruning then fails until the hold is released, and a stuck hold can pin a snapshot for months.

zfs diff tank/data@daily-2026-09-15 tank/data@daily-2026-09-16 reports per-path changes — + created, - removed, M modified, R renamed — and both snapshots must still exist, which is why diffing against yesterday is easy and against last month may not be. Read a snapshot’s contents through the hidden .zfs/snapshot/<name> directory; there is no separate mount for a snapshot, and snapdir=visible makes it listable.

When it breaks

The failure that matters is the one the title invites: a pool loss. A failed controller, a backplane fault, or pool-wide corruption removes the dataset and every snapshot of it atomically; there is no partial survival. The only defence is a copy outside that pool: zfs send -i tank/data@yesterday tank/data@today | zfs receive backup/data writes an independent, checksummed stream into a second pool, and only that stream survives the first pool’s death. A snapshot wall with no replica is a single point of failure with a nicer interface.

The second failure is silent space creep plus retention that cannot run. USEDSNAP grows while REFER stays flat, the pool passes 80 percent, and ZFS starts refusing writes with no space left even though the datasets look reasonable — slop and reservations are not visible from inside the filesystem. If a snapshot is also held for a clone, destroying it fails, and rolling back past a clone’s origin forces a recursive destroy. Long-lived holds turn retention into a dependency graph, so keep hold lifetimes short and audited.

Check it quickly

See what the snapshots cost and whether they can be removed:

zfs list -t snapshot -o name,used,refer,creation -s creation tank/data
zfs list -o space tank/data
zfs holds tank/data@daily-2026-09-16
zfs diff tank/data@daily-2026-09-15 tank/data@daily-2026-09-16
zfs send -i tank/data@daily-2026-09-15 tank/data@daily-2026-09-16 | zfs receive backup/data

used per snapshot is the space that snapshot alone pins, and a fresh snapshot’s value is near zero; the space columns then show USEDSNAP against REFER, and a rising USEDSNAP with a flat REFER is the signal to prune. If zfs destroy fails with a busy snapshot, zfs holds names the tag to release. The last line is the only one that protects against losing the pool; run it on a schedule.

Sources

zfs-snapshot(8), snapshot creation semantics and recursive snapshots, checked 2026-09-16.
zfs-list(8), the space column set including USEDSNAP, USEDDS and REFER, checked 2026-09-16.
zfs-hold(8), holds, zfs holds, and the EBUSY returned by destroy, checked 2026-09-16.
zfs-send(8), incremental replication streams that leave the pool, checked 2026-09-16.

Hugo publication flags decide what enters the artifact

Sat, 18 Mar 2028 00:00:00 +0000

Hugo decides which pages to render when a build runs. Front matter supplies publication metadata, configuration and command flags supply inclusion rules, and the build clock supplies the time used for evaluation. The deployed directory then stays unchanged until another artifact replaces it. A date reaching midnight is not itself a deployment event, and a file without draft: true is not automatically a draft.

Separate publication metadata from build options

Under the default rules, a page marked draft: true is excluded. A future publication date also excludes it, while an expired page is excluded after its expiry date. Hugo resolves dates according to its front matter configuration; an explicit publishDate can differ from the displayed date. Use complete timestamps with a time zone for a boundary that must mean the same thing on a laptop and a build worker.

--buildDrafts includes drafts, --buildFuture includes future content, and --buildExpired includes expired content. Their short forms are -D, -F, and -E. These switches address separate conditions: a future draft needs both draft and future inclusion before the date arrives. Project or environment configuration can enable the same behavior, so checking only a page’s front matter is insufficient.

This site’s configuration deliberately enables buildFuture: true to retain its published article inventory, including articles dated in the future. Consequently, those dates are not an embargo here. A tutorial about Hugo defaults must not be mistaken for a claim about this site’s active settings. Compare the effective options with the expected URL inventory before changing publication policy.

A preview artifact can still be published

A preview flag changes the generated output, not its sensitivity or deployability. If a pipeline uploads a directory built with -D -F, those pages can become public. Use a separate destination for preview builds and construct the release artifact with the intended options. A Git release tag identifies source history; it neither changes Hugo’s inclusion rules nor certifies that a particular generated directory was reviewed.

Scheduled publication requires a later build and deployment after the publication boundary. Conversely, changing a public page back to draft cannot withdraw HTML that is already deployed or cached. Rebuild, replace the served artifact, and inspect the public route. Hugo normally leaves unrelated files in an existing destination, so a fresh output directory or an appropriate clean build is part of verifying that excluded pages are absent.

When it breaks

A preview build copied over a release directory can leave a draft accessible by its direct URL even after the navigation no longer lists it. Inspect generated files, feeds, sitemaps, and the actual served route; hiding an index entry is not removal. A draft section page also does not automatically make all descendants drafts. Use explicit child metadata or the documented cascade mechanism when exclusion must apply to a subtree.

Time-zone mismatches cause a different failure: two workers include different pages from the same commit. Pin the generator, use an explicit build clock for boundary tests, and record the options. Do not repair the discrepancy by changing publication dates until the date resolution rules and worker time zones are understood.

Check it quickly

With Python 3 and Hugo 0.165.0 available, run this isolated experiment. It creates three pages in a temporary project and builds five independent destinations without touching this repository or deploying anything.

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess

with TemporaryDirectory(prefix="hugo-publication-") as directory:
 root = Path(directory)
 (root / "content/posts").mkdir(parents=True)
 (root / "layouts/_default").mkdir(parents=True)
 (root / "hugo.yaml").write_text("baseURL: https://example.org/\n")
 (root / "layouts/index.html").write_text("<h1>Publication test</h1>")
 (root / "layouts/_default/list.html").write_text("{{ .Title }}")
 (root / "layouts/_default/single.html").write_text("{{ .Title }}")
 for name, extra in {
 "visible": "",
 "draft": "draft: true\n",
 "future": "publishDate: 2030-01-01T00:00:00Z\n",
 }.items():
 (root / f"content/posts/{name}.md").write_text(
 f"---\ntitle: {name}\ndate: 2025-01-01T00:00:00Z\n"
 + extra + "---\nPublication fixture.\n"
 )
 cases = [
 ("normal", "2026", [], {"visible"}),
 ("drafts", "2026", ["-D"], {"visible", "draft"}),
 ("future", "2026", ["-F"], {"visible", "future"}),
 ("preview", "2026", ["-D", "-F"], {"visible", "draft", "future"}),
 ("later", "2031", [], {"visible", "future"}),
 ]
 for name, year, flags, expected in cases:
 destination = root / f"output-{name}"
 subprocess.run(
 ["hugo", "--source", str(root), "--destination", str(destination),
 "--clock", year + "-01-01T00:00:00Z", *flags],
 check=True, capture_output=True, text=True,
 )
 actual = {p.parent.name for p in destination.glob("posts/*/index.html")}
 assert actual == expected, (name, actual, expected)
 print(name, sorted(actual))

The assertions verify default exclusion, each inclusion flag, their combination, and publication after the boundary. The draft remains absent from the later normal build. This checks Hugo’s rendering decisions under controlled inputs; it does not test the deployment scheduler, CDN invalidation, or this site’s future-inclusive configuration. Those need separate artifact and public-route evidence at release time.

For a site using expiry dates, add an expired fixture and exercise -E independently. Include a future draft as well when the workflow relies on overlapping exclusion conditions.

Sources

Hugo: Basic usage, draft, future, expired content, and destination behavior, checked 2026-09-06.
Hugo: Front matter, publication metadata and cascading settings, checked 2026-09-06.
Hugo command reference, inclusion switches and the build clock, checked 2026-09-06.

iotop shows which process is burning the disks

Wed, 15 Mar 2028 00:00:00 +0000

A disk that spikes with no obvious writer is hard to chase; iotop maps throughput straight back to the task doing the damage. System-wide tools show the symptom — high await, saturated utilization — but never the suspect: fifty processes share the disk and the average blames all of them equally. The question that ends the investigation is always per-process — who, how much, since when — and iotop answers exactly that, live, from kernel counters no application can hide from.

Keep it beside the system-wide view rather than instead of it. iostat or vmstat says the disk is the bottleneck; iotop names the process making it so. Together they compress a storage mystery into a two-command diagnosis: confirm the resource, identify the consumer.

Per-process counters

iotop reads per-process IO from the kernel, so live read and write rates map to a pid rather than a nameless load average. Run it with -o to show only processes that are actually doing IO. The default view splits reads from writes and shows swap-in percentage alongside throughput, which separates the genuinely disk-hungry from the merely present — a process with high DISK READ but zero writes is scanning, one with writes is producing, and the distinction decides whether you throttle, relocate, or kill.

Note what the counters actually measure: bytes passing through the IO layer per process, including buffered writes not yet on disk. A process can show huge write rates while the disk looks idle (page cache absorbing) or modest rates while the disk saturates (many small synchronous writes). Read iotop together with device utilization: process-side blame plus device-side saturation is the complete picture, and either half alone misleads.

See whatever sweeps the disk

Sort by bandwidth and the offender stands out, whether it is a log hoarder, an indexer, or a runaway backup. Follow the pid to the exact process (and often the file) before acting. The usual lineup: an unrotated log written at debug level, a reindex job scheduled against peak hours, a backup writing to the same spindle it reads from, a container layer unpack looping on a corrupt image. Each has a different fix — rotate, reschedule, separate, prune — and killing the process treats the symptom while the cause reschedules itself for tomorrow.

Resolve the pid to its full context before touching anything: command line, working directory, open files via lsof, parent process. A java process burning writes might be ten applications behind one PID namespace; the fix belongs to the right one. Attribution first, action second — every storage war story that ends in regret skipped the first step.

Watch before you act

iotop observes rather than changes anything, so you can investigate the culprit before deciding what to throttle, move, or kill. Pair it with strace or lsof on the pid to pin the file and the syscall. The combination is decisive: iotop says process 18412 writes 400 MB/s, lsof says the writes go to one growing log file, strace shows O_APPEND writes of 4 KB — diagnosis complete, and the fix (log level, rotation, async appender) is obvious instead of debated.

Record the baseline while healthy so the incident comparison means something. A five-minute iotop batch capture on a normal day — -b -n 60 to a file — gives you “normal for this host” in numbers: which processes usually lead, what rates are ordinary. Without it, every spike looks unprecedented; with it, most incidents classify in seconds as “the indexer, again, an hour early.”

When it breaks

The classic meta-failure is permission blindness: run without privileges and iotop shows question marks instead of process details, or hides other users’ processes entirely — and the actual offender, running as another user, stays invisible. Run with the necessary privileges (root or the right capabilities) from the start, or the tool exonerates exactly the processes you cannot see. An investigation that “found nothing” with unprivileged iotop found nothing about your own processes only.

The second failure is sampling illusion on bursty IO: a process doing violent one-second bursts every minute looks innocent in a 5-second average and guilty in a 1-second one, or vice versa. Vary the delay interval when the numbers and the symptoms disagree, and prefer accumulated totals (-a) over instantaneous rates for “who hurt the disk over the last hour” questions. The right aggregation answers the question asked; the default answers whichever question its interval assumes.

Check it quickly

Capture who does IO, with context, non-interactively:

iotop -b -n 5 -o -d 2 | head -n 30
pid=$(iotop -b -n 1 -o | awk 'NR>5 {print $1}' | head -n 1); ls -l /proc/$pid/exe; cat /proc/$pid/cmdline | tr '\0' ' '
iostat -x 1 3 | grep -E "Device|sd|nvme"

The batch capture ranks offenders without a terminal UI, the pid resolution names the top writer’s binary, and iostat confirms the device side is actually saturated.

Sources

iotop man page, Official iotop reference, checked 2026-06-01.
vmstat man page, documents system-wide CPU and IO-wait reporting, checked 2026-09-05.

A microcache serves fresh entries briefly and coalesces misses explicitly

Sun, 12 Mar 2028 00:00:00 +0000

A microcache keeps reusable responses for a short freshness lifetime, often a few seconds. Requests for the same selected representation can reuse a fresh entry during that window. The response might lag a new origin update, but it is not HTTP-stale merely because it came from a cache. Expiration, permission to serve stale content, and concurrent cache filling are separate policies.

When it breaks

A short TTL does not make a shared response safe for every user. If personalized requests read or populate a generic entry, one user’s representation can be served in another context. proxy_cache_bypass prevents reading from cache under its conditions; proxy_no_cache prevents saving the response. Using only one side can leave the other path inconsistent with the intended privacy boundary.

The cache key also defines which requests share a response. Ignoring a meaningful query or host dimension can merge different representations. Including arbitrary varying data can instead destroy reuse. The fixture below uses one backend authority and preserves the complete request URI, including arguments. A service varying by tenant, language, or other inputs must establish its own representation identity.

Caching alone does not guarantee one origin request during a cold burst. Several requests can miss the same entry before any response is stored. proxy_cache_lock coordinates filling a new entry, with finite timeout and age controls. That lock is not the TTL, and a brief freshness lifetime does not implicitly turn lock behavior on.

Check it quickly

Save the script as check-microcache.py and run it with Python, Docker, and the pinned images. The recorded environment used nginx 1.28.0 on Linux arm64. A disposable backend reports per-path request counts, and nginx uses a two-second validity setting with explicit cookie and authorization bypass conditions. The listener is restricted to host loopback:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True)
 target.write_bytes(content if isinstance(content,bytes) else content.encode())
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 request.port=port
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'
from concurrent.futures import ThreadPoolExecutor
import threading
backend=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
from urllib.parse import urlsplit,parse_qs
import json,threading
lock=threading.Lock();counts={};gates={'/lock':threading.Event(),'/nolock':threading.Event()}
class Handler(BaseHTTPRequestHandler):
 protocol_version='HTTP/1.1'
 def log_message(self,*args):pass
 def do_GET(self):
 parsed=urlsplit(self.path)
 if parsed.path=='/control/state':
 with lock:data=dict(counts)
 elif parsed.path=='/control/release':
 gates[parse_qs(parsed.query)['path'][0]].set();data={'released':True}
 else:
 with lock:counts[self.path]=counts.get(self.path,0)+1;count=counts[self.path]
 if parsed.path in gates:assert gates[parsed.path].wait(15)
 data={'path':self.path,'count':count,'cookie':self.headers.get('Cookie'),'authorization_present':bool(self.headers.get('Authorization'))}
 body=json.dumps(data,sort_keys=True).encode();self.send_response(200)
 if parsed.path=='/cookie':self.send_header('Set-Cookie','fixture=public-example; Path=/')
 if parsed.path=='/private':self.send_header('Cache-Control','private')
 if parsed.path=='/no-store':self.send_header('Cache-Control','no-store')
 self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
ThreadingHTTPServer(('0.0.0.0',8081),Handler).serve_forever()
'''
extra='''
proxy_cache_path /tmp/cache levels=1:2 keys_zone=pages:1m inactive=30s max_size=10m;
map $http_cookie $skip_cookie { default 1; "" 0; }
map $http_authorization $skip_auth { default 1; "" 0; }
proxy_cache pages; proxy_cache_key "$scheme|$proxy_host|$request_uri"; proxy_cache_valid 200 2s;
proxy_cache_bypass $skip_cookie $skip_auth; proxy_no_cache $skip_cookie $skip_auth;
proxy_cache_lock on; proxy_cache_lock_timeout 5s; proxy_cache_lock_age 5s;
add_header X-Cache $upstream_cache_status always;
'''
locations='''
location /control/ { proxy_cache off; proxy_pass http://backend:8081; }
location = /nolock { proxy_cache_lock off; proxy_pass http://backend:8081; }
location / { proxy_pass http://backend:8081; }
'''
with lab(config(locations,extra),backend=backend) as (_,request):
 def get(path,headers=None):
 status,h,b=request(path,headers=headers);assert status==200;return h.get('x-cache'),json.loads(b)
 results={}
 first=get('/item?x=1');second=get('/item?x=1');assert first[0]=='MISS' and second==('HIT',first[1])
 other=get('/item?x=2');assert other[0]=='MISS' and other[1]['path']!=first[1]['path']
 results['fresh']=[first[0],second[0]];results['different_query']=other[0]
 for name,headers in [('cookie',{'Cookie':'session=public-fixture'}),('authorization',{'Authorization':'Bearer public-fixture'})]:
 private=get('/item?x=1',headers);assert private[0]=='BYPASS' and private[1]!=first[1]
 assert get('/item?x=1')==('HIT',first[1])
 new='/new-'+name;assert get(new,headers)[0]=='BYPASS'
 public=get(new);assert public[0]=='MISS' and public[1]['cookie'] is None and not public[1]['authorization_present']
 results[name]={'warm_key':'BYPASS','public_after_private_miss':public[0]}
 for path in ['/cookie','private','no-store']:
 path='/'+path.lstrip('/');a=get(path);b=get(path)
 assert a[0]==b[0]=='MISS' and a[1]['count']==1 and b[1]['count']==2
 results[path]=[a[0],b[0]]
 deadline=time.monotonic()+6
 while time.monotonic()<deadline:
 value=get('/item?x=2')
 if value[0]=='EXPIRED':break
 assert value[0]=='HIT';time.sleep(.15)
 else:raise AssertionError('TTL did not expire')
 assert value[1]['count']==2;results['after_ttl']={'status':value[0],'count':value[1]['count']}
 for path,number,origin_count in [('/lock',6,1),('/nolock',4,4)]:
 barrier=threading.Barrier(number+1)
 def client():barrier.wait();return get(path)
 with ThreadPoolExecutor(max_workers=number) as pool:
 futures=[pool.submit(client) for _ in range(number)];barrier.wait()
 wait(lambda:get('/control/state')[1].get(path)==origin_count)
 time.sleep(.3);before=get('/control/state')[1][path]
 assert before==origin_count and not any(f.done() for f in futures)
 get('/control/release?path='+path);responses=[f.result(timeout=10) for f in futures]
 assert get('/control/state')[1][path]==origin_count
 statuses=sorted(r[0] for r in responses)
 if path=='/lock':assert statuses==['HIT']*5+['MISS'] and all(r[1]==responses[0][1] for r in responses)
 else:assert statuses==['MISS']*4
 results[path]={'origin_requests_while_held':before,'cache_statuses':statuses}
 print(json.dumps(results,sort_keys=True))

The first public request to /item?x=1 reports MISS, and the next reports HIT with the same backend result. /item?x=2 has its own MISS and distinct path in the response. Polling that second entry eventually produces EXPIRED and a backend count of two. This observes actual expiration using the server’s clock; it does not assume an exact subsecond transition time.

Requests carrying the example session cookie or authorization marker report BYPASS even when a public entry is already warm. A later public request still returns the original public HIT. A second control first visits an unused key with those headers, then visits it publicly: the public request is MISS, proving that the preceding private-context response did not populate a reusable entry.

Backend responses containing Set-Cookie, Cache-Control: private, or Cache-Control: no-store each produce two successive MISS results and increasing origin counts. The configuration does not ignore those response controls. This distinguishes client-side bypass policy from origin-provided storage restrictions; the short TTL is not used as an excuse to override either privacy signal.

For a cold locked entry, six clients start together while the backend deliberately withholds the response. The backend sees one request before release; afterward, one client reports MISS and five report HIT with the same result. For a separate entry with the lock disabled, four held clients create four origin requests and all report MISS. The gate makes overlapping misses directly observable.

Choose freshness and overload behavior separately

The two-second setting applies to eligible 200 responses in this experiment. Relevant origin cache headers can take priority over configured validity. Decide which responses are shareable before selecting the lifetime, and keep the cache key aligned with the inputs that actually change their content. A high hit ratio is not evidence that the selected representation was correct.

Lock waiters exchange extra waiting for reduced origin work. This fixture releases the origin within the configured lock limits; it does not establish a permanent single-request guarantee when those limits expire or a backend fails. Review lock timeout, lock age, and any stale-serving policy together with the service’s acceptable latency and failure behavior.

Observe cache status alongside origin counts and response identity. MISS means the requested representation was not served as a cache hit, while BYPASS identifies the explicit read exclusion shown here. Fresh HIT, EXPIRED refill, and intentionally stale service should not be collapsed into one “cached” result. Their different effects on correctness and backend load are the reason to measure them separately.

Sources

nginx proxy module, cache validity, cache locks, bypass, no-cache, and response header controls, checked 2026-09-12.
nginx content caching guide, cache keys, status observation, and caching behavior, checked 2026-09-12.
RFC 9111 HTTP Caching, fresh versus stale responses and private/no-store directives, checked 2026-09-12.

etcd elects a leader so writes happen in order

Thu, 09 Mar 2028 00:00:00 +0000

etcd is a replicated log, and Raft is the algorithm that keeps the replicas identical. One member is leader for a term and is the only one allowed to propose entries; an entry is committed once a majority has it. Leader election decides who may write, and log replication decides what is durable — separate mechanisms with separate failure modes.

Both are governed by one number. The quorum sets how many acknowledgements a write costs, how many members you can lose, and what a partition does. Get the member count wrong and you pay more for the same resilience; lose the quorum and the cluster stops writing while still answering reads.

One leader per term

A member becomes a candidate when it has not heard from a leader within its election timeout, randomised so members do not all campaign at once. It increments the term, votes for itself, and asks the others; a member grants at most one vote per term and refuses a lower term. A majority makes it leader. If two members time out together the vote can split — in a four-member cluster the result can be two against two, short of the quorum of three — wasting the round for another election timeout.

During an election the cluster cannot process writes: etcd queues them until a leader exists. Writes sent to the old leader but not yet committed may be lost, because the new leader may overwrite uncommitted entries; committed writes are never lost. The term is the clock that makes this safe, and why an isolated member with a higher term is a hazard.

Quorum arithmetic, and why odd wins

Quorum is a bare majority, floor(N/2) + 1, so the failures a cluster tolerates are N - quorum. Three members tolerate one failure and need two acknowledgements; four tolerate one but need three. Five tolerate two with three acks; six also tolerate two but need four. An even member count buys no resilience and costs an extra acknowledgement per write.

Membership changes move the quorum immediately, which is where even counts get dangerous. The documented example is a one-member cluster: adding a member raises the quorum from one to two, and the previous leader, finding it has no active quorum, triggers an election. Growing three members to four raises the quorum from two to three, so a 2-and-2 partition leaves neither side able to elect anyone — a split the three-member cluster would have survived.

Learners, and the two mechanisms apart

A learner receives the leader’s log but does not vote, which is what you want for a node that just joined: adding one to a three-member cluster leaves the quorum at two, so the join cannot trigger an election. A learner rejects client reads and writes, leadership cannot be transferred to it, and it never promotes itself. Only once its log has caught up can an operator promote it; only then does it count toward the quorum.

Keeping election and replication distinct explains the confusing symptoms. Losing the leader stops appends until a new one is elected, even though every member is healthy. A healthy leader with unreachable followers cannot commit. And a follower can be arbitrarily far behind while the cluster commits happily, because only a majority needs the entry — lag is a replication problem, not an availability one, until the majority itself is gone.

When it breaks

The classic failure is losing the quorum. Writes stop: a cluster whose majority has failed cannot accept more writes, and recovers only once a majority is available again — otherwise disaster recovery from a backup is the only route. The trap is that reads may still work. A linearizable read is a consensus operation and fails like a write, but a serializable read is served from the member’s own state and can return data the cluster has already moved past, so “it still answers” is not evidence of health.

The second failure is the partitioned minority. It cannot elect a leader, so it has no quorum and no writes, and there is no split brain because membership changes need the majority. At the Raft level those isolated members keep timing out and raising their term while the majority stays put; when the partition heals, the higher term forces the leader to step down and a fresh election runs. A healed partition therefore produces a short write outage.

Check it quickly

Read the leader, the term and the per-member log lag:

etcdctl --endpoints=https://10.0.1.11:2379 endpoint status --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379,https://10.0.1.12:2379,https://10.0.1.13:2379 endpoint health --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379 member list --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379 --consistency=s get /registry/ --prefix --keys-only | head

The status table has exactly one true in the LEADER column and a RAFT TERM every member should share; a much higher term marks the member that was partitioned. If endpoint health fails on enough endpoints to break the majority, writes will be refused. A member listed learner=true does not vote and does not count toward the quorum. The last command is the serializable read that still succeeds without a quorum, which is why it is a poor health check.

Sources

etcd failures, leader failure, majority failure, network partitions and what happens to writes, checked 2026-09-16.
Runtime reconfiguration, majority commit and the cost of changing cluster size, checked 2026-09-16.
etcd learner design, the non-voting learner state and promotion, checked 2026-09-16.
etcd API guarantees, linearizable versus serializable reads, checked 2026-09-16.

BPF tracepoints peek inside the running kernel

Mon, 06 Mar 2028 00:00:00 +0000

BPF and kernel tracepoints let you observe behavior at the source with a tiny footprint: no agent, no restart, no stopping the workload. Traditional profiling means installing something, restarting with flags, or pausing the process — each of which perturbs the system under study or is forbidden in production. A BPF probe attaches to an already-running kernel, collects its answer, and detaches, leaving the workload exactly as it found it.

For the questions “what is slow” and “what is happening right now,” a minute with the right one-liner beats an hour of log archaeology.

Probe without pausing

A small eBPF program attaches to an existing tracepoint and accumulates counts or histograms while the work keeps running. Because it runs in kernel context, the probe rarely becomes the bottleneck. The program is verified before loading — the kernel checks memory safety and termination — so a buggy probe is rejected rather than crashing the machine. Aggregation happens in kernel maps, meaning millions of events collapse into a small histogram before anything crosses to userspace; the common case costs single-digit percent overhead, often less than the logging the team already runs.

Start from questions, not tools. “Which process opens the most files” maps to one tracing shape, “p99 disk latency by device” to another, and the bpftrace one-liner gallery covers both without writing C. Prefer high-level frontends — bpftrace, bcc tools — over raw BPF unless the measurement truly needs custom logic; the one-liner you can write during an incident beats the perfect program you finish next week.

Tracepoints are the stable names

Kernel tracepoints mark events such as block I/O, scheduling, syscall entry/exit that are less brittle than raw kprobes, so probes survive kernel updates. A kprobe pins a kernel function’s address, which shifts between versions; a tracepoint names a semantic event the kernel promises to keep, so the same probe works after Tuesday’s patch reboot. Attach to the ones closest to your behavior: block tracepoints for disk questions, scheduler tracepoints for latency-off-CPU questions, syscall tracepoints for “who calls what” questions.

List what exists before guessing names — tracepoints are enumerated under the kernel’s tracing interface, and every standard tool can print them. When no tracepoint covers the event, fall back to kprobes or uprobes deliberately and pin the probe to kernel versions in comments, accepting the maintenance cost with open eyes. Stability is a spectrum: tracepoints first, well-known kprobes second, raw addresses never in anything you intend to keep.

Read a histogram, not a firehose

Aggregate a noisy event stream into a latency or count histogram to expose the top offenders without dumping each event. Use that shape to find the culprit, then drill in. A per-event dump of a million syscalls answers nothing a human can read; a log-scale latency histogram shows the bimodal split — fast path versus the slow one percent — in a single screen. The histogram is the diagnosis; individual events are only the confirmation, sampled after the shape tells you where to look.

Keep the aggregation key tight: by process, by file, by syscall — one dimension at a time. Two-dimensional breakdowns explode cardinality and bury the signal, and full-argument dumps reintroduce the firehose through the back door. When the histogram fingers a suspect, switch to targeted tracing on that process or file alone, with a time bound, and stop once the confirming sample exists.

When it breaks

The classic failure is the probe that never attaches: wrong tracepoint name, missing kernel headers, or a locked-down environment blocking BPF loading — and the tool exits quietly or with a cryptic verifier message. Verify the environment first (tracepoints listed, loader permissions present) before doubting the probe logic; most BPF failures are environmental, not logical. In containers, remember the probe usually needs host-level privileges the container runtime does not grant by default.

The second failure is observer effect at scale: an unfiltered probe on a hot path — every syscall on a 100k-rps box — burns whole cores and skews the measurement it was taking. Always aggregate in-kernel, filter to the suspect early, and time-box production probes. If overhead looks suspicious, halve the sampling or narrow the filter and check the answer stays stable; a measurement that changes shape when observed more cheaply was never trustworthy.

Check it quickly

Confirm the tooling works and take a first histogram:

bpftrace -l "tracepoint:syscalls:*" | head -n 5
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @opens[comm] = count(); }' &
BPF_PID=$!; sleep 10; kill -INT $BPF_PID; wait

The first proves tracepoints are visible, and the ten-second sample ranks processes by file opens — the shape of every BPF investigation in miniature.

Sources

Kernel tracing docs on bpftrace, Official bpftrace reference, checked 2026-06-01.
eBPF introduction, Official eBPF documentation, checked 2026-06-01.

InnoDB covering indexes carry the query's columns

Fri, 03 Mar 2028 00:00:00 +0000

A covering index contains the columns needed to evaluate a particular query and return its requested values. In InnoDB, secondary-index records also carry the primary-key columns, so a query can be covered without explicitly listing the primary key again in the secondary-index definition. Coverage is a relationship between one query and an index, not a permanent property that applies to every query against the table.

When it breaks

Adding a projected column can remove coverage. An index that supplies an identifier and timestamp may not contain the large payload that a later application change begins selecting. The optimizer can still use that same index to locate matching rows, then obtain missing values from the clustered records. Index use and covering access are different facts.

Coverage also does not guarantee a selective access path. A wide scan can be fully covered while examining many entries. Column order, predicates, sorting requirements, and row estimates still matter. Do not interpret Using index as evidence that a query is fast, or confuse it with Using index condition, which describes index condition pushdown rather than complete projection coverage.

InnoDB visibility checks qualify the usual index-only explanation. Secondary records do not contain the same hidden transaction metadata as clustered records. Delete-marked secondary entries or pages changed by newer transactions can require clustered-record lookups to obtain the correct visible version. A covering plan is therefore not a universal promise of zero clustered access under concurrent MVCC activity.

Check it quickly

Use a fresh MySQL 8.4 instance and a client session with permission to create a disposable schema. The recorded run used MySQL 8.4.11. Ensure xbcatm_index_lab is unused, stop on unexpected SQL errors, and keep the same session so its selected database remains active. Create ten thousand rows, with one hundred paid orders and a deliberately wider payload:

CREATE DATABASE xbcatm_index_lab;
USE xbcatm_index_lab;
CREATE TABLE orders (
 id INT PRIMARY KEY,
 status VARCHAR(10) NOT NULL,
 created_at DATETIME NOT NULL,
 payload VARCHAR(1000) NOT NULL,
 INDEX by_status_created (status,created_at)
) ENGINE=InnoDB;
INSERT INTO orders
WITH digits(n) AS (
 SELECT 0 UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3
 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6
 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9
)
SELECT number+1,IF(MOD(number+1,100)=0,'paid','open'),
 TIMESTAMP '2026-01-01 00:00:00' + INTERVAL (number+1) SECOND,
 REPEAT('x',640)
FROM (SELECT a.n+10*b.n+100*c.n+1000*d.n AS number
 FROM digits a CROSS JOIN digits b CROSS JOIN digits c CROSS JOIN digits d) AS numbered;
ANALYZE TABLE orders;
SELECT COUNT(*),SUM(status='paid') FROM orders;

The final count must be ten thousand and one hundred. The secondary index contains status and creation time explicitly; InnoDB supplies the primary identifier implicitly. Inspect and run a query that needs only those values:

EXPLAIN SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
EXPLAIN ANALYZE SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
SELECT id,created_at FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;

The recorded tabular plan reports by_status_created and Using index. The analyzed plan identifies a covering index lookup and returns twenty rows. Result identifiers are one hundred, two hundred, and so on through two thousand, ordered by the timestamp and identifier. These are checked results on populated data, not a plan inferred from an empty schema.

Now request the payload as well:

EXPLAIN SELECT id,created_at,payload FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;
EXPLAIN ANALYZE SELECT id,created_at,payload FROM orders WHERE status='paid' ORDER BY created_at,id LIMIT 20;

The recorded plan still uses by_status_created, but loses Using index; the analyzed plan describes an index lookup without the covering qualification. This isolates the effect of changing the requested columns while keeping the filter, ordering, and index unchanged. The diagnostic execution does run the SELECT, so account for that when applying EXPLAIN ANALYZE to expensive real queries.

Remove the schema created for this exercise:

DROP DATABASE xbcatm_index_lab;

The timings from this tiny warm experiment are not a throughput or latency benchmark. Scheduling, caching, and instrumentation can outweigh the difference between two twenty-row reads. The verified conclusion concerns access shape and returned values, not a promised percentage reduction in physical I/O.

Pay for coverage deliberately

Appending columns to a secondary index consumes storage and buffer-pool capacity and increases maintenance when those indexed values change. Long primary keys enlarge secondary entries too. Cover the important narrow query rather than automatically copying a large payload into every index. Keep projected columns explicit so later application changes make their access requirements visible in review.

Compare representative plans, examined rows, foreground latency, and write cost before retaining a new index. Test with realistic selectivity and concurrent activity; a quiet fixture does not measure MVCC fallback frequency or replica maintenance cost. An index should earn its place through the workload it serves, not through the appearance of a desirable label in one explanation.

Sources

MySQL 8.4 clustered and secondary indexes, primary-key columns carried by secondary records, checked 2026-09-12.
MySQL 8.4 EXPLAIN output, index-use and covering-plan indicators, checked 2026-09-12.
MySQL 8.4 InnoDB multiversioning, clustered lookups required for secondary-index visibility checks, checked 2026-09-12.

DNS TTL trades repeated upstream queries against retained answers

Wed, 01 Mar 2028 00:00:00 +0000

A DNS TTL is a lifetime attached to cached data, not a refresh job scheduled for every second. A cache can reuse a valid answer while it retains that entry. After expiry, a subsequent request usually requires fresh resolution unless another configured behavior applies. An idle record need not produce an upstream query merely because its TTL has reached zero.

When it breaks

Shortening TTL increases potential upstream work when clients keep asking for the same data. It does not multiply all authoritative traffic by one universal factor: query frequency, resolver sharing, cache capacity, prefetching, and record diversity affect the result. A short TTL on one popular name behaves differently from a large stream of names that are each queried only once.

A long TTL permits continued use of an answer after the authority changes its data. That cached answer may still be valid under the lifetime originally received even though it no longer matches the newest zone content. Calling every mismatch an expired or broken cache obscures the actual contract. TTL expresses permitted retention; it does not promise continuous synchronization with the authority.

Multiple forwarding layers do not ordinarily receive a fresh copy of the original TTL at every hop. A cache returns remaining lifetime, which the next cache then ages. Adding full TTLs per layer invents a propagation delay that ordinary compliant forwarding does not create. Local TTL overrides and serve-stale policies are separate behaviors that should be identified explicitly when present.

Check it quickly

Save this as check-ttl-cost.py and run python3 check-ttl-cost.py with Docker. The pinned container uses BIND 9.18.49 and dnsmasq 2.90, with no external runtime network. The authoritative server logs actual queries. One forwarder initially serves two names, and the script later starts a second forwarder behind the first to inspect the TTL handed downstream:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends bind9=1:9.18.49-1~deb12u2 bind9-dnsutils=1:9.18.49-1~deb12u2 dnsmasq-base=2.90-4~deb12u2 python3-dnspython=2.3.0-1 && rm -rf /var/lib/apt/lists/*\nCMD ["/usr/bin/python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,re,signal,subprocess,time
import dns.exception,dns.message,dns.query,dns.rcode,dns.rdatatype

def run(*args,check=True):
 result=subprocess.run(args,capture_output=True,text=True,timeout=15)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def query(name,kind='A',port=1053):
 return dns.query.udp(dns.message.make_query(name,kind),'127.0.0.1',port=port,timeout=1)
def values(response,kind='A'):
 wanted=dns.rdatatype.from_text(kind)
 return [item.to_text() for rrset in response.answer if rrset.rdtype==wanted for item in rrset]
def show(response):
 def section(records):return [{'name':str(rr.name),'type':dns.rdatatype.to_text(rr.rdtype),'ttl':rr.ttl,'data':[item.to_text() for item in rr]} for rr in records]
 return {'rcode':dns.rcode.to_text(response.rcode()),'answer':section(response.answer),'authority':section(response.authority)}
def wait(predicate,timeout=15):
 deadline=time.monotonic()+timeout
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except (OSError,dns.exception.Timeout):pass
 time.sleep(.05)
 raise AssertionError('DNS fixture condition did not become true')
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=5)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=5)

@contextmanager
def lab(records,soa_ttl=8,minimum=4,extra=''):
 with TemporaryDirectory(prefix='xbcatm-dns-') as directory,ExitStack() as stack:
 root=Path(directory);zone=root/'zone';conf=root/'named.conf';log=root/'named.log';serial=1
 def save(body):
 zone.write_text(f'$ORIGIN lab.test.\n$TTL 30\n@ {soa_ttl} IN SOA ns.lab.test. hostmaster.lab.test. ({serial} 60 60 3600 {minimum})\n@ IN NS ns.lab.test.\nns IN A 127.0.0.1\n'+body+'\n')
 run('named-checkzone','lab.test',str(zone))
 save(records)
 conf.write_text(f'''options {{ directory "{root}"; listen-on port 1053 {{127.0.0.1;}}; listen-on-v6 {{none;}};
recursion no; dnssec-validation no; notify no; querylog yes; max-cache-size 8m; pid-file "{root}/named.pid"; session-keyfile none;
{extra} }};
controls {{}};
zone "lab.test" {{ type primary; file "{zone}"; }};
''')
 run('named-checkconf',str(conf))
 output=stack.enter_context(log.open('w'))
 authority=subprocess.Popen(['named','-g','-c',str(conf),'-n','1','-U','1'],stdout=output,stderr=subprocess.STDOUT)
 stack.callback(stop,authority)
 def ready():
 assert authority.poll() is None,log.read_text()
 response=query('lab.test.','SOA');return response.rcode()==0 and response.answer[0][0].serial==serial
 wait(ready)
 def update(body):
 nonlocal serial
 serial+=1;save(body);authority.send_signal(signal.SIGHUP);wait(ready)
 def cache(port=1054,upstream=1053,size=150,options=()):
 logfile=root/('cache-'+str(port)+'.log');stream=stack.enter_context(logfile.open('w'))
 command=['dnsmasq','--no-daemon','--conf-file=/dev/null','--no-resolv','--no-hosts','--bind-interfaces',
 '--listen-address=127.0.0.1','--port='+str(port),'--server=127.0.0.1#'+str(upstream),
 '--cache-size='+str(size),'--log-queries','--log-facility=-',*options]
 process=subprocess.Popen(command,stdout=stream,stderr=subprocess.STDOUT);stack.callback(stop,process)
 def cache_ready():
 assert process.poll() is None,logfile.read_text()
 return query('lab.test.','SOA',port).rcode()==0
 wait(cache_ready);return process,logfile
 def queries(name,kind='A'):
 return sum(bool(re.search(r'query: '+re.escape(name.rstrip('.'))+r' IN '+kind+r'\b',line)) for line in log.read_text().splitlines())
 yield root,update,cache,queries
with lab('short 1 IN A 127.0.0.2\nlong 20 IN A 127.0.0.3\nchain 6 IN A 127.0.0.4') as (_,update,cache,queries):
 cache();samples=[]
 for i in range(3):
 pair={name:show(query(name+'.lab.test.',port=1054)) for name in ('short','long')};samples.append(pair)
 if i<2:time.sleep(1.2)
 wait(lambda: queries('short.lab.test')==3 and queries('long.lab.test')==1)
 first=query('chain.lab.test.',port=1054);assert first.answer[0].ttl==6
 wait(lambda: queries('chain.lab.test')==1)
 time.sleep(2.2);cache(port=1055,upstream=1054)
 downstream=query('chain.lab.test.',port=1055);upstream=query('chain.lab.test.',port=1054)
 assert values(downstream)==['127.0.0.4'] and 0<downstream.answer[0].ttl<6
 assert abs(downstream.answer[0].ttl-upstream.answer[0].ttl)<=1
 assert queries('chain.lab.test')==1
 update('short 1 IN A 127.0.0.2\nlong 20 IN A 127.0.0.3\nchain 6 IN A 127.0.0.5')
 wait(lambda: values(query('chain.lab.test.',port=1055))==['127.0.0.5'],timeout=8)
 wait(lambda: queries('chain.lab.test')==2)
 print(json.dumps({'three_lookups_each':samples,'authority_query_counts':{'short':queries('short.lab.test'),'long':queries('long.lab.test')},
 'cache_chain':{'original':show(first),'downstream_first':show(downstream),'upstream_remaining':show(upstream),'authority_queries_after_expiry':queries('chain.lab.test')}},sort_keys=True))
"""
name='xbcatm-dns-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 built=docker('build','-t',name,str(root));assert built.returncode==0,built.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--memory','384m','--pids-limit','128',name,
 '/usr/bin/python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The short.lab.test record has TTL 1 and long.lab.test has TTL 20. The program queries each three times, waiting 1.2 seconds between rounds. The authoritative log records three queries for the short-lived record and one for the long-lived record. Both produce successful answers on every round. The difference is upstream work under this controlled request schedule, not a measured application-latency improvement.

The returned TTL for the long-lived record decreases across rounds, while each refreshed short-lived answer starts at 1 again. The script checks the server log rather than assuming a fast response means a cache hit. On loopback, even an uncached query can be quick, so response time alone would not establish whether the authority was contacted or the answer came from memory.

For the layered case, the first forwarder caches chain.lab.test with TTL 6. After 2.2 seconds, a newly started second forwarder queries through it. The recorded downstream answer has TTL 3, matching the upstream remaining value within integer-second timing. It does not regain TTL 6. The authoritative count remains one, confirming that this downstream fill reused the first forwarder’s cached answer.

The authority then changes the chain record’s address. Queries through the second forwarder eventually receive the new address, and the authoritative count rises to two. This checks successful refresh through both layers after the retained data expires. The exact expiry fraction is implementation-dependent; the asserted properties are preserved remaining lifetime, no premature extra authoritative query, and eventual acquisition of the changed value.

Choose TTL from observed demand and change needs

Use expected change frequency and acceptable old-answer overlap alongside query volume. A busy shared resolver can amortize one answer across many clients, whereas isolated caches duplicate fills. Eviction can make an entry disappear before its TTL, so a long configured lifetime does not guarantee a hit. Measure hits, misses, upstream requests, and available capacity instead of treating TTL as a cache-size control.

For a planned move, lower TTL early enough for earlier longer-lived copies to expire. Keep the destination overlap aligned with the actual retained data. Raising TTL after stability reduces repeated resolution for busy records, but it also extends the next change’s ordinary waiting window. The relevant unit is the RRset: records of the same name, class, and type should have a consistent TTL.

Finally, compare answer data and remaining lifetime at each resolver actually used by the application. A direct authoritative query and a cached recursive answer reveal different states. Application caches and existing connections may add their own behavior outside this DNS experiment. Separate those lifetimes before estimating how quickly users will discover a changed endpoint or how much authoritative capacity a TTL adjustment requires.

Sources

RFC 1035 domain names, TTL as a cache lifetime and treatment of cached records, checked 2026-09-12.
RFC 2181 DNS clarifications, TTL limits and consistent lifetime within an RRset, checked 2026-09-12.
dnsmasq manual, cache sizing, forwarding, and TTL options, checked 2026-09-12.

Bash history is your searchable command log

Mon, 28 Feb 2028 00:00:00 +0000

A useful command history saves retyping, but it is not a complete record of everything executed. Bash normally enables history in interactive shells, and settings can exclude entries or limit retention. The searchable list in a running shell and the history file on disk are separate states. That distinction explains why a command may be visible in one terminal, absent in another, and lost when an unwritten session ends unexpectedly.

Session memory differs from the file

HISTSIZE limits the in-memory history entries; HISTFILESIZE limits the saved file’s lines. Multiline commands and timestamp records mean that line counts and command counts are not interchangeable. HISTFILE selects the file, normally the user’s Bash history file. An unset or empty value prevents ordinary saving, and an unwritable location can leave apparently healthy in-memory recall with no persistent record.

Bash records command text before parameter expansion but after history expansion, subject to HISTCONTROL and HISTIGNORE. A saved line containing a variable therefore does not necessarily reveal the value used during the original execution. Current directories, aliases, functions, environment values, and external files can all change before a command is recalled. History preserves useful input, not a reproducible execution environment or a trustworthy audit trail.

Appending does not refresh other terminals

On normal exit, Bash saves history; histappend selects appending rather than overwriting the file. Without earlier writes, abnormal termination can lose all unsaved entries, not merely the final command. history -a appends newly entered lines that this session has not already appended. Saving before exit reduces that loss window, but does not promise durable storage through every system failure.

Another running shell has its own list. It needs an import such as history -n to read newly appended lines; enabling histappend alone does not make another terminal’s searches update live. history -r reads the entire file into the existing list and can introduce duplicates. history -w replaces the file with the current list and can discard other sessions’ records. These operations should not be treated as interchangeable synchronization commands.

If append and import operations are integrated into PROMPT_COMMAND, preserve existing prompt hooks and any use of the preceding command’s status. Replacing a configured hook wholesale can break other shell behavior. Concurrent sessions also do not create a reliable global execution order merely by sharing a file. Keep separate operational logs when ordering, exit status, or accountability matters.

Search, inspect, then execute

In the usual Emacs-style Readline bindings, Ctrl-R searches backward incrementally, and pressing it again finds older matches. Use an editing key to leave the search and inspect the selected line before running it; Ctrl-G cancels the search. Enter accepts the command. Ctrl-J is also a newline and must not be described as a harmless preview key. Bindings can differ in another editing mode or a customized terminal.

Inspect the hostname, directory, options, and intended target whenever reusing an old command. A command that was correct last month may address a different environment today. Prefer a distinctive search fragment and review the whole line, rather than trusting that the first matching command is the right operation.

When it breaks

Missing commands can result from filtering, retention limits, a different history file, failed saving, or an unrefreshed session. Check those causes separately. ignoredups excludes consecutive duplicates, not every duplicate in the entire list. ignoreboth combines that behavior with ignorespace; it does not produce a globally unique command archive.

A leading space excludes an entry only when the relevant setting is active. It does not remove terminal recordings, process arguments, or logs written elsewhere, so do not put credentials directly into command lines and rely on history filtering for secrecy. Diagnostic examples should use synthetic entries and a temporary history file, rather than printing someone’s real recent commands.

Check it quickly

Run this with Python 3, Bash, and standard Unix utilities. The environment points Bash to a temporary file before it starts. Only synthetic entries are inspected. The brace group avoids recording the demonstration’s own control commands after history is enabled.

import os
import subprocess
import tempfile
from pathlib import Path

script = r'''{
set -eu
set -o history
history -c
history -s 'demo original'
history -w "$HISTFILE"
printf 'first-file:\n'
cat "$HISTFILE"
history -c
history -s 'demo replacement'
history -w "$HISTFILE"
printf 'replacement-file:\n'
cat "$HISTFILE"
history -r "$HISTFILE"
printf 'memory-after-reread:\n'
history | sed 's/^[[:space:]]*[0-9][0-9]*[[:space:]]*//'
unset HISTFILE
}'''
with tempfile.TemporaryDirectory(prefix="bash-history-demo-") as directory:
 history_path = Path(directory) / "history"
 env = dict(os.environ, BASH_ENV="", HISTFILE=str(history_path), HISTSIZE="100",
 HISTFILESIZE="100", HISTCONTROL="", HISTIGNORE="",
 HISTTIMEFORMAT="")
 subprocess.run(["bash", "--noprofile", "--norc", "-c", script],
 env=env, check=True)

The first file contains demo original; after replacement it contains only demo replacement. Reading that file into a list already containing the replacement produces two identical entries in memory. This verifies overwrite and import semantics without executing the stored strings. It does not claim to test keyboard bindings or crash durability, which require their own observations.

Sources

Bash history facilities, session lists and saving to the history file, checked 2026-09-07.
Bash history builtins, append and replacement versus importing entries, checked 2026-09-07.
Readline history commands, search and command-acceptance bindings, checked 2026-09-07.

Protobuf lets old and new services talk

Sat, 26 Feb 2028 00:00:00 +0000

A gRPC service and its clients ship on their own schedules, so the schema has to change while both old and new binaries are running. Protobuf is built for exactly that: the compatibility contract lives in field numbers and wire types rather than in names, and a reader that does not recognise a field skips it instead of failing.

The tension is that this safety is narrow and unenforced. A small set of edits is compatible; the rest corrupt data silently, and the compiler only refuses the edits you told it to refuse. Knowing which is which is the whole skill.

The wire speaks numbers, not names

Every field on the wire is a tag — the field number shifted left three bits and OR’d with a wire type — followed by the payload. For a message with id=7, name=“alpha”, code=1234 the bytes are 08 07 12 05 61 6c 70 68 61 18 d2 09, where 0x18 marks field 3 as a varint and d2 09 is 1234. protoc --encode produces byte-identical output, so this is the format rather than a description of it.

Because the name never appears, renaming is free: the same bytes decoded against a schema that calls field 1 event_id and field 3 status produce the same values under the new labels. Adding a field with a fresh number is safe too — an old reader surfaces the unknown field number and preserves its bytes, and re-encoding reproduces the newer message exactly. That preservation is what lets an old proxy forward a message it does not understand.

Never reuse a field number

The corrupting edit is deleting a field and reusing its number. There are two variants and both are bad. If the wire type changes — an int32 varint becomes a string — the new reader sees the old value stranded as an unrecognised field number while the declared string stays empty: silent loss, with nothing logged. If the wire type stays compatible it is worse, because the value decodes cleanly into the wrong meaning: the same bytes read as int32 give -1 and as uint32 give 4294967295, and 1 read as a bool is true.

The tool for this is reserved. Declaring reserved 3; reserved "code"; makes the compiler reject any field using number 3 or the name code, and suggest the next free number. Without that declaration the same schema compiles without complaint, which is how a corrupting change reaches production.

Presence is a separate decision

proto3 fields have implicit presence: an absent field and a field set to zero encode to identical bytes, so a reader cannot distinguish “the client did not send it” from “the client sent zero”. Adding optional changes that — a zero now emits its tag — and the generated accessor reads that bit. Use it wherever zero is a meaningful value rather than a synonym for missing.

The related failure is assuming a field is required. There is no required on the wire, and a new field that old producers never send will simply be absent, so treating absence as an error breaks every client that has not been upgraded yet. Reserve numbers when you delete, add optional fields rather than changing types, and let absence mean what it says.

When it breaks

The classic failure is a reused tag number silently mis-decoding. A producer writes the new schema, an older consumer reads the same bytes into a field that now means something else, and nothing is logged because nothing failed: the values are simply wrong. It presents as one service computing nonsense from correct input while every other service agrees, and the way to diagnose it is to dump the wire bytes and walk the tags by hand.

The second failure is a required-field-style assumption breaking old clients. A consumer that treats a missing field as a protocol error rejects every producer that predates it, and a consumer that treats a missing field as zero takes the wrong branch. Both are the same mistake — reading semantics into absence — and both are avoided by reserving retired numbers and adding optional fields.

Check it quickly

Look at the numbers on the wire, then make the compiler enforce the retirement:

protoc --decode_raw < captured.bin
protoc -I. --descriptor_set_out=/dev/null api.proto
grep -rn 'reserved' proto/

--decode_raw prints field numbers and values with no names at all, which is what a peer actually receives. The compile must succeed; if it fails with a reserved-number error, someone tried to reuse a retired tag. The grep must show a reserved statement for every number you have removed. A wrong result is a number you thought was retired still carrying a value, or a schema that compiles only because the number was never reserved.

Sources

Protobuf: encoding, tag layout, varints and the wire types behind int32, uint32 and bool, checked 2026-09-16.
Protobuf: proto3 language guide, field numbers, reserved, optional and presence semantics, checked 2026-09-16.
Protobuf: dos and don’ts, the rules for retiring and reusing field numbers, checked 2026-09-16.

Kubernetes CronJobs need a history policy

Fri, 25 Feb 2028 00:00:00 +0000

A CronJob does not run your work; it creates a Job, and the Job creates a pod. When the pod finishes, neither the Job nor the pod disappears. The pod is kept on purpose so you can still read its logs after the fact, and the Job is kept so you can still read its status.

That default is right for one run and wrong for ten thousand. Every finished Job is an object in etcd, a row in every list response, and a watch event for every controller that cares about Jobs. Two separate mechanisms bound the pile, and they are independent: count-based history limits applied by the CronJob controller, and age-based TTL applied by the TTL controller.

A finished Job is not free

A completed Job keeps its pod, and a pod keeps its container logs for as long as the pod exists. Retention is the only thing between you and a namespace full of tombstones. Thirty days of an hourly schedule is 720 runs: 720 Jobs and 720 pods, each with a name, a status, an owner reference and a line in every kubectl get jobs. A finished Job is not inert either: its status conditions stay in the API for every watcher to re-read. The cost per object is trivial, but it is linear in time, so it stays invisible until it is a problem.

The visibility cost arrives before the resource cost. Once the namespace holds hundreds of finished Jobs, kubectl get jobs no longer fits on a screen and a genuine failure is one more line among the successes. API server memory pressure is real but gradual; the degradation of your ability to debug is immediate, and it is what makes people reach for a cleanup policy. The arithmetic compounds quietly: an hourly job accumulates a year of tombstones, and a per-minute job gets there in a week.

Two mechanisms, two controllers

The CronJob controller trims by count. successfulJobsHistoryLimit defaults to 3 and failedJobsHistoryLimit defaults to 1, and setting either to 0 keeps none of that class. The trim is per class and oldest-first, so a chronically failing job keeps only its newest failure. Neither limit is enforced by the API; an absent policy is a valid policy, and a leaking one.

The TTL controller trims by age. ttlSecondsAfterFinished lives on the Job, and the controller deletes the Job that many seconds after it finishes, cascading to its pods. 0 makes the Job eligible immediately after it finishes; unset means the TTL controller never touches it. The feature is stable since v1.23 and reads the Job rather than the CronJob, which matters below. The two do not coordinate: whichever fires first wins, so a TTL shorter than your history window silently overrides the count you configured.

Owner references decide who cleans up

Jobs created by a CronJob carry an owner reference to that CronJob, so deleting the CronJob cascades to its Jobs and their pods. Cascading deletion is why deleting a CronJob looks like it cleaned everything up: the Jobs and pods go with it, and the history limits never enter into it. The count-based trim, however, is performed by the CronJob controller, which needs the CronJob to exist in order to know the limits. kubectl delete cronjob --cascade=orphan therefore leaves Jobs with a dangling owner reference and nothing left to trim them, while TTL still applies because it is evaluated per Job.

For Jobs you create directly rather than through a CronJob, the default deletion policy is orphanDependents: deleting the Job can leave its pods behind. The control plane garbage collects them eventually, but the docs warn that lingering pods can degrade a cluster and recommend ttlSecondsAfterFinished for exactly that reason. A cleanup policy is a retention decision, not a hygiene preference.

When it breaks

Setting both history limits to 0 to “keep the namespace clean” deletes every Job the moment it finishes, and the pod and its logs go with it. The one failure you needed to read is gone, with no recovery path. Keep a few successes and enough failures to cover a weekend; evidence you cannot read is not evidence.

TTL only counts from completion, so a Job that never finishes never ages out. A pod stuck in ImagePullBackOff leaves its Job active forever: not a success, not a failure, so neither history limit counts it. Give the job template an activeDeadlineSeconds so a stuck run fails, and let TTL collect it.

Check it quickly

Compare what is configured against what is actually accumulating:

kubectl get cronjob hourly-rollup -o jsonpath='{.spec.successfulJobsHistoryLimit} {.spec.failedJobsHistoryLimit} {.spec.jobTemplate.spec.ttlSecondsAfterFinished}{"\n"}'
kubectl get jobs --sort-by=.metadata.creationTimestamp --no-headers | wc -l
kubectl get pods --field-selector=status.phase=Succeeded --no-headers | wc -l
kubectl get jobs -o jsonpath='{range .items[*]}{.metadata.name} ttl={.spec.ttlSecondsAfterFinished} owner={.metadata.ownerReferences[0].name}{"\n"}{end}'

Two empty values on the first line mean nothing bounds retention. A steady Job count near the sum of the two limits means the CronJob controller is trimming; a count that grows with the clock means it is not. In the last command, a Job with an empty ttl and an empty owner is the orphan case, and neither controller will ever remove it.

Sources

Kubernetes Job, “Clean up finished jobs automatically” and the TTL mechanism for finished Jobs, checked 2026-09-16.
Kubernetes CronJob, default history limits and the job template, checked 2026-09-16.
Kubernetes garbage collection, owner references and cascading deletion, checked 2026-09-16.

A Redis hash stores many fields in one key

Thu, 24 Feb 2028 00:00:00 +0000

A Redis hash is a field-addressable record under one key, not a schema-checked object or an automatically nested document. It is useful when callers need a few attributes without replacing the whole record. That advantage depends on access patterns, field sizes, and the meaning of replies, not merely on choosing the hash type.

Read the fields your caller actually needs

HGET retrieves one field. HMGET retrieves selected fields in the requested order, while HGETALL returns the entire mapping. With redis-py’s binary responses, an absent field becomes None and an existing empty value becomes empty bytes. A truthiness check incorrectly treats those two states as equivalent. An absent key also produces missing values through HMGET.

Field names and values are bytes; a name containing a dot is still a literal name. Redis does not interpret name.first as traversal through a nested object. Likewise, a string containing JSON remains a Redis string: HGET does not parse it. Decide serialization and validation in the application and preserve the distinction between an omitted attribute and an intentionally empty one.

Updates have precise return values

HSET changes the specified fields and leaves others untouched. A multi-field call is one command, but its integer reply counts newly added fields, not successful replacements. Updating an existing name can correctly return zero. Treating that zero as failure can cause needless retries or misleading error reports.

HINCRBY changes an integer field in one command and starts from zero when the field is absent. It rejects noninteger values and overflow beyond signed 64-bit range. This avoids a read-modify-write race for that increment; it does not make surrounding application checks atomic. Separate HGET, local arithmetic, and HSET can still overwrite another client’s update.

HDEL counts fields actually removed, and deleting the final field removes the key. Do not use an empty hash as a persistent placeholder for an entity. Whole-key expiration also removes its fields. Current Redis additionally supports field deadlines; the exercise below assigns no expirations and does not test lifetime behavior.

Compact representation is an implementation choice

Small hashes can use listpack encoding instead of a hash table. Redis 8.10.1’s standard thresholds are 512 entries and 64 bytes per field name or value; deployed settings can differ. Crossing an applicable limit changes the representation, not the public hash operations or the stored values.

Compact storage trades internal scanning work for reduced overhead and locality. Raising thresholds substantially requires measurement: a command’s advertised complexity is not a promise that arbitrarily large fields or changed configuration have no cost. Inspect OBJECT ENCODING when diagnosing representation, but do not make application correctness depend on that internal label or an assumed memory-saving ratio.

When it breaks

HGETALL grows with the whole hash and transfers both names and values. Fetching thousands of fields when the caller needs two wastes work, even if they share one key. HMGET’s work grows with the requested field count; it is not free either. Never use returned map iteration order as an application ordering contract.

WRONGTYPE indicates that the key holds another Redis type, not that a field is missing. Noninteger counters require an explicit data decision, not a retry loop. One huge hash also remains one key rather than automatically spreading its fields across cluster nodes. Choose record boundaries around actual access and ownership rather than packing unrelated data together.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Redis 8.10.1, redis-py 8.1.0, and Python 3.14.4. The setup builds only Redis core with libc and installs the client in a temporary virtual environment.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It starts a private Unix-socket server with TCP and persistence disabled. Only this server receives deliberately small thresholds: four entries and eight bytes. The test closes its client, terminates its own process, and removes the temporary runtime directory; it never connects to a default local Redis.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-hash-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--hash-max-listpack-entries", "4", "--hash-max-listpack-value", "8",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 assert client.hset("profile", mapping={"name": "Ada", "visits": "0", "empty": b""}) == 3
 assert client.hset("profile", "name", "Ada L") == 0
 assert client.hmget("profile", ["visits", "name", "missing", "empty"]) == [
 b"0", b"Ada L", None, b"",
 ]
 assert client.hmget("absent", ["a", "b"]) == [None, None]
 assert client.hgetall("profile") == {b"name": b"Ada L", b"visits": b"0", b"empty": b""}
 assert client.hincrby("profile", "visits", 1) == 1
 assert client.hget("profile", "name") == b"Ada L"
 print("HSET counts new fields; HMGET preserves order and distinguishes missing from empty")

 client.set("json", '{"name":"Ada"}')
 try:
 client.hget("json", "name")
 except redis.ResponseError as error:
 assert "WRONGTYPE" in str(error)
 else:
 raise AssertionError("a JSON string is not a hash")
 client.hset("literal", "name.first", b"Ada")
 assert client.hget("literal", "name") is None
 assert client.hget("literal", "name.first") == b"Ada"
 client.hset("invalid", "counter", "3.5")
 try:
 client.hincrby("invalid", "counter", 1)
 except redis.ResponseError as error:
 assert "integer" in str(error)
 else:
 raise AssertionError("an integer counter cannot contain 3.5")
 assert client.hget("invalid", "counter") == b"3.5"
 client.hset("overflow", "counter", 2**63 - 1)
 try:
 client.hincrby("overflow", "counter", 1)
 except redis.ResponseError as error:
 assert "overflow" in str(error)
 else:
 raise AssertionError("signed 64-bit integer overflow must fail")
 assert client.hget("overflow", "counter") == str(2**63 - 1).encode()
 print("wrong types and invalid or overflowing counters fail; dotted fields are literal")

 assert client.object("ENCODING", "profile") == b"listpack"
 assert client.hset("profile", "fourth", "ok") == 1
 assert client.object("ENCODING", "profile") == b"listpack"
 assert client.hset("profile", "fifth", "ok") == 1
 assert client.object("ENCODING", "profile") == b"hashtable"
 client.hset("value-limit", "field", b"abcdefgh")
 assert client.object("ENCODING", "value-limit") == b"listpack"
 client.hset("value-limit", "field", b"abcdefghi")
 assert client.object("ENCODING", "value-limit") == b"hashtable"
 assert client.hget("profile", "visits") == b"1"
 assert client.hget("value-limit", "field") == b"abcdefghi"
 print("private 4-field and 8-byte thresholds each trigger listpack to hashtable conversion")

 assert client.hdel("value-limit", "absent") == 0
 assert client.hdel("value-limit", "field") == 1
 assert client.exists("value-limit") == 0
 print("deleting the last field removes the hash key")

The assertions distinguish added fields from replacements, preserve HMGET ordering and missing-versus-empty values, and verify unchanged neighboring fields. A JSON string raises WRONGTYPE, a dotted field stays literal, and invalid or overflowing counters remain unchanged after rejection.

At four short fields the profile remains listpack; the fifth switches it to hashtable. A separate hash switches when its value grows from eight to nine bytes. Reads still return the expected values, and deleting the final field removes that key. These results demonstrate configured encoding transitions and command semantics, not production memory savings, latency, concurrent-load behavior, or cluster distribution.

Sources

Redis hashes, field-oriented operations and current expiration capabilities, checked 2026-09-08.
Redis HSET, partial updates and added-field counts, checked 2026-09-08.
Redis HMGET, requested ordering and missing-field replies, checked 2026-09-08.
Redis HGETALL, full-hash reads and complexity, checked 2026-09-08.
Redis HINCRBY, integer increments and supported range, checked 2026-09-08.
Redis HDEL, removal counts and final-field deletion, checked 2026-09-08.
Redis OBJECT ENCODING, internal representations and conversion, checked 2026-09-08.
Redis memory optimization, compact encoding thresholds and measurement guidance, checked 2026-09-08.
Redis 8.10.1 configuration, default hash thresholds, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only build and allocator selection, checked 2026-09-08.

Grafana variables need the query language to agree

Wed, 23 Feb 2028 00:00:00 +0000

Grafana dashboard variables let one dashboard select an environment, service, or instance without duplicating every panel. A query variable obtains options from a data source; a custom variable can hold a deliberately fixed list. Neither choice is universally better. Use query-backed options when discovery should follow the source, and a maintained static list when the allowed choices are intentionally stable and small.

A variable is substituted into a query before the data source executes it. That makes its formatting part of query correctness. For Prometheus multi-value selections, use a regular-expression label matcher such as environment=~"$environment". Grafana’s Prometheus interpolation handles selected values as regular-expression alternatives with the required escaping. An exact matcher treats the resulting combined text as one label value and therefore does not implement the intended multi-selection. The variable editor’s label_values helper discovers options; it is not a PromQL function to paste into a panel expression.

Check the resulting expression

There are two relevant syntactic layers: the regular expression and the PromQL string containing it. A literal dot in production.eu must not become a wildcard that also matches productionXeu. Let the data source’s interpolation perform its documented formatting, and inspect the expression actually sent when using an explicit formatter. The raw formatter bypasses that data-source formatting; it is not a universal fix for difficult variable values.

The All option also has semantics. With no custom All value, this example expands the available, filtered options. Its menu includes production.eu and staging, so All covers those two environments. A custom expression such as .* would instead match whatever values the backend regex admits. Decide whether “all” means the menu’s current choices or a broader query, and test that decision with deliberately excluded data.

When it breaks

Query variables are not guaranteed to stay fresh forever. Their refresh setting determines when Grafana updates the option query, and time-dependent options may need refresh on time-range changes. Chained variables can still produce an empty downstream selection when the data contains no matching combination. Distinguish a legitimately empty result from a stale selection or a broken query before presenting “No data” as a service outage.

Variable filters are not authorization controls. A menu that hides an environment does not remove that environment from the data source or stop another permitted query from requesting it. Enforce access at the appropriate Grafana and backend boundaries. Shared dashboard URLs also carry selections rather than permissions; multiple values use repeated var-environment parameters, not an invented comma-separated permission list.

Check it quickly

Save the code as grafana-variables.py. With Python and Docker available, run python3 grafana-variables.py --serve. It provisions Grafana 12.1.0 and Prometheus 3.5.1 in disposable containers on a dedicated bridge network, exposes Grafana only on loopback, and mounts no host directories. Open the printed dashboard URL. Press Control-C afterward to remove both containers and their network. Running without --serve executes backend checks and immediately cleans up.

import json
import secrets
import subprocess
import sys
import time
import uuid
from http.client import HTTPException
from threading import Event
from urllib.error import HTTPError, URLError
from urllib.parse import quote
from urllib.request import urlopen

prefix = "xbcatm-variable-" + uuid.uuid4().hex[:10]
network, prom, grafana = [prefix + suffix for suffix in ("-net", "-prom", "-grafana")]


def command(*args):
 try:
 result = subprocess.run(["docker", *args], capture_output=True, text=True, timeout=30)
 except subprocess.TimeoutExpired:
 raise RuntimeError("Docker " + args[0] + " timed out") from None
 if result.returncode:
 raise RuntimeError(result.stderr.strip())
 return result.stdout.strip()


def get(path):
 with urlopen(origin + path, timeout=5) as response:
 return json.load(response)


def eventually(check):
 for attempt in range(200):
 try:
 value = check()
 if value:
 return value
 except (HTTPError, URLError, TimeoutError, ConnectionError, HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Local lab did not become ready")


source = {"type": "prometheus", "uid": "lab_prom"}
dashboard = {
 "uid": "variable-lab", "title": "Variable escaping lab", "schemaVersion": 41,
 "version": 1, "timezone": "utc", "refresh": "", "time": {"from": "now-5m", "to": "now"},
 "templating": {"list": [{
 "name": "environment", "label": "Environment", "type": "query", "datasource": source,
 "query": {"query": "label_values(lab_active_workers, environment)", "refId": "VariableQuery"},
 "definition": "label_values(lab_active_workers, environment)",
 "regex": "/^(production[.]eu|staging)$/", "refresh": 1, "sort": 1,
 "multi": True, "includeAll": True, "allValue": "", "options": [],
 "current": {"text": ["production.eu"], "value": ["production.eu"], "selected": True},
 }]},
 "panels": [{
 "id": number, "title": title, "type": "stat", "datasource": source,
 "gridPos": {"h": 7, "w": 12, "x": (number - 1) * 12, "y": 0},
 "targets": [{"refId": "A", "expr": expression, "instant": True, "range": False}],
 "options": {"reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": False}},
 "fieldConfig": {"defaults": {"decimals": 0}, "overrides": []},
 } for number, title, expression in (
 (1, "Selected workers", 'sum(lab_active_workers{environment=~"$environment"})'),
 (2, "Unescaped control", 'sum(lab_active_workers{environment=~"${environment:raw}"})'),
 )],
}
prom_script = """set -eu
mkdir -p /tmp/www
cat > /tmp/www/metrics.txt <<'METRICS'
# HELP lab_active_workers Synthetic active worker count.
# TYPE lab_active_workers gauge
lab_active_workers{environment="production.eu"} 10
lab_active_workers{environment="staging"} 2
lab_active_workers{environment="productionXeu"} 100
METRICS
cat > /tmp/prom.yml <<'CONFIG'
global:
 scrape_interval: 1s
scrape_configs:
 - job_name: lab
 metrics_path: /metrics.txt
 static_configs:
 - targets: ['127.0.0.1:8080']
CONFIG
/bin/httpd -f -p 127.0.0.1:8080 -h /tmp/www &
exec /bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data
"""
grafana_script = """set -eu
mkdir -p /tmp/data /tmp/provisioning/datasources /tmp/provisioning/dashboards /tmp/dashboards
mkdir -p /tmp/provisioning/plugins /tmp/provisioning/alerting
cat > /tmp/provisioning/datasources/lab.yml <<'DATASOURCE'
apiVersion: 1
datasources:
 - name: Lab Prometheus
 type: prometheus
 uid: lab_prom
 access: proxy
 url: http://metrics:9090
 isDefault: true
 jsonData:
 timeInterval: 1s
DATASOURCE
cat > /tmp/provisioning/dashboards/lab.yml <<'PROVIDER'
apiVersion: 1
providers:
 - name: lab
 type: file
 options:
 path: /tmp/dashboards
PROVIDER
cat > /tmp/dashboards/lab.json <<'DASHBOARD'
""" + json.dumps(dashboard) + """
DASHBOARD
exec /run.sh
"""
try:
 command("network", "create", network)
 command("run", "-d", "--rm", "--name", prom, "--network", network, "--network-alias", "metrics",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-c", prom_script)
 command("run", "-d", "--rm", "--name", grafana, "--network", network,
 "-p", "127.0.0.1::3000", "-e", "GF_PATHS_DATA=/tmp/data",
 "-e", "GF_PATHS_PROVISIONING=/tmp/provisioning",
 "-e", "GF_AUTH_ANONYMOUS_ENABLED=true", "-e", "GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer",
 "-e", "GF_AUTH_DISABLE_LOGIN_FORM=true", "-e", "GF_AUTH_BASIC_ENABLED=false",
 "-e", "GF_SECURITY_ADMIN_PASSWORD=" + secrets.token_urlsafe(24),
 "-e", "GF_ANALYTICS_REPORTING_ENABLED=false", "-e", "GF_ANALYTICS_CHECK_FOR_UPDATES=false",
 "-e", "GF_ANALYTICS_CHECK_FOR_PLUGIN_UPDATES=false", "-e", "GF_PLUGINS_PREINSTALL_DISABLED=true",
 "--entrypoint", "/bin/sh", "grafana/grafana:12.1.0", "-c", grafana_script)
 origin = "http://" + command("port", grafana, "3000/tcp").splitlines()[0]
 health = eventually(lambda: get("/api/health"))
 assert health["version"] == "12.1.0" and health["database"] == "ok"
 loaded = eventually(lambda: get("/api/dashboards/uid/variable-lab"))
 assert loaded["dashboard"]["templating"]["list"][0]["multi"] is True
 checks = {}
 for pattern, expected in ((r"production\\.eu", 10), (r"(production\\.eu|staging)", 12), ("production.eu", 110)):
 expression = 'sum(lab_active_workers{environment=~"' + pattern + '"})'
 result = eventually(lambda: get("/api/datasources/proxy/uid/lab_prom/api/v1/query?query=" +
 quote(expression, safe=""))["data"]["result"])
 value = float(result[0]["value"][1])
 assert value == expected, (expression, result)
 checks[expression] = value
 print(json.dumps({"app_url": origin + "/d/variable-lab/variable-escaping-lab",
 "grafana_version": health["version"], "query_checks": checks}), flush=True)
 if "--serve" in sys.argv:
 Event().wait()
except KeyboardInterrupt:
 pass
finally:
 for name in (grafana, prom):
 subprocess.run(["docker", "rm", "-f", name], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
 subprocess.run(["docker", "network", "rm", network], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The synthetic series hold ten workers for production.eu, two for staging, and one hundred for productionXeu. With only production.eu selected, “Selected workers” must show ten, while the deliberately unescaped control shows one hundred ten. Select both available options, then All: the correct panel must show twelve in each case. The raw control becomes empty because its comma-joined text does not match the intended alternatives.

These three selections were exercised in Chrome 152 against the real Grafana dashboard, and the outgoing query expressions and response values were checked. Repeated URL parameters produced the same multi-selection. This proves interpolation for the provisioned variable and synthetic data, not tenant isolation or freshness under changing discovery. When adapting the dashboard, keep the excluded environment as a regression case and verify the final expression whenever variable type, formatter, refresh policy, or data source changes.

Sources

Grafana variables, variable types and dashboard reuse, checked 2026-09-13.
Grafana Prometheus template variables, multi-value regex matching and refresh behavior, checked 2026-09-13.
Grafana variable syntax, data-source interpolation and explicit formatting overrides, checked 2026-09-13.
Grafana dashboard URL variables, repeated URL parameters for multiple selected values, checked 2026-09-13.