Service · XB Field Notes

Consul keeps a live map of healthy services

Thu, 06 Apr 2028 00:00:00 +0000

A service registry is only as useful as its honesty. Consul keeps a catalog of registered instances and attaches health checks to them, so a caller asking where a healthy instance of a service lives gets an answer that changes as reality changes. Registration is trivial; the check is the whole product, because a registry that lists dead instances is worse than a static list — it actively routes traffic into a hole.

The freshness of that answer is a function of two choices nobody revisits after the first week: how often the check runs, and what it actually tests. A check that runs every 30 seconds is blind for 30 seconds, and a check that only proves the process exists is blind forever.

Registration, and two levels of checks

An instance registers a name, an address, a port and one or more checks, through a config file or the registration API. Checks start in a critical state by default, deliberately, so a service cannot enter the pool before its health has been verified at least once — an initial passing status is something you opt into explicitly.

Two levels of check sit on the same node. A node-level check such as serfHealth carries an empty service name and describes the machine; a service-level check such as service:web-1 is bound to one registration. They fail independently, but the healthy-only filter requires every check on the node to pass, so a critical node-level check removes all of that machine’s services at once while a critical service check removes exactly one instance.

What a caller actually receives

The catalog answers two different questions. Unfiltered, /v1/health/service/web returns every registration with each check’s status; with passing=true it returns only the instances where all checks pass. DNS and SRV always answer the second way, so a resolver never sees a name it cannot use, and the passing set is the load-balancing pool.

The catalog is also eventually consistent, and the mechanism is anti-entropy. Registration is immediate — the agent tells the leader — but repair is periodic: each agent reconciles its local state with the catalog on an interval of one minute for clusters up to 128 nodes, growing with cluster size, and each agent staggers its start so they do not all sync at once. A sync that fails is logged and retried next period, so a wrong catalog entry can survive quietly for up to one interval. The read path has its own trade: by default the agent forwards catalog reads to the leader using a leader lease that assumes leadership is briefly stable, which is fast and can occasionally return a stale result, while the consistent mode makes the leader verify with a quorum first and stale lets any server answer without checking.

When it breaks

The first failure is the interval itself. A check is only as fresh as its period, so a service that dies right after a passing probe stays in rotation until the next probe — 30 seconds for a 30-second check, 60 for a 60-second one. A TTL check is worse than a slow interval because it is passive: the instance reports in on its own schedule, so a dead instance with a 120-second TTL keeps its passing status for the full 120 seconds.

The second failure is a check that answers the wrong question. A script that runs pgrep against the process reports passing for the entire duration of a deadlock — the process exists, the port is bound, and every request routed to it hangs. The catalog cannot tell that instance from a healthy one, so the failure is silent and total for every caller in the pool. A check that exercises a dependency, even imperfectly, converges on the truth within one interval; a liveness-only check never does. The direction of the error is what makes it expensive: a check that fails when it should pass removes a healthy instance and costs capacity, while a check that passes when it should fail keeps feeding traffic to something that cannot serve it, and the caller only finds out through its own error rate.

Check it quickly

Compare the filtered and unfiltered views, and see what DNS hands out:

consul catalog services
consul health service web
curl -s 'http://127.0.0.1:8500/v1/health/service/web' | jq -r '.[] | "\(.Service.ID) \(.Checks[] | select(.Status!="passing") | .CheckID + "=" + .Status)"'
curl -s 'http://127.0.0.1:8500/v1/health/service/web?passing=true' | jq -r '.[] | "\(.Service.ID) \(.Node.Address):\(.Service.Port)"'
dig +short SRV web.service.consul

An instance that shows up unfiltered with a failing CheckID must be absent from the passing=true list and from the SRV answer; if it is still there, the check is not being evaluated at all. A node whose node-level check is critical drops out of both lists entirely, taking every service on that machine with it.

Sources

Service discovery, the catalog, registration and the role of health checks, checked 2026-09-16.
Health checks, check types, intervals and the default critical initial state, checked 2026-09-16.
Health HTTP API, /v1/health/service and the passing filter, checked 2026-09-16.
Anti-entropy, the periodic sync interval and its failure behaviour, checked 2026-09-16.

Socket activation starts a service only when needed

Tue, 01 Feb 2028 00:00:00 +0000

The port is always ready while the process only starts when someone knocks, a neat trick for rare but heavy services. Every idle daemon costs something: resident memory, periodic wakeups, patch surface, log noise. Multiplied across dozens of seldom-used services on one host, the tax funds nothing. Socket activation inverts the deal — the kernel holds the listening socket, systemd owns it before any daemon exists, and the service process is born only when the first connection arrives carrying real work.

The handoff is what makes it transparent. The client connects to the port as usual, unaware nobody is home; systemd starts the service and passes the already-accepted connection in, so not even the first client waits twice. Done right, on-demand startup is indistinguishable from always-on except on the resource bill — and on the first-hit latency, which is the one cost to budget honestly.

The listener waits first

systemd opens the socket ahead of the service, and on the first connection starts the unit and hands it the accepted file descriptor. Configure the ListenStream once so the socket is born and bound before any daemon is up. The socket unit and service unit pair by name — foo.socket launches foo.service — and Accept=no (the default) hands the whole listening socket to one service instance, while Accept=yes spawns a per-connection instance for simple per-request handlers.

This split also buys dependency-free boot ordering. Because the socket exists from early boot, clients can start in any order and simply block or retry until the service is up, instead of encoding fragile After= chains. Queued connections wait in the kernel backlog rather than failing, so boot races become queue depth rather than errors. The backlog size still deserves sizing: a flood of early clients against a slow-starting service overflows the same as any listen queue.

Service wakes on demand

An idle daemon can stay completely stopped, trading per-instance memory and startup cycles for a single handoff cost. Socket-activated services therefore sleep until the first request justifies running them. The pattern shines for administrative tools, legacy protocol gateways, and scheduled-but-rare workers — services whose idle-to-busy ratio is measured in days, not minutes.

Idle exit completes the loop: the service stops itself after N idle seconds, so the system oscillates between zero cost and full capacity. The socket never closes, so the next client re-triggers startup. Without idle exit the process still accumulates — start on demand plus stop on idle is what converts the trick into steady-state savings.

Trade speed for memory

The price is a short warm-up gap on the first hit, which is fine for jobs that run rarely but wasteful on the always-hot path. Enable it where latency on the initial call is acceptable and memory savings actually matter. Quantify both sides before committing: measure cold-start time from connection to first response, and measure the resident set the daemon would otherwise hold forever. A 300ms start against 800MB saved is an easy yes; a 20-second JVM warmup against 50MB is a no.

Some workloads cannot tolerate the gap no matter the savings. Health-check-driven orchestrators may mark the endpoint down during startup, load balancers may eject the slow first backend, and clients with tight timeouts fail before the service finishes warming. For those, keep the daemon running and spend the memory — or warm the service at boot and let activation cover only the long idle stretches afterwards.

When it breaks

The signature failure is the thundering start: a monitoring sweep opens connections to fifty sleeping services at once, all fifty start simultaneously, and the host OOMs or load-spikes into a brownout. Each activation was reasonable alone; together they stampede. Mitigate with staggered check intervals, per-service start limits, and idle-exit timeouts short enough that the steady-state overlap stays small. Test activation under the monitoring load, not just under one curious curl.

The second failure is the half-adopted protocol: a service that binds the port itself at startup collides with the socket systemd already bound. It fails exactly when activated, the connection hangs, logs blame the address in use. Services must accept the passed file descriptor instead of binding; if upstream code cannot, a tiny shim that inherits the socket and execs the daemon bridges the gap.

Check it quickly

Prove on-demand behavior end to end:

systemctl status foo.socket foo.service
ss -ltnp | grep ':8080'
curl -m 10 localhost:8080/health; systemctl show foo.service -p ActiveState,ExecMainStartTimestamp

Before the curl the service is inactive with the socket listening; after, timestamps show a fresh start triggered by the connection.

Sources

systemd.socket manual, Official socket unit reference, checked 2026-06-01.
systemd.service manual, service unit behavior including idle exit, checked 2026-09-05.

NodePort publishes a service on every node's fixed port

Tue, 11 May 2027 00:00:00 +0000

A Service of type NodePort does one blunt thing: it reserves a port in the 30000-32767 range and makes every node in the cluster listen on it. kube-proxy programs the same forwarding rule into the iptables or IPVS tables of every machine, so curl <any-node-ip>:<nodePort> reaches the Service no matter which node answers. That is the whole appeal — external reachability with no cloud load balancer, no DNS record and no ingress controller.

The cost is hidden inside the same design. Because every node answers, a packet frequently lands on a machine that is not running the pod, and the cluster then has to decide whose address the backend sees. externalTrafficPolicy is that decision: Cluster spreads load but rewrites the client address, Local preserves it but drops traffic on any node without a local endpoint. Choosing one is unavoidable, and each choice breaks something different.

A port every node owns

kubectl create service nodeport without --node-port emits no nodePort field at all; the API server allocates one from the range. The allocation is cluster-wide rather than per node, so a given number belongs to exactly one Service across the whole fleet. That is why two Services pinning 31000 cannot coexist, and also why a single firewall rule for one number is portable to every machine.

Every node receives the rule whether or not it runs a backend. That is what makes the type useful — there is no node to avoid and no membership list for the caller to maintain — and it is also what makes it expensive, because the port is open on machines that may have no business accepting that traffic. When the endpoint set is empty the rule still exists and the connection is refused rather than silently dropped, which is at least honest.

The extra hop and the client address

A request arriving on node B is forwarded to node A when that is where the pod lives, so the datapath is one leg longer than hitting node A directly. With the default externalTrafficPolicy: Cluster, kube-proxy also source-NATs the packet to node B’s address, so the pod sees a cluster-internal address rather than the client. Anything that depends on the client address — rate limiting, geo rules, audit logs, an allow list — is then reading the wrong number.

externalTrafficPolicy: Local keeps the original source address by forwarding only to endpoints on the node that received the packet. The price is availability: a node with no local replica has no working node port at all, so a three-node cluster running one replica answers on one node and drops on two. That is the trade — a correct client address and one less hop, in exchange for a node port that only works on some of the nodes.

A side door around the ingress

A node port terminates nothing and inspects nothing. A request sent straight to node:31000 skips the load balancer, TLS termination, the WAF and every host or path routing rule that would have applied, then lands on the pod with whatever the caller chose to send. The pod is left making security decisions it was never written to make.

Treat it as a stopgap with an owner and an expiry date. Reserve the range, pin a number only when a firewall or DNS record forces you to, and restrict who can reach the range at the network layer instead of assuming the ingress is the only way in.

When it breaks

The classic failure is the collision. A nodePort is a single cluster-wide resource, so the second Service that pins a number already in use is rejected with provided port is already allocated, and a Service asking for an automatic port can never get one once the 2768-port range is full. Both present as a Service that exists but has no usable external port, which reads like a networking fault and is really a capacity one.

The second failure is quieter. A node port is a real socket bind on every host, so anything else already listening on that number makes it dead on that machine while kubectl get svc still reports a healthy Service — health checks, dashboards and the Service object all lie together. Because the same port bypasses the ingress layer, the rule that makes debugging easy also makes the cluster reachable in a way nobody is filtering. The blast radius is every node in the fleet, not the one you were looking at.

Check it quickly

Confirm the port, the endpoints, and who else owns the port:

kubectl get svc web -o jsonpath='{.spec.type} {.spec.ports[0].nodePort}{"\n"}'
kubectl get endpoints web
kubectl get svc -A -o json | jq -r '.items[] | select(.spec.type=="NodePort") | "\(.spec.ports[0].nodePort) \(.metadata.namespace)/\(.metadata.name)"' | sort -n | uniq -D -w5
ss -lnt | grep -E ':3[0-9]{4}\b'
curl -sS -o /dev/null -w '%{http_code}\n' "http://$(hostname -I | awk '{print $1}'):$(kubectl get svc web -o jsonpath='{.spec.ports[0].nodePort}')/"

The third line prints any nodePort claimed twice, and the fourth shows whether a process other than kube-proxy already owns that port on this host. The curl should print 200 from every node; a refusal on exactly one node means something local took the port, and a refusal everywhere means the Service has no ready endpoints.

Sources

Kubernetes Service, NodePort type, the 30000-32767 default range and cluster-wide port allocation, checked 2026-09-16.
Virtual IPs and Service Proxies, kube-proxy iptables/IPVS rule programming and externalTrafficPolicy, checked 2026-09-16.
Using Source IP, SNAT on NodePort and the client-IP consequence, checked 2026-09-16.

A Service gives a pod a stable virtual address

Tue, 09 Feb 2027 00:00:00 +0000

The selector turns the Service from a fixed Endpoint into a live view: refresh happens in the background, and a client never edits its config when pods restart.

Declared, then given an address

A Service turns the constant churn of pods into a stable name. You declare a selector (which pods), a port, and a protocol, and the control plane assigns a virtual IP (cluster IP) and a DNS name that stay fixed while everything behind them keeps changing. The stable name is the contract for every other resource: a Deployment, an Ingress, or another pod can address a service by name without caring which concrete pod exists right now. That indirection is what lets the runtime rearrange the membership without a client ever noticing.

Selector is the live membership

The backend list a Service routes to is not a fixed set, it is rebuilt continuously from the selector the moment pods match or stop matching it. A pod that fails its readiness probe is dropped from the endpoints; a new pod that passes joins; the update happens in the background. Because membership is derived, you never hand-edit the list and you never drift: as long as the selector and the pod labels line up, the Service automatically reflects the current, healthy backends. That readme is what makes a Service ahead of static lists you would have to maintain by hand.

Stable even as pods churn

Clients talk only to the virtual address, never to an individual pod IP, which is exactly what decouples the caller from the fleet. You can scale up, scale down, replace, or kill a pod and the Service keeps answering through the survivors; the client sees no change because its target never changes. That stability is the whole reason Services allow rolling restarts and horizontal scaling without a coordinated announcement. Keep the selector precise and probe-gated, and the Service gives you a stable facade over an inherently unstable set of workers.

When it breaks

The pool is finite, and exhaustion is the most literal failure. Every ordinary Service takes one address from --service-cluster-ip-range, so a small CIDR or leftover Services eventually reject creation with an allocation error while the pods look fine. Treat failed to allocate IP as capacity accounting: delete orphaned Services before widening the range, or widening only postpones the outage.

Mixing headless and ordinary Services is the next confusion. Setting clusterIP: None allocates no virtual address, so DNS returns pod IPs instead of one stable VIP, and a client written for a single endpoint must suddenly balance across shifting addresses. The reverse also hurts: discovery code expecting per-pod DNS keeps hitting one VIP after someone recreates the Service with a ClusterIP. Because clusterIP is effectively immutable, switching modes means recreating the Service.

The last two failures live below the API. Iptables and IPVS modes program the same VIP differently, and a stale or restarting kube-proxy can leave an allocated ClusterIP resolving but blackholing traffic until rules resync. Finally, copying a ClusterIP into configs freezes a value the control plane may reassign: the address is stable only while the Service object lives. Callers must use the DNS name, and remember the VIP routes only inside the cluster.

Check it quickly

Save the following as clusterip_pool.py in a scratch directory and run python3 clusterip_pool.py. It models a miniature allocator over 10.96.0.0/29 with the first address reserved, using only the standard library and changing nothing outside the process: one class hands out addresses, refuses duplicates and out-of-range requests, lets headless Services bypass the pool, and reuses released addresses, with five scenarios asserting each rule.

"""Simulate ClusterIP allocation from a small service CIDR."""

import ipaddress


class ClusterIPPool:
 """Hand out virtual addresses; headless Services bypass the pool."""

 def __init__(self, cidr, reserved=1):
 network = ipaddress.ip_network(cidr)
 hosts = list(network.hosts())
 self._all = set(hosts)
 self._reserved = set(hosts[:reserved])
 self.free = sorted(set(hosts) - self._reserved)
 self.allocated = {}

 def allocate(self, service, requested=None, headless=False):
 if headless:
 return None
 if service in self.allocated:
 raise ValueError(f"{service} already holds {self.allocated[service]}")
 if requested is not None:
 ip = ipaddress.ip_address(requested)
 if ip in self._reserved or ip not in self._all:
 raise ValueError(f"{requested} is outside the allocatable range")
 if ip not in self.free:
 raise ValueError(f"{requested} is already taken")
 self.free.remove(ip)
 self.allocated[service] = ip
 return str(ip)
 if not self.free:
 raise RuntimeError("service CIDR exhausted: no ClusterIP left")
 ip = self.free.pop(0)
 self.allocated[service] = ip
 return str(ip)

 def release(self, service):
 ip = self.allocated.pop(service, None)
 if ip is None:
 raise ValueError(f"{service} holds nothing to release")
 self.free.append(ip)
 self.free.sort()
 return str(ip)


def check(condition, message):
 if not condition:
 raise AssertionError(message)


def main():
 pool = ClusterIPPool("10.96.0.0/29", reserved=1)
 web = pool.allocate("web")
 api = pool.allocate("api", requested="10.96.0.5")
 check(web != api, "sequential Services must get distinct addresses")
 print("allocated:", web, api)

 try:
 pool.allocate("worker", requested="10.96.0.5")
 raise SystemExit("conflict was silently shared")
 except ValueError as exc:
 print("conflict rejected:", exc)

 try:
 pool.allocate("evil", requested="10.99.0.1")
 raise SystemExit("out-of-range request was accepted")
 except ValueError as exc:
 print("out-of-range rejected:", exc)

 headless = pool.allocate("db-headless", headless=True)
 check(headless is None, "headless Service must bypass the pool")
 check(len(pool.allocated) == 2, "headless must not consume the pool")
 print("headless bypass: None")

 freed = pool.release("web")
 worker = pool.allocate("worker")
 check(worker == freed, "released address must be reusable")
 print("reused:", freed, "-> worker")

 pool.allocate("extra-1")
 pool.allocate("extra-2")
 pool.allocate("extra-3")
 try:
 pool.allocate("overflow")
 raise SystemExit("exhaustion was not reported")
 except RuntimeError as exc:
 print("exhaustion reported:", exc)

 print("PASS: allocation, conflict rejection, release reuse, and exhaustion hold")


if __name__ == "__main__":
 main()

The run behaves like a healthy allocator: sequential Services get distinct addresses, a pinned request holds its address, and the deliberate double allocation is rejected instead of shared. Releasing one Service makes its address reusable, and filling the last slot proves exhaustion errors instead of recycling a live address. All assertions passed with Python 3.14.4 on macOS.

The simulation checks bookkeeping only. It does not run the API server, etcd, or kube-proxy, and its tiny /29 stands in for a real service CIDR. Before deploying, inspect the live service CIDR, allocated ClusterIPs, and proxy rules on a node.

Sources

Kubernetes Service, documents service semantics, checked 2026-06-01.
Virtual IPs and Service Proxies, documents ClusterIP allocation and kube-proxy modes, checked 2026-06-01.

Service · XB Field Notes

Consul keeps a live map of healthy services

Registration, and two levels of checks

What a caller actually receives

When it breaks

Check it quickly

Sources

Socket activation starts a service only when needed

The listener waits first

Service wakes on demand

Trade speed for memory

When it breaks

Check it quickly

Sources

NodePort publishes a service on every node's fixed port

A port every node owns

The extra hop and the client address

A side door around the ingress

When it breaks

Check it quickly

Sources

A Service gives a pod a stable virtual address

Declared, then given an address

Selector is the live membership

Stable even as pods churn

When it breaks

Check it quickly

Further reading

Sources