Services · XB Field Notes

Kubernetes Services give pods a stable home

Mon, 23 Aug 2027 00:00:00 +0000

You cannot memorise a Pod that will be recreated. A Service fixes an address in front of them, keeping callers stable while pods churn.

One address in front of many

A Deployment gives you multiple, interchangeable pods, and each one has its own IP that will soon be recreated. A Service is the stable home in front of them: you declare a name and a selector, and the platform hands the Service a virtual cluster IP plus a DNS name that stay fixed. Cluster DNS resolves that name, so components (other pods, an Ingress, a load balancer) refer to the Service, never to an individual pod address. That is the point of the abstraction, the address is durable even though everything standing behind it is disposable.

Selection drives which pods respond

A pod does not join a Service by being listed; it joins by carrying the labels the Service’s selector asks for, and it leaves when it no longer matches or is deleted. The platform maintains the endpoint set from the selector, adding a pod once it is running and ready, removing it on scale-down or failure. So the ‘backend list’ is a derived, always-changing view rather than a hand-edited roster. Because membership is derived from labels plus readiness, a pod that becomes unhealthy is dropped automatically and traffic is never aimed at a pod that should not serve.

Traffic goes to whatever matches

All of this gives you indirection that survives churn. Clients hold one name; the set behind it can be scaled, replaced, or rolled without them changing anything. A rolling update swaps pods one by one, readiness-gated, and the Service keeps forwarding to the current healthy matching set, so callers see no interruption. The guarantee you rely on is: as long as the selector is precise and the pods match it, the Service routes to the right set at every moment. Keep the selector tight, gate readiness, and this stable home stays stable even when every backer is new.

The stable name is a promise to callers

A companion article explains how the virtual address itself is allocated; this one is about what that stability means to whoever calls it. A well-behaved client resolves the Service name once, keeps using it for the life of the process, and survives every backend being replaced without a config push. That promise holds only while three habits hold: the selector keeps matching exactly the intended pods, readiness keeps unready pods out of the set, and no caller cheats by caching a pod IP it observed once. Treat the Service name as the only supported handle, and let scaling, crashes, and rollouts happen behind it.

When it breaks

The quietest failure is a selector that matches nothing. A typo such as app: webs instead of app: web, or a Deployment that stopped setting the expected label, leaves the Service with zero endpoints. The cluster IP and DNS name still resolve, so the outage looks like refused connections rather than a missing name. When a Service answers lookups but carries no traffic, inspect its endpoint set first; an empty set points at labels, not at the network.

A headless Service chosen by habit is the next trap. Setting clusterIP: None removes the virtual IP, so DNS returns the changing pod addresses and each client must balance across them. Code written for one stable endpoint suddenly faces many shifting ones, with uneven retries and broken connection reuse. Choose headless only when the caller explicitly wants discovery.

Ports break the promise from the other side. A Service port pointed at the wrong container targetPort accepts traffic it can never deliver, and a hand-picked NodePort can collide with another Service on the same nodes. Finally, leaving sessionAffinity: ClientIP on a stateless pool pins each client to one backend, so one dying pod looks like a total outage. Keep affinity off unless stickiness is the requirement.

Check it quickly

Save the following as service_endpoints.py in a scratch directory and run python3 service_endpoints.py. It models endpoint convergence with plain dictionaries and changes nothing outside the process: one function derives the endpoint set from a selector plus pod labels and readiness, and four scenarios exercise it, including a deliberate empty selector as the negative control.

"""Simulate Service endpoint convergence from selector plus readiness."""

PODS = [
 {"name": "web-1", "labels": {"app": "web"}, "ready": True},
 {"name": "web-2", "labels": {"app": "web"}, "ready": True},
 {"name": "web-3", "labels": {"app": "web"}, "ready": False},
 {"name": "worker-1", "labels": {"app": "worker"}, "ready": True},
]


def select_endpoints(pods, selector):
 """Derive the endpoint set; an empty selector warns instead of matching."""
 if not selector:
 return [], "warning: empty selector matches nothing"
 matched = sorted(
 pod["name"]
 for pod in pods
 if pod.get("ready")
 and all(pod.get("labels", {}).get(key) == value for key, value in selector.items())
 )
 return matched, ""


def check(condition, message):
 if not condition:
 raise AssertionError(message)


def main():
 endpoints, warning = select_endpoints(PODS, {"app": "web"})
 check(endpoints == ["web-1", "web-2"], "ready web pods should form the set")
 check(warning == "", "a precise selector must not warn")
 print("steady state:", endpoints)

 rolling = [dict(pod, ready=True) if pod["name"] == "web-3" else pod for pod in PODS]
 rolling = [pod for pod in rolling if pod["name"] != "web-1"]
 endpoints, _ = select_endpoints(rolling, {"app": "web"})
 check(endpoints == ["web-2", "web-3"], "rollover must swap one ready pod for another")
 print("after rollover:", endpoints)

 endpoints, _ = select_endpoints(PODS, {"app": "web", "track": "stable"})
 check(endpoints == [], "an over-narrow selector should converge to empty, not error")
 print("over-narrow selector:", endpoints)

 endpoints, warning = select_endpoints(PODS, {})
 check(endpoints == [] and "empty selector" in warning, "empty selector must warn loudly")
 print("empty selector:", repr(warning))

 print("PASS: four convergences; readiness-gated selection, rollover, and empty warning hold")


if __name__ == "__main__":
 main()

The four scenarios converge as a healthy Service should: the steady state holds the two ready web pods while the unready one stays out, and the rollover swaps one ready pod for another without changing the selector. The empty selector is the negative control: it warns loudly instead of silently matching everything or nothing. All assertions passed with Python 3.14.4 on macOS.

The simulation checks label-plus-readiness convergence only. It does not run kube-proxy, cluster DNS, or real probe timing. Before deploying, inspect the live endpoint set alongside pod labels and readiness, and repeat the lookup against the real Service name.

Sources

Kubernetes: Services, the Service abstraction, checked 2027-08-15.
Kubernetes: Pods, the pods a Service selects, checked 2027-08-15.
Kubernetes: Deployments, managing the pod set, checked 2027-08-15.

Linger extends a systemd user manager beyond login

Tue, 20 Apr 2027 00:00:00 +0000

Linger changes when an account’s systemd user manager is kept available. With lingering enabled, logind can start that manager at boot and retain it after the account’s login sessions end. This supports unattended work owned by a normal account. The flag controls manager lifetime; individual services still have their own activation, failure, shutdown, and installation rules.

When it breaks

Enabling linger does not enable every unit file found in the account’s configuration directory. A service must be requested through enablement, a dependency, a timer, a socket, or another explicit activation path. A manager can be running correctly while the intended application remains inactive. Check both objects instead of interpreting the linger flag as a list of running services.

Processes do not survive a machine reboot. Linger can arrange for a new user manager to start at boot, and enabled units can then start new processes. Their previous in-memory state is gone. An encrypted or late-mounted home can also make unit files or application data unavailable at the time the manager starts. Persist required state and make boot-time dependencies intentional.

Linger is not an instruction to restart a crashed application forever or to adopt arbitrary shell jobs into managed services. Restart policy, resource limits, and logind’s treatment of session processes are separate controls. A stable background role needs a real user unit and an observed result, not just a shell command launched before logout.

Check it quickly

Run this Python script as an administrator in a disposable Linux system with systemd, logind, user D-Bus support, and account-management tools. The recorded environment was Debian 12 with systemd 252.39. It creates a temporary account and home, enables lingering only for that account, and removes its changes afterward. No login session is opened for the account:

from pathlib import Path
from tempfile import TemporaryDirectory
import os,pwd,subprocess,time,uuid

name='xbcatm-linger-'+uuid.uuid4().hex[:8];unit='field-worker.service'
def run(*args):
 result=subprocess.run(args,capture_output=True,text=True,timeout=20)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def wait_for(check,seconds=5):
 deadline=time.monotonic()+seconds
 while not check():
 assert time.monotonic()<deadline,'state transition timed out';time.sleep(0.05)
with TemporaryDirectory(prefix=name+'-',dir='/run') as directory:
 root=Path(directory);root.chmod(0o755);home=root/'home';home.mkdir()
 run('useradd','--no-create-home','--user-group','--home-dir',str(home),'--shell','/bin/sh',name)
 account=pwd.getpwnam(name);uid=account.pw_uid;gid=account.pw_gid;manager=f'user@{uid}.service'
 def userctl(*args):
 return run('setpriv','--reuid='+str(uid),'--regid='+str(gid),'--init-groups',
 'env',f'XDG_RUNTIME_DIR=/run/user/{uid}',f'DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/{uid}/bus',
 'systemctl','--user',*args)
 def manager_value(key):return run('systemctl','show',manager,'-p',key,'--value')
 try:
 units=home/'.config/systemd/user';units.mkdir(parents=True)
 (home/'worker.py').write_text("from pathlib import Path\nimport os,time\np=Path(__file__).with_name('worker-pid');t=p.with_suffix('.tmp');t.write_text(str(os.getpid()));t.replace(p)\ntime.sleep(60)\n")
 (units/unit).write_text('[Service]\nType=exec\nExecStart=/usr/bin/python3 %h/worker.py\n[Install]\nWantedBy=default.target\n')
 for path in [home,*home.rglob('*')]:os.chown(path,uid,gid)
 run('loginctl','enable-linger',name)
 wait_for(lambda:Path(f'/run/user/{uid}/bus').exists())
 assert run('loginctl','show-user',name,'-p','Linger','--value')=='yes'
 assert run('loginctl','show-user',name,'-p','Sessions','--value')==''
 assert manager_value('ActiveState')=='active'
 assert userctl('show',unit,'-p','ActiveState','--value')=='inactive'
 print('linger starts a manager with zero login sessions; merely installing a user unit does not start it')
 userctl('enable','--now',unit);wait_for(lambda:(home/'worker-pid').exists())
 first=(home/'worker-pid').read_text();first_manager=manager_value('MainPID')
 run('systemctl','restart',manager)
 wait_for(lambda:Path(f'/run/user/{uid}/bus').exists() and (home/'worker-pid').read_text()!=first)
 assert manager_value('MainPID')!=first_manager
 assert userctl('show',unit,'-p','MainPID','--value')==(home/'worker-pid').read_text()
 assert run('loginctl','show-user',name,'-p','Sessions','--value')==''
 print('enabled user service starts as a new process after a real manager restart')
 userctl('disable',unit)
 assert userctl('show',unit,'-p','ActiveState','--value')=='active'
 print('disable removes future enablement but does not stop the current service')
 userctl('stop',unit);run('loginctl','disable-linger',name)
 wait_for(lambda:manager_value('ActiveState')=='inactive',seconds=20)
 print('with no sessions and linger disabled, user manager stops after policy delay')
 finally:
 run('loginctl','disable-linger',name)
 run('systemctl','stop',manager,f'user-runtime-dir@{uid}.service')
 run('userdel',name)

After enable-linger, the user manager becomes active and its bus appears. logind reports lingering enabled and an empty session list. However, the installed worker unit is still inactive. This confirms that keeping the manager available is different from requesting the application to run, even though the configuration file already exists.

The script then enables and starts the worker through the correct user manager. Its installation section links it to the user’s default target, while the explicit start begins current execution. A PID marker identifies the actual worker. The controller uses setpriv and the user’s runtime bus, so the control command itself does not quietly create a login session.

Restarting the system-level user@UID.service replaces the account manager. The enabled worker subsequently publishes a different PID, and the new user manager reports that same process as its main worker. The session list remains empty. This tests reconstruction from persistent unit enablement across a real manager restart, rather than merely observing the same process for a little longer.

Disabling the worker without a stop request leaves the current service active. Disablement removes future enablement links; it is not an implicit termination command. The script checks that distinction, then explicitly stops the worker and disables lingering. With no sessions remaining, the account manager becomes inactive under the configured stop policy.

Cleanup disables lingering again if needed, stops the experiment’s manager and runtime-directory unit, and deletes the temporary account and home. The exercise deliberately uses temporary configuration and does not reboot the machine. Its reboot-related statements come from the documented lifecycle contract; the observed process replacement is a manager restart, not a claimed reboot test.

Operate an unattended account deliberately

Keep the account’s enabled units reviewable and remove services that are no longer needed. A service that remains available after logout can retain memory, open files, credentials, and network listeners. Ordinary user authority limits some actions but still permits everything granted to that identity. Use suitable permissions and supported resource controls for the actual workload.

Make configuration and credentials available through mechanisms that work without an interactive shell. A successful launch from a terminal does not prove unattended startup has the same environment. Verify working directories, input paths, and access after restarting the manager, then perform a real boot test when boot-time availability is a deployment requirement.

When retiring the role, distinguish disabling a service, stopping current execution, disabling linger, and deleting account data. Other sessions or activation paths may still matter. Inspect the account’s actual manager and service states after the intended change. The useful guarantee is a documented unattended lifecycle with explicit activation and shutdown, not the indefinite survival of one PID.

Sources

systemd 252 loginctl, enabling and disabling per-account lingering and inspecting sessions, checked 2026-09-12.
systemd 252 user@.service, account manager and runtime-directory lifecycles, checked 2026-09-12.
systemd 252 logind.conf, user stop delay and the distinction between sessions and user services, checked 2026-09-12.
systemd 252 systemctl, enable/disable versus start/stop behavior, checked 2026-09-12.

Services · XB Field Notes

Kubernetes Services give pods a stable home

One address in front of many

Selection drives which pods respond

Traffic goes to whatever matches

The stable name is a promise to callers

When it breaks

Check it quickly

Further reading

Sources

Linger extends a systemd user manager beyond login

When it breaks

Check it quickly

Operate an unattended account deliberately

Further reading

Sources