Autoscaling · XB Field Notes

The Horizontal Pod Autoscaler scales by a target utilization

Wed, 22 Dec 2027 00:00:00 +0000

Because it can act on the mean of a series instead of a single spike, you need to pair it with a max-rc that keeps the maths from running forever.

A smoothed average decides

The HPA does not react to a single instantaneous read; it averages the observed value over a window and smooths it before comparing against the target. That is deliberate: a one-second spike in a metric should not repaint the replica count, so the controller aggregates first, then only changes the count when the smoothed value moves past the target. To make that work you choose the metric source (CPU, custom metrics API, or external) and the target, because the smoothing window and the aggregation method, per-pod average versus cluster level, together define what ’the load’ actually is. A metric read over too short a window reacts to noise; too long and it lags demand. Reading the document’s sync-period and aggregation paragraph is the first step in tuning, not the last.

Pick the metric you can act on

The real design choice is the metric, because scaling is only as useful as the target you give it. CPU is easy on any cluster and works well for CPU-bound workloads, but it is a proxy: a service waiting on queues or a database is often standing idle on CPU while its latency explodes. Demand-driven metrics such as request rate or a custom kafka_consumer_lag are harder to wire but report the thing users feel. Whatever you pick, choose one you can act on, a target that, when crossed, really changes the experience, and use the same definition the app uses, so the HPA is scaling the number that matters rather than a number that happens to be cheap to read.

Automation is orthogonal to defaults

Set the initial replicas in the Deployment because that is the starting point, not because the HPA reads it, once the HPA exists it owns the count and the initial value is only a floor or a first estimate. The two are orthogonal in the good way: you can disable the controller and still have a sane static stated count, and you can run a steady HPA without a fancy initial guess. Keep the minimum realistic (that is the real floor), pick a maximum that bounds spend, and remember that the initial value is what makes the very first seconds sane before the controller’s first reconcile lands.

When it breaks

A missing custom metric does not fall back to CPU. When the custom-metrics API returns an error or no series, the HPA marks ScalingActive False and holds replicas instead of guessing, so a workload with only a queue metric freezes exactly when it needed scaling. List CPU alongside the custom metric and let the maximum win, keeping a degraded but live input.

A Prometheus Adapter pipeline also adds delay. Scrape intervals plus adapter caching can serve queue depth from minutes ago, causing scale-up against drained work and overshoot. Check the series timestamp and adapter latency before retuning the target, and expose one pre-aggregated series per workload.

Cardinality breaks the same path differently: per-user or per-request labels multiply one signal into thousands of series, slow the API, and flip readings to Unknown. Aggregate in Prometheus first. A target-type mismatch does the same: custom metrics accept only Value or AverageValue, and CPU utilization without pod requests stays Unknown. Describe the HPA and align the type with what the pipeline serves.

Check it quickly

Save the following as hpa_metric_sim.py and run python3 hpa_metric_sim.py. It uses only the standard library, touches no cluster, and models the rules above: multiple metrics combine by maximum, missing metrics are skipped, and an all-missing poll holds replicas.

import math
import sys


def desired_for(current, value, target):
 """Replicas wanted by one metric; None means unavailable."""
 if value is None or target is None or target <= 0:
 return None
 return math.ceil(current * value / target)


def decide(current, metrics, min_replicas, max_replicas):
 """Combine metrics by maximum; hold when every metric is missing."""
 wants = []
 for _, value, target in metrics:
 want = desired_for(current, value, target)
 if want is not None:
 wants.append(want)
 if not wants:
 return current
 return max(min_replicas, min(max_replicas, max(wants)))


def check(actual, expected, message):
 if actual != expected:
 raise AssertionError(f"{message}: got {actual}, want {expected}")
 print(f"OK: {message} -> {actual}")


def main():
 current = 4
 cpu = ("cpu", 80, 50) # 4 * 80/50 = 6.4 -> 7
 queue = ("queue", 900, 300) # 4 * 900/300 = 12
 check(decide(current, [cpu, queue], 1, 20), 12, "max wins across metrics")
 check(decide(current, [cpu, ("queue", None, 300)], 1, 20), 7, "one missing metric still scales on CPU")
 check(decide(current, [("cpu", None, 50), ("queue", None, 300)], 1, 20), 4, "all missing holds replicas")
 check(decide(current, [cpu, queue], 1, 10), 10, "maxReplicas still clamps metric math")
 print("PASS: 4 scenarios; max-wins and hold-on-missing verified")
 print(sys.version.split()[0])


if __name__ == "__main__":
 main()

The four scenarios prove the maximum wins when CPU asks for seven and queue depth asks for twelve, a single missing series still leaves CPU in charge, all-missing holds instead of scaling to zero, and the maximum bound still clamps the result. Expected output ends with PASS: 4 scenarios; max-wins and hold-on-missing verified.

The assertions passed with Python 3.12.6 on macOS. The script checks replica math only, not scrape latency, adapter caching, or cardinality. Before changing production, verify timestamps and HPA events against the real custom-metrics API.

Sources

Kubernetes Horizontal Pod Autoscaling, documents algorithm and metric sources, checked 2026-06-01.
Resource Metrics Pipeline, documents metrics-server and the resource metrics API, checked 2026-06-01.

The horizontal autoscaler scales on demand

Thu, 04 Nov 2027 00:00:00 +0000

The autoscaler derives desired replicas from current usage and a target, so the metric you choose matters more than the replica count you set.

Metrics drive the count

The HPA does not watch load directly; it polls a configured metric on a schedule, reads a current value against the target you set, and derives a desired replica count from the ratio. If the observed value runs twice the target it scales up, if half it scales down, and the result is recomputed on each reconciliation. That is why the metric choice, its aggregation (per pod or cluster-level), and the window matter, they define what the load even is. The desired count is then clamped by maxReplicas and minReplicas, and reconciled on the default --horizontal-pod-autoscaler-sync-period. To make scaling predictable you reason about the input metric, not about a single constant.

Scale up eagerly, down gently

Scale-down is deliberately asymmetric. The controller does not remove a pod the instant utilization dips, because a transient spike would otherwise ping-pong the replica count and thrash the cluster. It waits out a stabilization window set by --horizontal-pod-autoscaler-downscale-stabilization, keeps a floor via minReplicas, and only then shrinks. Scaling up is fast, because queued latency hurts right now; scaling down is slow, because idle capacity is cheaper than a flapping fleet. A dependable setup states a floor generous enough for the spikes you actually see, a target it can digest, and treats the stabilization window as the amount of load noise you are willing to ride out, so the group does not bounce every time the graph blips.

The metric is the weak link

The whole controller inherits the quality of your metric. If the metric lags real demand, the controller is always chasing a stale number; if it watches a proxy that correlates poorly with the actual bottleneck, the controller scales the wrong thing. So the hard part of an HPA is rarely the YAML, it is the metric and the target. Putting CPU on a latency-bound service, or memory on an I-O-bound worker, often points scaling in the wrong direction. Begin from what actually saturates, requests in flight, queue depth, or connections, choose an average over a window you can act on, and prove that a rise in that metric really predicts worse latency. Then the HPA becomes a tool that scales your workload, not a timer that wastes a guess.

When it breaks

Stale or missing metrics make the controller flap. When the metrics pipeline lags, HPA scales against last minute’s load: it adds replicas after the spike already passed, then holds them through the downscale stabilization window, so dashboards show oscillation rather than tracking. If metrics go missing entirely, the controller refuses to scale and surfaces ScalingActive=False; treating “no data” as “no load” and deleting capacity is the classic outage version of this bug.

Bounds freeze the loop in the other direction. A maxReplicas set from cost anxiety caps the fleet during a real surge and reports ScalingLimited=True, while a generous minReplicas hides an overestimated target because the floor never lets the controller prove demand fell. Finally, do not let two controllers steer the same knob: HPA fighting VPA (one changes count, the other changes requests, which moves utilization) or an aggressive rolling update (new pods report zero usage while old pods still serve) produces sawtooth replicas that no target tuning will fix.

Check it quickly

Save the following as hpa_loop.py in a scratch directory and run python3 hpa_loop.py. It models one HPA control loop with pure standard library: desired replicas are the ceiling of current average utilization over target, clamped to min and max, with missing metrics holding the last decision and a downscale stabilization window blocking fast shrink. The script asserts scale-up on a hot sample, cooldown-blocked scale-down on the next cool sample, eventual shrink after the window, and clamping at both bounds.

import math

TARGET = 50.0
MIN_REPLICAS = 2
MAX_REPLICAS = 10
STABILIZATION_S = 300


def desired(current_replicas, current_util):
 raw = math.ceil(current_replicas * current_util / TARGET)
 return max(MIN_REPLICAS, min(MAX_REPLICAS, raw))


def step(state, now, current_util):
 replicas, last_scale = state
 if current_util is None:
 print(f"t={now}s metric=missing hold={replicas}")
 return state
 want = desired(replicas, current_util)
 if want < replicas and now - last_scale < STABILIZATION_S:
 print(f"t={now}s metric={current_util} want={want} hold={replicas} (cooldown)")
 return state
 print(f"t={now}s metric={current_util} {replicas}->{want}")
 if want != replicas:
 return (want, now)
 return state


def main():
 state = (3, -10**9)
 state = step(state, 0, 100)
 assert state[0] == 6, state
 state = step(state, 60, 15)
 assert state[0] == 6, state
 state = step(state, 400, 15)
 assert state[0] == 2, state
 state = step(state, 460, None)
 assert state[0] == 2, state
 assert desired(2, 500) == 10
 assert desired(2, 5) == 2
 print("PASS: scale-up 3->6, cooldown holds 6, then 6->2; clamps at 10 and 2")


if __name__ == "__main__":
 main()

The run prints each decision and ends with PASS. The hot sample grows 3 replicas to 6, the immediate cool sample stays at 6 while the stabilization window is fresh, the same cool value shrinks to 2 after the window expires, and out-of-range calculations stick at max 10 and min 2. The fixture checks control-loop math, not metrics-server latency or startup time. Validate cluster tuning against kubectl describe hpa conditions.

Sources

Kubernetes horizontal pod autoscaling, how HPA derives replica counts, checked 2026-05-06.
HorizontalPodAutoscaler walkthrough, worked example with multiple metrics and status conditions, checked 2026-05-06.

Autoscaling · XB Field Notes

The Horizontal Pod Autoscaler scales by a target utilization

A smoothed average decides

Pick the metric you can act on

Automation is orthogonal to defaults

When it breaks

Check it quickly

Further reading

Sources

The horizontal autoscaler scales on demand

Metrics drive the count

Scale up eagerly, down gently

The metric is the weak link

When it breaks

Check it quickly

Further reading

Sources