Monitoring

A disk that spikes with no obvious writer is hard to chase; iotop maps throughput straight back to the task doing the damage. System-wide tools show the symptom — high await, saturated utilization — but never the suspect: fifty processes share the disk and the average blames all of them equally. The question that ends the investigation is always per-process — who, how much, since when — and iotop answers exactly that, live, from kernel counters no application can hide from. ...

Grafana dashboard variables let one dashboard select an environment, service, or instance without duplicating every panel. A query variable obtains options from a data source; a custom variable can hold a deliberately fixed list. Neither choice is universally better. Use query-backed options when discovery should follow the source, and a maintained static list when the allowed choices are intentionally stable and small. ...

A request counter answers how many events an instrumented process has accumulated. A rate answers how quickly those events are arriving. Prometheus rate estimates average growth per second over a selected range, adjusts observed counter resets, and extrapolates toward the range boundaries. The result is useful precisely because it has a time unit, but it is not an exact reconstruction of every event between scrapes. ...

An alerting rule evaluates a PromQL expression and tracks the returned label sets over time. Prometheus decides whether each resulting alert is pending or firing. Alertmanager receives alert information and handles notification routing, grouping, silencing, and inhibition. A reliable investigation follows those stages separately: a firing alert is evidence of rule state, not proof that a person received a message. ...

A metric costs more than its short name suggests. Prometheus identifies a time series by its metric name and complete label set. A new observed label combination creates a different series, which must be ingested, indexed, retained, and queried. Cardinality budgeting asks which distinctions are worth paying for and whether their number stays bounded as the service grows. ...

A useful dashboard helps a reader decide what to inspect next. It reduces a large collection of measurements into a small set of questions: are users succeeding, where is the impact, and is capacity becoming constrained? Reduction must preserve important differences. A green global percentage can conceal a failing region, and a blank panel can mean missing telemetry rather than a quiet, healthy service. ...

When several Prometheus servers send data to a common system, downstream readers need to know which collector or environment supplied it. Global external_labels can add that context during communication with external systems. They do not rewrite every locally stored series, and they do not override labels already present on the outgoing series. Understanding both boundaries prevents misleading configuration checks. ...

A missing metric is not a numeric zero. Prometheus distinguishes a series with a recent value, a series marked stale, and a selector with no usable sample. Alerting then applies its own evaluation and waiting rules. Treating all these mechanisms as one universal grace period produces alerts that miss disappearing targets or report an outage without establishing what actually disappeared. ...