Prometheus has several relabeling stages, and their similar syntax can hide different effects. Target relabeling transforms discovered targets before collection. Metric relabeling transforms scraped samples just before ingestion. A drop action at the first stage can prevent an endpoint from being scraped; the same action at the second stage discards matching samples from an endpoint that was already contacted. ...
One incident can produce many alerts, but a group is not proof that those alerts have one cause. Alertmanager groups alerts according to configured labels and routing context so their notifications can be handled together. The original alert identities remain available inside the group. Silences add another operation: they suppress matching notifications for a time window without resolving the underlying conditions. ...
When several Prometheus servers send data to a common system, downstream readers need to know which collector or environment supplied it. Global external_labels can add that context during communication with external systems. They do not rewrite every locally stored series, and they do not override labels already present on the outgoing series. Understanding both boundaries prevents misleading configuration checks. ...
A short-lived job can finish before Prometheus next scrapes it. Pushgateway offers an intermediary that retains pushed metrics for later scraping, but it is not the default answer for every short process. Prometheus recommends it primarily for suitable service-level batch outcomes. The retained data has a lifecycle independent of the job, so successful scraping does not prove the job recently ran or succeeded. ...
Service discovery produces candidate targets and metadata. Target relabeling transforms those candidates before Prometheus scrapes them. It can keep or drop targets, change how they are addressed, and create labels carried into collected series. This is a different stage from rewriting individual scraped samples. Understanding the stage prevents a configuration intended to remove one metric from accidentally removing an entire target. ...
A missing metric is not a numeric zero. Prometheus distinguishes a series with a recent value, a series marked stale, and a selector with no usable sample. Alerting then applies its own evaluation and waiting rules. Treating all these mechanisms as one universal grace period produces alerts that miss disappearing targets or report an outage without establishing what actually disappeared. ...
A latency mean can conceal a slow subset of requests. A histogram retains more distribution information by counting observations into buckets, but it still discards detail within each bucket. Prometheus can estimate quantiles from those counts without storing every observation. The estimate depends on bucket boundaries and aggregation; it is not a recovery of the complete original latency distribution. ...
A metric named request_duration leaves readers guessing whether 250 means seconds, milliseconds, or something else. A Prometheus-facing name such as http_request_duration_seconds carries that contract into queries, recording rules, and alerts. The contract still needs a correct implementation. Appending _seconds to measurements collected in milliseconds changes the spelling, not the magnitude, and can create a thousandfold monitoring error. ...