Alert labels identify which condition is being tracked and provide fields for routing and grouping. Annotations explain that condition: a current value, a concise summary, or a runbook link. Putting changing prose in labels can split one ongoing problem into many identities. Putting routing fields only in annotations can leave a route unable to recognize the alert’s intended owner. ...
A recording rule runs a PromQL expression at scheduled evaluation times and stores the result as a new metric. Repeated dashboard queries can then read that metric instead of recomputing the expression. Calling this a cache can obscure an important difference: the recorded series has its own sample history, cadence, and labels. It is not a transparent cache of every possible query request. ...
A Prometheus series is identified by a metric name and its complete label set. Adding customer or request identity to a counter can create many series even when every dashboard immediately sums those identities away. Query aggregation changes the returned result; it does not remove the raw series that were ingested. Storage and ingestion costs arise before a user opens the dashboard. ...
A request counter answers how many events an instrumented process has accumulated. A rate answers how quickly those events are arriving. Prometheus rate estimates average growth per second over a selected range, adjusts observed counter resets, and extrapolates toward the range boundaries. The result is useful precisely because it has a time unit, but it is not an exact reconstruction of every event between scrapes. ...
Alertmanager selects notification receivers by walking a routing tree against an alert’s labels. The tree expresses ownership and escalation policy, while receivers describe notification integrations. A configuration can parse correctly and still send the wrong class of alert to the wrong destination. Verify the label contract and tree behavior before treating a successful reload as evidence that routing works. ...
An alerting rule evaluates a PromQL expression and tracks the returned label sets over time. Prometheus decides whether each resulting alert is pending or firing. Alertmanager receives alert information and handles notification routing, grouping, silencing, and inhibition. A reliable investigation follows those stages separately: a firing alert is evidence of rule state, not proof that a person received a message. ...
Repeated service-level queries are good candidates for recording rules when their cost and reuse justify precomputation. The design question is what to store. Saving only an instance’s error percentage can discard the traffic volume needed for later aggregation. Saving error rates and request rates preserves the numerator and denominator, allowing consumers to calculate a correctly weighted service fraction. ...
Prometheus collects metrics by requesting an HTTP endpoint on each configured target. The target exposes observations; the collector chooses when to scrape, parses the response, and stores samples. This arrangement makes collection failures observable from the collector’s perspective. It does not guarantee that the endpoint is reachable, that every expected target was discovered, or that the application’s user journeys work. ...