A useful dashboard helps a reader decide what to inspect next. It reduces a large collection of measurements into a small set of questions: are users succeeding, where is the impact, and is capacity becoming constrained? Reduction must preserve important differences. A green global percentage can conceal a failing region, and a blank panel can mean missing telemetry rather than a quiet, healthy service.

Give the page a reading order

Start with the service outcome and observation coverage, then show traffic, latency, errors, and relevant saturation. Provide drilldowns for regions, operations, or dependencies when those distinctions change the next action. Put deployment annotations and links to diagnostic material near the relevant views. Correlated lines can suggest where to look, but their timing alone does not prove that a deployment or dependency caused the incident.

Each panel should answer a clear question, not necessarily contain exactly one metric. Comparing queue depth with its configured limit can be useful when both share a unit. Mixing unrelated units on a complicated dual-axis chart makes apparent relationships easy to invent. State the unit, aggregation window, population, and threshold meaning. A reader should not need to remember whether an axis represents milliseconds, seconds, fractions, or percentages.

Preserve weighting and coverage

Compute a service error fraction from the total bad events divided by total eligible events over matching windows. An unweighted mean of regional percentages gives a quiet region the same influence as a busy one. That is a different statistic, not a harmless display choice. Even a correctly weighted result should have a useful breakdown when small populations matter to the service commitment.

Freshness is another dimension of scope. Display the last successful observation or an explicit coverage signal, and define when stale data becomes unknown. A service-specific freshness threshold is separate from the storage system’s staleness rules. Filling gaps with zero can turn an unavailable exporter into an apparently perfect error rate or an empty queue. Missing data should remain visible, with a next step to investigate collection or the service itself.

When it breaks

Automatically chosen axis ranges can make a tiny variation look like a major incident, while an overly broad fixed range can conceal a meaningful threshold crossing. Select scales according to the operational question and keep comparable panels consistent. An average latency also hides a slow minority; use appropriate distributions and counts without averaging precomputed percentiles across instances.

An attractive dashboard can still become a collection of unrelated snapshots. Different time zones, filters, refresh intervals, and aggregation windows prevent a coherent comparison. Shared variables should have explicit defaults and visible scope, especially for production environments. Test the page with a regional failure, sparse traffic, and absent measurements. If those situations all appear green, the page is compressing away information needed for decisions.

Check it quickly

Save this as dashboard.py and run Python 3. It computes panel-ready values from synthetic counts for one completed sixty-second interval. Two regions have fresh observations; a third has missing values and an older observation time. The ninety-second freshness threshold is an example dashboard policy, not a claim about Prometheus defaults. No Grafana installation is required for this aggregation exercise.

import json
from fractions import Fraction

# Counts cover the same completed 60-second interval.
rows = [
    {"region": "west", "requests": 9800, "errors": 0, "age_seconds": 10},
    {"region": "east", "requests": 200, "errors": 100, "age_seconds": 10},
    {"region": "south", "requests": None, "errors": None, "age_seconds": 240},
]
fresh = [r for r in rows if r["age_seconds"] <= 90
         and r["requests"] is not None and r["errors"] is not None]
missing = [r["region"] for r in rows if r not in fresh]
requests = sum(r["requests"] for r in fresh)
errors = sum(r["errors"] for r in fresh)
regional = {r["region"]: Fraction(r["errors"], r["requests"]) for r in fresh}
known_ratio = Fraction(errors, requests)
unweighted = sum(regional.values()) / len(regional)

assert known_ratio == Fraction(1, 100)
assert regional["east"] == Fraction(1, 2) and unweighted == Fraction(1, 4)
assert missing == ["south"]
print(json.dumps({
    "coverage": "incomplete",
    "fresh_requests_per_second": round(requests / 60, 3),
    "fresh_error_percent": float(known_ratio * 100),
    "regional_error_percent": {k: float(v * 100) for k, v in regional.items()},
    "wrong_unweighted_percent": float(unweighted * 100),
    "unknown_regions": missing,
}, indent=2))

Fresh observations contain 10,000 requests and 100 errors, giving 166.667 requests per second and a weighted 1% error fraction. The small eastern region is nevertheless failing half its requests. Averaging the two regional percentages would incorrectly label the combined request fraction as 25%. Show the 1% overview with its regional breakdown instead of choosing one statistic to hide the other.

The output also declares coverage incomplete and lists the southern region as unknown. Neither its requests nor its errors are invented as zero. This fixture checks the arithmetic and missing-data meaning behind the panels; it does not validate an actual dashboard query or screen. Carry the same definitions into the rendered dashboard and review it against known incident and telemetry-loss cases.

Sources