Grafana variables need the query language to agree

Wed, 23 Feb 2028 00:00:00 +0000

Grafana dashboard variables let one dashboard select an environment, service, or instance without duplicating every panel. A query variable obtains options from a data source; a custom variable can hold a deliberately fixed list. Neither choice is universally better. Use query-backed options when discovery should follow the source, and a maintained static list when the allowed choices are intentionally stable and small.

A variable is substituted into a query before the data source executes it. That makes its formatting part of query correctness. For Prometheus multi-value selections, use a regular-expression label matcher such as environment=~"$environment". Grafana’s Prometheus interpolation handles selected values as regular-expression alternatives with the required escaping. An exact matcher treats the resulting combined text as one label value and therefore does not implement the intended multi-selection. The variable editor’s label_values helper discovers options; it is not a PromQL function to paste into a panel expression.

Check the resulting expression

There are two relevant syntactic layers: the regular expression and the PromQL string containing it. A literal dot in production.eu must not become a wildcard that also matches productionXeu. Let the data source’s interpolation perform its documented formatting, and inspect the expression actually sent when using an explicit formatter. The raw formatter bypasses that data-source formatting; it is not a universal fix for difficult variable values.

The All option also has semantics. With no custom All value, this example expands the available, filtered options. Its menu includes production.eu and staging, so All covers those two environments. A custom expression such as .* would instead match whatever values the backend regex admits. Decide whether “all” means the menu’s current choices or a broader query, and test that decision with deliberately excluded data.

When it breaks

Query variables are not guaranteed to stay fresh forever. Their refresh setting determines when Grafana updates the option query, and time-dependent options may need refresh on time-range changes. Chained variables can still produce an empty downstream selection when the data contains no matching combination. Distinguish a legitimately empty result from a stale selection or a broken query before presenting “No data” as a service outage.

Variable filters are not authorization controls. A menu that hides an environment does not remove that environment from the data source or stop another permitted query from requesting it. Enforce access at the appropriate Grafana and backend boundaries. Shared dashboard URLs also carry selections rather than permissions; multiple values use repeated var-environment parameters, not an invented comma-separated permission list.

Check it quickly

Save the code as grafana-variables.py. With Python and Docker available, run python3 grafana-variables.py --serve. It provisions Grafana 12.1.0 and Prometheus 3.5.1 in disposable containers on a dedicated bridge network, exposes Grafana only on loopback, and mounts no host directories. Open the printed dashboard URL. Press Control-C afterward to remove both containers and their network. Running without --serve executes backend checks and immediately cleans up.

import json
import secrets
import subprocess
import sys
import time
import uuid
from http.client import HTTPException
from threading import Event
from urllib.error import HTTPError, URLError
from urllib.parse import quote
from urllib.request import urlopen

prefix = "xbcatm-variable-" + uuid.uuid4().hex[:10]
network, prom, grafana = [prefix + suffix for suffix in ("-net", "-prom", "-grafana")]


def command(*args):
 try:
 result = subprocess.run(["docker", *args], capture_output=True, text=True, timeout=30)
 except subprocess.TimeoutExpired:
 raise RuntimeError("Docker " + args[0] + " timed out") from None
 if result.returncode:
 raise RuntimeError(result.stderr.strip())
 return result.stdout.strip()


def get(path):
 with urlopen(origin + path, timeout=5) as response:
 return json.load(response)


def eventually(check):
 for attempt in range(200):
 try:
 value = check()
 if value:
 return value
 except (HTTPError, URLError, TimeoutError, ConnectionError, HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Local lab did not become ready")


source = {"type": "prometheus", "uid": "lab_prom"}
dashboard = {
 "uid": "variable-lab", "title": "Variable escaping lab", "schemaVersion": 41,
 "version": 1, "timezone": "utc", "refresh": "", "time": {"from": "now-5m", "to": "now"},
 "templating": {"list": [{
 "name": "environment", "label": "Environment", "type": "query", "datasource": source,
 "query": {"query": "label_values(lab_active_workers, environment)", "refId": "VariableQuery"},
 "definition": "label_values(lab_active_workers, environment)",
 "regex": "/^(production[.]eu|staging)$/", "refresh": 1, "sort": 1,
 "multi": True, "includeAll": True, "allValue": "", "options": [],
 "current": {"text": ["production.eu"], "value": ["production.eu"], "selected": True},
 }]},
 "panels": [{
 "id": number, "title": title, "type": "stat", "datasource": source,
 "gridPos": {"h": 7, "w": 12, "x": (number - 1) * 12, "y": 0},
 "targets": [{"refId": "A", "expr": expression, "instant": True, "range": False}],
 "options": {"reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": False}},
 "fieldConfig": {"defaults": {"decimals": 0}, "overrides": []},
 } for number, title, expression in (
 (1, "Selected workers", 'sum(lab_active_workers{environment=~"$environment"})'),
 (2, "Unescaped control", 'sum(lab_active_workers{environment=~"${environment:raw}"})'),
 )],
}
prom_script = """set -eu
mkdir -p /tmp/www
cat > /tmp/www/metrics.txt <<'METRICS'
# HELP lab_active_workers Synthetic active worker count.
# TYPE lab_active_workers gauge
lab_active_workers{environment="production.eu"} 10
lab_active_workers{environment="staging"} 2
lab_active_workers{environment="productionXeu"} 100
METRICS
cat > /tmp/prom.yml <<'CONFIG'
global:
 scrape_interval: 1s
scrape_configs:
 - job_name: lab
 metrics_path: /metrics.txt
 static_configs:
 - targets: ['127.0.0.1:8080']
CONFIG
/bin/httpd -f -p 127.0.0.1:8080 -h /tmp/www &
exec /bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data
"""
grafana_script = """set -eu
mkdir -p /tmp/data /tmp/provisioning/datasources /tmp/provisioning/dashboards /tmp/dashboards
mkdir -p /tmp/provisioning/plugins /tmp/provisioning/alerting
cat > /tmp/provisioning/datasources/lab.yml <<'DATASOURCE'
apiVersion: 1
datasources:
 - name: Lab Prometheus
 type: prometheus
 uid: lab_prom
 access: proxy
 url: http://metrics:9090
 isDefault: true
 jsonData:
 timeInterval: 1s
DATASOURCE
cat > /tmp/provisioning/dashboards/lab.yml <<'PROVIDER'
apiVersion: 1
providers:
 - name: lab
 type: file
 options:
 path: /tmp/dashboards
PROVIDER
cat > /tmp/dashboards/lab.json <<'DASHBOARD'
""" + json.dumps(dashboard) + """
DASHBOARD
exec /run.sh
"""
try:
 command("network", "create", network)
 command("run", "-d", "--rm", "--name", prom, "--network", network, "--network-alias", "metrics",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-c", prom_script)
 command("run", "-d", "--rm", "--name", grafana, "--network", network,
 "-p", "127.0.0.1::3000", "-e", "GF_PATHS_DATA=/tmp/data",
 "-e", "GF_PATHS_PROVISIONING=/tmp/provisioning",
 "-e", "GF_AUTH_ANONYMOUS_ENABLED=true", "-e", "GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer",
 "-e", "GF_AUTH_DISABLE_LOGIN_FORM=true", "-e", "GF_AUTH_BASIC_ENABLED=false",
 "-e", "GF_SECURITY_ADMIN_PASSWORD=" + secrets.token_urlsafe(24),
 "-e", "GF_ANALYTICS_REPORTING_ENABLED=false", "-e", "GF_ANALYTICS_CHECK_FOR_UPDATES=false",
 "-e", "GF_ANALYTICS_CHECK_FOR_PLUGIN_UPDATES=false", "-e", "GF_PLUGINS_PREINSTALL_DISABLED=true",
 "--entrypoint", "/bin/sh", "grafana/grafana:12.1.0", "-c", grafana_script)
 origin = "http://" + command("port", grafana, "3000/tcp").splitlines()[0]
 health = eventually(lambda: get("/api/health"))
 assert health["version"] == "12.1.0" and health["database"] == "ok"
 loaded = eventually(lambda: get("/api/dashboards/uid/variable-lab"))
 assert loaded["dashboard"]["templating"]["list"][0]["multi"] is True
 checks = {}
 for pattern, expected in ((r"production\\.eu", 10), (r"(production\\.eu|staging)", 12), ("production.eu", 110)):
 expression = 'sum(lab_active_workers{environment=~"' + pattern + '"})'
 result = eventually(lambda: get("/api/datasources/proxy/uid/lab_prom/api/v1/query?query=" +
 quote(expression, safe=""))["data"]["result"])
 value = float(result[0]["value"][1])
 assert value == expected, (expression, result)
 checks[expression] = value
 print(json.dumps({"app_url": origin + "/d/variable-lab/variable-escaping-lab",
 "grafana_version": health["version"], "query_checks": checks}), flush=True)
 if "--serve" in sys.argv:
 Event().wait()
except KeyboardInterrupt:
 pass
finally:
 for name in (grafana, prom):
 subprocess.run(["docker", "rm", "-f", name], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
 subprocess.run(["docker", "network", "rm", network], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The synthetic series hold ten workers for production.eu, two for staging, and one hundred for productionXeu. With only production.eu selected, “Selected workers” must show ten, while the deliberately unescaped control shows one hundred ten. Select both available options, then All: the correct panel must show twelve in each case. The raw control becomes empty because its comma-joined text does not match the intended alternatives.

These three selections were exercised in Chrome 152 against the real Grafana dashboard, and the outgoing query expressions and response values were checked. Repeated URL parameters produced the same multi-selection. This proves interpolation for the provisioned variable and synthetic data, not tenant isolation or freshness under changing discovery. When adapting the dashboard, keep the excluded environment as a regression case and verify the final expression whenever variable type, formatter, refresh policy, or data source changes.

Sources

Grafana variables, variable types and dashboard reuse, checked 2026-09-13.
Grafana Prometheus template variables, multi-value regex matching and refresh behavior, checked 2026-09-13.
Grafana variable syntax, data-source interpolation and explicit formatting overrides, checked 2026-09-13.
Grafana dashboard URL variables, repeated URL parameters for multiple selected values, checked 2026-09-13.

Variables · XB Field Notes

Grafana variables need the query language to agree

Check the resulting expression

When it breaks

Check it quickly

Sources