Prometheus collects metrics by requesting an HTTP endpoint on each configured target. The target exposes observations; the collector chooses when to scrape, parses the response, and stores samples. This arrangement makes collection failures observable from the collector’s perspective. It does not guarantee that the endpoint is reachable, that every expected target was discovered, or that the application’s user journeys work.

A job contains independently scraped instances

A job groups targets with a shared purpose, such as several API replicas. Each target is an instance with its own scrape outcome. The usual job and instance labels identify that context, although relabeling can change the final labels. Service discovery supplies candidates; scrape configuration determines their endpoint, protocol, interval, timeout, and authentication. Discovery and collection therefore have separate failure modes.

The collector must be able to connect to the target. An exporter needs an inbound path from authorized collectors, even though it does not initiate metric uploads to them. Network policy, TLS identity, credentials, and routing remain deployment responsibilities. A pull design changes who initiates the connection; it does not remove network boundaries or make arbitrary exporters safe to expose publicly.

Read the generated metric precisely

For a configured instance, Prometheus generates up, with value one for a successful scrape and zero for a failed scrape. It is a metric, not an up label. Scrape duration and sample counts provide additional collection evidence. An exporter can return a valid response while the database behind it is unavailable, or while an important business metric is missing. Such a scrape can still succeed.

Use application outcomes and exporter-specific dependency metrics to answer those further questions. Conversely, up=0 describes a collection failure, which might be caused by the monitoring path while users remain unaffected. Keep that distinction in alert wording so responders investigate the correct surface instead of assuming every collection problem is a customer outage.

When it breaks

A connection refusal, timeout, certificate error, or invalid exposition can prevent a scrape from succeeding. Inspect the target’s last scrape error before changing the interval. A slower interval does not repair a wrong port, and a larger timeout can conceal a slow exporter without removing its cost. Review emitted series and response size when collection becomes expensive. Keep scrape timeouts within the configured interval, and evaluate exporter load before shortening the schedule. Faster polling can multiply database queries performed by an exporter without making those queries cheaper.

A target removed from discovery is different from a configured target returning zero. Its absence can eventually remove the corresponding series from instant queries. Therefore, a fleet-wide sum of up cannot prove that every expected instance still exists. Compare discovery inventory with the intended deployment when completeness matters. Likewise, a successful /metrics request is not a substitute for testing a checkout or login operation.

Check it quickly

Save this as pull.py and run python3 pull.py with Docker available. It uses prom/prometheus:v3.5.1; an uncached image requires an initial registry download. One disposable container scrapes its own loopback endpoint and an unused loopback port. No host port is published, and the container has no external network access.

import json
import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
# All HTTP traffic stays on loopback inside one disposable container.
script = r"""set -eu
cat > /tmp/prom.yml <<'CONFIG'
global:
  scrape_interval: 1s
  scrape_timeout: 500ms
scrape_configs:
  - job_name: self
    static_configs:
      - targets: ['127.0.0.1:9090']
  - job_name: refused
    static_configs:
      - targets: ['127.0.0.1:1']
CONFIG
/bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data \
  --web.listen-address=127.0.0.1:9090 > /tmp/prom.log 2>&1 &
for attempt in $(seq 1 30); do
  if wget -qO /tmp/ready.json 'http://127.0.0.1:9090/api/v1/query?query=count%28up%29%20%3D%3D%202%20and%20sum%28up%29%20%3D%3D%201'; then
    if grep -q '"status":"success"' /tmp/ready.json && ! grep -q '"result":\[\]' /tmp/ready.json; then
      wget -qO- 'http://127.0.0.1:9090/api/v1/query?query=up'
      exit 0
    fi
  fi
  sleep 1
done
cat /tmp/prom.log >&2
exit 1
"""
name = "xbcatm-prom-live-" + uuid.uuid4().hex[:12]
try:
    result = subprocess.run(
        ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
         "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
        input=script, text=True, capture_output=True, timeout=45,
    )
finally:
    subprocess.run(["docker", "rm", "-f", name],
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if result.returncode:
    raise RuntimeError(result.stderr or result.stdout)
payload = json.loads(result.stdout)
assert payload["status"] == "success"
values = {item["metric"]["job"]: float(item["value"][1])
          for item in payload["data"]["result"]}
assert values == {"self": 1.0, "refused": 0.0}, values
print(json.dumps({"up_by_job": values, "scrape_reachability_only": True}, indent=2))

The result must contain self: 1.0 and refused: 0.0. Assertions check actual query results after both targets have been evaluated; the startup loop avoids interpreting an initially empty database as a failure. The final cleanup removes only the uniquely named test container. This verifies collection outcomes, not business availability or discovery completeness.

Turn the outcome into a useful investigation

In a real incident, identify the failing instance, read its scrape error, and test connectivity from the collector’s network context. Compare the response with the expected exporter format and check its dependency status. Keep secrets out of copied diagnostic output. Once collection recovers, confirm both up and the specific business series used by dashboards. Restoring one does not automatically establish the other.

Sources