A short-lived job can finish before Prometheus next scrapes it. Pushgateway offers an intermediary that retains pushed metrics for later scraping, but it is not the default answer for every short process. Prometheus recommends it primarily for suitable service-level batch outcomes. The retained data has a lifecycle independent of the job, so successful scraping does not prove the job recently ran or succeeded.

Identify a stable batch outcome

A service-level job, such as a nightly reconciliation, can publish its last result under a stable grouping key. Putting a new run identifier, timestamp, or machine identity in that key creates additional groups instead of updating one result. Pushgateway has no automatic per-group TTL that removes obsolete runs. Owners must replace or delete the relevant data deliberately.

Machine-specific batch results can often use a node exporter’s textfile collector instead. A gateway also introduces a shared availability dependency. Its up value describes whether Prometheus can scrape the gateway, not whether each original pusher is healthy. Choose the collection model around the monitored event and ownership of cleanup, rather than treating pushing as a universal way around discovery or network policy.

Replacement is defined by the HTTP method

The URL path defines a grouping key, including the job and any additional label pairs. HTTP PUT replaces the metrics belonging to that exact group. POST replaces metric families with the names included in the request while preserving other families in the group. POST does not mean increment the previous value or append an event to a durable log.

DELETE removes the exact group identified by its URL. Deleting a job-only grouping key does not delete every group that also has that job plus additional labels. The gateway’s documented method ordering allows a later successful push to follow an earlier queued deletion. This matters when verifying cleanup instead of assuming that an accepted deletion has already affected every matching-looking series.

Scrape time is not last success time

The gateway exports push_time_seconds for the last successful push to a group. Reading the metrics endpoint does not advance that value. Prometheus assigns scrape timestamps to the samples it collects, so recently scraped samples can still describe a very old job result. Publish a separate last-success timestamp as a gauge value when the job’s successful completion time is the actual question.

Do not attach an explicit sample timestamp to pushed exposition: Pushgateway rejects those timestamps. A Unix timestamp stored as a gauge’s value is different and is supported. A successful push also establishes neither business success nor immediate durable persistence. A job can report failure successfully, and a gateway without persistence can lose held data when it restarts.

When it breaks

A failed or retired job can leave its old success metric available indefinitely. An alert checking only that the series exists will miss the missing execution. Compare the last-success value with the expected schedule, and keep job removal synchronized with group cleanup. A changing grouping label can make old and new runs appear side by side forever.

Using POST when a complete snapshot was intended can retain a metric the new run no longer publishes. Using PUT with only one metric can erase other results needed by dashboards. Review which component owns the entire group and which method it uses. There is no --replace option that substitutes for these HTTP semantics.

Check it quickly

Save this as pushgateway.py and run python3 pushgateway.py. Docker and the Pushgateway 1.11.1 image are required; an uncached image may be downloaded. The disposable server accepts requests only through container loopback. No host port is published, no host directory is mounted, and cleanup removes only the uniquely named test container.

import http.client
import io
import json
import subprocess
import time
import uuid

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
    def __init__(self, data):
        self.data = data

    def makefile(self, mode):
        return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
    body = body.encode()
    headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
               f"Connection: close\r\nContent-Type: {content_type}\r\n"
               f"Content-Length: {len(body)}\r\n\r\n").encode()
    result = subprocess.run(
        ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
        input=headers + body, capture_output=True, timeout=5,
    )
    if result.returncode:
        raise RuntimeError(result.stderr.decode())
    response = http.client.HTTPResponse(ResponseSocket(result.stdout))
    response.begin()
    return response.status, response.read().decode()


def get_json(path):
    status, body = request("GET", path)
    assert status == 200, (status, body)
    return json.loads(body)


def eventually(check):
    for attempt in range(150):
        try:
            value = check()
            if value:
                return value
        except (RuntimeError, AssertionError, http.client.HTTPException):
            pass
        time.sleep(0.1)
    raise AssertionError("Expected local server state did not appear")


port = 9091
script = r"""exec /bin/pushgateway --web.listen-address=127.0.0.1:9091
"""
try:
    subprocess.run(
        ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
         "--entrypoint", "/bin/sh", "prom/pushgateway:v1.11.1", "-c", script],
        check=True, capture_output=True, text=True, timeout=20,
    )
    eventually(lambda: request("GET", "/metrics")[0] == 200)
    group = "/metrics/job/nightly_sync/team/data"
    first = "# TYPE items_processed gauge\nitems_processed 7\n# TYPE last_success_unixtime gauge\nlast_success_unixtime 1234\n"
    assert request("PUT", group, first, "text/plain; version=0.0.4")[0] == 200
    def snapshot():
        status, body = request("GET", "/metrics")
        assert status == 200
        return {line.split("{", 1)[0]: float(line.rsplit(" ", 1)[1])
                for line in body.splitlines()
                if not line.startswith("#") and 'job="nightly_sync"' in line}
    def first_snapshot():
        data = snapshot()
        return data if data.get("items_processed") == 7 else None
    initial = eventually(first_snapshot)
    assert initial["last_success_unixtime"] == 1234
    time.sleep(0.2)
    assert snapshot()["push_time_seconds"] == initial["push_time_seconds"]
    partial = "# TYPE items_processed gauge\nitems_processed 9\n"
    assert request("POST", group, partial, "text/plain; version=0.0.4")[0] == 200
    after_post = snapshot()
    assert after_post["items_processed"] == 9 and after_post["last_success_unixtime"] == 1234
    replacement = "# TYPE items_processed gauge\nitems_processed 11\n"
    assert request("PUT", group, replacement, "text/plain; version=0.0.4")[0] == 200
    after_put = snapshot()
    assert after_put["items_processed"] == 11 and "last_success_unixtime" not in after_put
    assert request("DELETE", "/metrics/job/nightly_sync")[0] == 202
    # A successful later POST orders the preceding queued deletion before this check.
    assert request("POST", group, "", "text/plain; version=0.0.4")[0] == 200
    assert snapshot()["items_processed"] == 11
    assert request("DELETE", group)[0] == 202
    eventually(lambda: not snapshot())
    print(json.dumps({"initial_items": 7, "post_items": 9, "put_items": 11,
                      "post_preserved_other_metric": True, "put_removed_other_metric": True,
                      "scrapes_did_not_refresh_push_time": True,
                      "parent_group_delete_did_not_match": True, "group_deleted": True}, indent=2))
finally:
    subprocess.run(["docker", "rm", "-f", name],
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

Assertions show seven processed items after the initial PUT, nine after POST while the separate last-success gauge remains, and eleven after a replacement PUT that removes that other gauge. Repeated reads leave push time unchanged. Deleting the job-only group leaves the team-qualified group intact; deleting the exact group removes its metrics. The fixture tests gateway API behavior, not job scheduling, Prometheus integration or durable recovery.

Sources