A service registry is only as useful as its honesty. Consul keeps a catalog of registered instances and attaches health checks to them, so a caller asking where a healthy instance of a service lives gets an answer that changes as reality changes. Registration is trivial; the check is the whole product, because a registry that lists dead instances is worse than a static list — it actively routes traffic into a hole.

The freshness of that answer is a function of two choices nobody revisits after the first week: how often the check runs, and what it actually tests. A check that runs every 30 seconds is blind for 30 seconds, and a check that only proves the process exists is blind forever.

Registration, and two levels of checks

An instance registers a name, an address, a port and one or more checks, through a config file or the registration API. Checks start in a critical state by default, deliberately, so a service cannot enter the pool before its health has been verified at least once — an initial passing status is something you opt into explicitly.

Two levels of check sit on the same node. A node-level check such as serfHealth carries an empty service name and describes the machine; a service-level check such as service:web-1 is bound to one registration. They fail independently, but the healthy-only filter requires every check on the node to pass, so a critical node-level check removes all of that machine’s services at once while a critical service check removes exactly one instance.

What a caller actually receives

The catalog answers two different questions. Unfiltered, /v1/health/service/web returns every registration with each check’s status; with passing=true it returns only the instances where all checks pass. DNS and SRV always answer the second way, so a resolver never sees a name it cannot use, and the passing set is the load-balancing pool.

The catalog is also eventually consistent, and the mechanism is anti-entropy. Registration is immediate — the agent tells the leader — but repair is periodic: each agent reconciles its local state with the catalog on an interval of one minute for clusters up to 128 nodes, growing with cluster size, and each agent staggers its start so they do not all sync at once. A sync that fails is logged and retried next period, so a wrong catalog entry can survive quietly for up to one interval. The read path has its own trade: by default the agent forwards catalog reads to the leader using a leader lease that assumes leadership is briefly stable, which is fast and can occasionally return a stale result, while the consistent mode makes the leader verify with a quorum first and stale lets any server answer without checking.

When it breaks

The first failure is the interval itself. A check is only as fresh as its period, so a service that dies right after a passing probe stays in rotation until the next probe — 30 seconds for a 30-second check, 60 for a 60-second one. A TTL check is worse than a slow interval because it is passive: the instance reports in on its own schedule, so a dead instance with a 120-second TTL keeps its passing status for the full 120 seconds.

The second failure is a check that answers the wrong question. A script that runs pgrep against the process reports passing for the entire duration of a deadlock — the process exists, the port is bound, and every request routed to it hangs. The catalog cannot tell that instance from a healthy one, so the failure is silent and total for every caller in the pool. A check that exercises a dependency, even imperfectly, converges on the truth within one interval; a liveness-only check never does. The direction of the error is what makes it expensive: a check that fails when it should pass removes a healthy instance and costs capacity, while a check that passes when it should fail keeps feeding traffic to something that cannot serve it, and the caller only finds out through its own error rate.

Check it quickly

Compare the filtered and unfiltered views, and see what DNS hands out:

consul catalog services
consul health service web
curl -s 'http://127.0.0.1:8500/v1/health/service/web' | jq -r '.[] | "\(.Service.ID) \(.Checks[] | select(.Status!="passing") | .CheckID + "=" + .Status)"'
curl -s 'http://127.0.0.1:8500/v1/health/service/web?passing=true' | jq -r '.[] | "\(.Service.ID) \(.Node.Address):\(.Service.Port)"'
dig +short SRV web.service.consul

An instance that shows up unfiltered with a failing CheckID must be absent from the passing=true list and from the SRV answer; if it is still there, the check is not being evaluated at all. A node whose node-level check is critical drops out of both lists entirely, taking every service on that machine with it.

Sources

  • Service discovery, the catalog, registration and the role of health checks, checked 2026-09-16.
  • Health checks, check types, intervals and the default critical initial state, checked 2026-09-16.
  • Health HTTP API, /v1/health/service and the passing filter, checked 2026-09-16.
  • Anti-entropy, the periodic sync interval and its failure behaviour, checked 2026-09-16.