Postgres connections are expensive, so pool and reuse

Mon, 09 Aug 2027 00:00:00 +0000

A warm pool reduces repeated PostgreSQL connection setup, but an idle connection can disappear while it sits in that pool. Server termination, network failure, or an intermediary’s idle policy can leave a client holding a socket that is no longer usable. Reuse must include failure detection and replacement; keeping an object in a pool is not evidence that its next query will succeed.

When it breaks

TCP keepalives detect some failures on otherwise idle connections. They do not set a deadline for SQL execution, prove the database is ready to process a particular request, or roll back an uncertain operation. Libpq’s client keepalive options apply to TCP and are ignored for Unix-domain sockets; server-side keepalive settings govern the other endpoint. Operating-system support also affects which options have an effect.

A checkout health check can reject a stale connection before handing it to the caller. Psycopg does not perform that check by default; it offers a check callback for applications that need one. The extra probe costs a round trip and still cannot prevent failure immediately afterward. Connection establishment, waiting for a lease, and executing a statement therefore need distinct budgets and error handling.

Check it quickly

Use Python 3.10 or later, PostgreSQL 18 over TCP with the default zero statement timeout, and the tested packages installed with python3 -m pip install "psycopg[binary]==3.3.5" "psycopg-pool==3.3.1". Set PG_DSN to a disposable database and a role allowed to terminate its own backend connections. Save the program as liveness-demo.py. It terminates only the idle connection each test pool just created, after recording that connection’s backend identifier.

import os

import psycopg
from psycopg.errors import QueryCanceled
from psycopg_pool import ConnectionPool

dsn = os.environ['PG_DSN']
with psycopg.connect(dsn, autocommit=True) as admin:
 for checked in (False, True):
 with ConnectionPool(
 dsn, min_size=1, max_size=1, timeout=3, open=True,
 check=ConnectionPool.check_connection if checked else None,
 kwargs={
 'connect_timeout': 2,
 'keepalives': 1,
 'keepalives_idle': 30,
 'keepalives_interval': 10,
 'keepalives_count': 3,
 },
 ) as pool:
 pool.wait(timeout=3)
 with pool.connection() as conn:
 assert conn.execute('SELECT inet_server_addr() IS NOT NULL').fetchone()[0]
 original_pid = conn.info.backend_pid
 parameters = conn.info.get_parameters()
 assert [parameters[k] for k in (
 'keepalives', 'keepalives_idle', 'keepalives_interval', 'keepalives_count'
 )] == ['1', '30', '10', '3']
 # Terminate only the idle connection just created by this fixture.
 assert admin.execute('SELECT pg_terminate_backend(%s,2000)', (original_pid,)).fetchone()[0]

 if not checked:
 try:
 with pool.connection() as conn:
 conn.execute('SELECT 1')
 except psycopg.OperationalError as error:
 print('unchecked_first_use', error.sqlstate)
 else:
 raise AssertionError('terminated connection unexpectedly worked')

 with pool.connection() as conn:
 replacement_pid = conn.info.backend_pid
 assert replacement_pid != original_pid
 assert conn.execute('SELECT 1').fetchone()[0] == 1
 print('checked', checked, 'replacement', True)

 try:
 with pool.connection() as conn:
 conn.execute("SET LOCAL statement_timeout='50ms'")
 conn.execute('SELECT pg_sleep(0.2)')
 except QueryCanceled as error:
 assert error.sqlstate == '57014'
 print('statement_timeout', error.sqlstate)
 else:
 raise AssertionError('long statement did not time out')

 with pool.connection() as conn:
 assert conn.info.backend_pid == replacement_pid
 assert conn.execute('SHOW statement_timeout').fetchone()[0] == '0'
 assert conn.execute('SELECT 1').fetchone()[0] == 1
 print('healthy_after_rollback', True)

Both cases configure client keepalives and verify the effective parameters without printing the connection string. The first case has no checkout check. After its idle backend is terminated, the next read encounters an operational error; this tested server shutdown reports 57P01. The broken connection is then discarded, and a subsequent checkout obtains a different backend that successfully answers one.

The second case enables ConnectionPool.check_connection. After the same controlled termination, the check detects the unusable connection during acquisition, and the borrower receives a working replacement with a different identifier. The output reports successful replacement in both cases, but only the unchecked case exposes its stale first use to the application. This exercises explicit server termination, not a silent network partition or the keepalive probe timer.

Each case then runs a two-hundred-millisecond sleep under a fifty-millisecond local statement timeout. Expect 57014. The connection is alive, TCP keepalives are enabled, and the statement still needs its own deadline. The exception leaves the connection context, which rolls back the failed transaction. A later checkout uses the same replacement backend, sees the original zero statement timeout, and successfully reads one.

These checks distinguish a dead connection needing replacement from a live connection recovering after a canceled statement. They also verify that the transaction-local timeout did not leak to another borrower. Do not treat every SQL error as proof that a socket must be destroyed, or every successful health probe as permission to remove statement deadlines.

Reconnect at a known operation boundary

Pool maximum idle time and maximum lifetime help manage connection turnover, but neither guarantees a socket survives until its configured age. A server or intermediary can close it sooner. Coordinate policies, inspect discard and reconnect rates, and measure acquisition latency under failures. An aggressive health-check policy can also add material overhead to short queries.

The example replaces an idle connection before application writes begin. A disconnect during commit is different: the transaction may already have committed even if confirmation was lost. Reconnecting does not determine that outcome, and automatically repeating the write can duplicate effects. Preserve operation identifiers and reconcile durable results before replaying non-idempotent work. Use caller cancellation and an overall request deadline to stop repeated connection attempts, rather than letting reconnection activity outlive the request indefinitely.

Sources

PostgreSQL 18 libpq connection parameters, client keepalives, socket applicability, and connection options, checked 2026-09-10.
Psycopg pooling documentation source, checkout checks and broken-connection replacement, checked 2026-09-10.
Psycopg pool API documentation source, optional check callback and connection lifetime controls, checked 2026-09-10.
PostgreSQL 18 client connection defaults, statement timeouts distinct from connection liveness, checked 2026-09-10.

Connection · XB Field Notes

Postgres connections are expensive, so pool and reuse

When it breaks

Check it quickly

Reconnect at a known operation boundary

Sources