Pipelining reduces the waits between sending Redis commands, but its benefit must be measured for a particular workload. A remote synchronous loop may mostly wait for network latency; a local loop may spend more time in the client, scheduling, and socket I/O. Neither case justifies a universal promise of several-fold improvement.

Compare equivalent work

For N commands and a batch size B, a simple synchronous model changes approximately N request-response waits into ceiling(N/B) batch waits. It does not remove command execution, response bytes, encoding, or parsing. One batch can span multiple socket writes and network packets, so this estimate is not a packet count or a measured network RTT.

Keep the command, key distribution, payload, connection count, client version, and persistence settings fixed while changing B. Otherwise a faster result might come from doing less work, removing durability, or adding concurrent clients. Use transaction=False when the question is plain pipelining; redis-py’s default pipeline also includes MULTI/EXEC.

Batch size one below uses a synchronous client loop, not a one-command pipeline. The comparison measures this application’s batching benefit, not Redis’s maximum throughput or server cost alone.

Include the costs the application pays

Time from command preparation through receipt of the final response. Ignoring request construction or response parsing makes large batches look cheaper than they are to the application. The example includes buffering, execution, and result collection in its timer, while resetting the fixture and asserting correctness outside it.

Warm each case, vary test order, repeat, and report variation. Three measured runs here provide a small reproducible comparison, not a rigorous capacity study. A laptop’s scheduler and competing work can still influence the outcome. Do not promote one run’s median into a service-level objective.

Throughput is not individual-request latency. Dividing total time by the number of commands gives an average completion cost, not a p95 or p99 latency distribution. Waiting to fill a batch can delay its first request before execution even begins. This experiment constructs batches immediately and therefore does not measure arrival-time queueing.

When it breaks

A huge batch can improve commands per second while causing unacceptable pauses and memory growth. Bound the number of commands, request and response bytes, and formation delay. Choose those limits from representative measurements rather than automatically using hundreds or thousands of entries for every payload.

If larger batches stop helping, inspect client CPU, response decoding, server command cost, bandwidth, and competing workload. Pipelining can improve achieved server throughput by amortizing I/O, but cannot remove every bottleneck. Sharding is one possible architecture decision, not the inevitable explanation for a flat graph.

Speed measurements must still check failures. Dropping responses or silently accepting errors can produce an impressive but meaningless rate. Network retries are disabled below, and every increment response is checked against the expected sequence. A real disconnect can leave execution uncertain; replaying a batch after the last known response is not automatically safe for increments.

Keep this workload on its private instance. It repeatedly resets and increments a counter, so changing the endpoint to a shared production database would change the operational risk. The local Unix socket also deliberately excludes TCP, TLS, and a remote network path.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. The setup compiles Redis core with libc and installs the client in a temporary virtual environment, not a system service.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It starts a private Unix-socket Redis under /tmp with TCP and persistence disabled, then stops only its own process and cleans its runtime directory. Downloads require network access. Existing Redis instances are not contacted.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
    with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
        endpoint = str(Path(directory) / "redis.sock")
        process = subprocess.Popen([
            os.environ["REDIS_SERVER"], "--port", "0",
            "--unixsocket", endpoint, "--unixsocketperm", "700",
            "--save", "", "--appendonly", "no", "--dir", directory,
            "--maxmemory-policy", "noeviction",
        ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
        client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
                             socket_timeout=2, retry=Retry(NoBackoff(), 0))
        try:
            deadline = time.monotonic() + 5
            while True:
                assert process.poll() is None, "Redis exited during startup"
                try:
                    if client.ping():
                        break
                except redis.ConnectionError:
                    if time.monotonic() >= deadline:
                        raise
                    time.sleep(0.02)
            yield client
        finally:
            client.close()
            if process.poll() is None:
                process.terminate()
            try:
                _, errors = process.communicate(timeout=5)
            except subprocess.TimeoutExpired:
                process.kill()
                process.communicate()
                raise
            assert process.returncode == 0, errors.decode()


from statistics import median


TOTAL = 2048
SIZES = (1, 8, 64, 256)


def measure(client, size):
    client.set("bench:counter", 0)
    started = time.perf_counter()
    responses = []
    if size == 1:
        for _ in range(TOTAL):
            responses.append(client.incr("bench:counter"))
    else:
        for offset in range(0, TOTAL, size):
            with client.pipeline(transaction=False) as batch:
                for _ in range(min(size, TOTAL - offset)):
                    batch.incr("bench:counter")
                responses.extend(batch.execute())
    elapsed = time.perf_counter() - started
    assert responses == list(range(1, TOTAL + 1))
    assert client.get("bench:counter") == str(TOTAL).encode()
    return elapsed


with instance() as client:
    for size in SIZES:
        measure(client, size)
    measurements = {size: [] for size in SIZES}
    for repetition in range(3):
        order = SIZES[repetition:] + SIZES[:repetition]
        for size in order:
            measurements[size].append(measure(client, size))
    for size in SIZES:
        samples = measurements[size]
        duration = median(samples)
        waits = (TOTAL + size - 1) // size
        print(f"batch={size:3d} waits={waits:4d} median_ms={duration * 1000:.2f} "
              f"commands_per_second={TOTAL / duration:.0f} "
              f"range_ms={min(samples) * 1000:.2f}-{max(samples) * 1000:.2f}")
    print("all replies and final counters checked; no speedup threshold is asserted")

Each case performs 2,048 increments of the same counter, reusing a connection. The logical blocking-call counts are 2,048, 256, 32, and eight for sizes one, eight, 64, and 256. Those are application waits, not captured packets. Every trial checks the complete reply sequence and the final stored counter.

One recorded run produced median completion times of 35.24, 9.40, 5.61, and 5.00 milliseconds respectively. The shrinking improvement at larger sizes illustrates diminishing returns for this local workload. The program prints your measurements and their observed ranges; it deliberately has no assertion requiring a speedup.

All four configurations run once as warmup, followed by three measured repetitions with rotated order. Commands per second divides 2,048 by the measured duration. It must not be read as a prediction for a remote application with different concurrency or payloads.

This exercise validates equivalent successful work and measures local batch completion. It does not test production arrival rates, tail latency, TCP/TLS transport, persistence overhead, connection failures, or cluster scaling. Those are separate dimensions of a deployment benchmark.

Sources