A shell pipeline connects one command’s standard output to the next command’s standard input. The commands can run concurrently, so the producer need not finish before the consumer starts. This supports incremental processing, but the pipe itself does not guarantee that either program processes data incrementally. Buffering, early termination, and the status of each stage determine whether the complete operation does what you intended.

A byte stream with finite capacity

A pipe transports bytes rather than records. A reader is not promised one read per line or one read per producer write. Programs must implement any framing they need. Standard error normally remains separate, which is why an upstream diagnostic can appear on the terminal without entering the downstream filter. Decide explicitly whether a consumer should receive diagnostics as data.

The kernel provides finite buffering. A blocking writer waits when the pipe fills, and a reader waits when the pipe is empty while writers remain open. This backpressure limits how far a producer can advance through that particular pipe. Capacity depends on the operating system and configuration; it is not universally sixty-four kibibytes. Applications can still accumulate arbitrarily large internal data structures, so a pipeline alone does not bound total memory.

User-space buffering adds another delay. A program that flushes every line to a terminal may buffer larger chunks when stdout is a pipe. An apparently idle consumer may therefore be waiting for data still buffered in the producer. Check the producer’s documented streaming and flush controls before changing kernel limits or adding arbitrary sleeps. Some transformations, such as a complete sort, inherently need more input before producing useful output.

When it breaks

A reader sees end-of-file only after every descriptor referring to the write end is closed. A child that inherited an unused write descriptor can keep a consumer waiting even after the apparent producer exited. In the other direction, writing after all read ends are closed generates SIGPIPE, or yields an EPIPE error when the signal is ignored. A consumer that deliberately stops early can therefore cause an otherwise healthy producer to terminate unsuccessfully.

By default, Bash returns the last stage’s status. An upstream failure followed by a successful cat can look successful overall. With pipefail, the aggregate result is the rightmost nonzero stage status, not the first failure in time or the leftmost failing command. The PIPESTATUS array retains individual stage results briefly. Capture it immediately: even an assignment or printf can replace the observation.

Most Bash pipeline stages run in subshell environments. An assignment in a loop on the right side of a pipe normally does not update the caller’s variable. Bash’s lastpipe option can alter the final stage’s execution when job control is inactive, so state assumptions need an explicit shell configuration. A redirected loop can avoid a pipeline when the caller must retain the loop’s state.

Finally, a foreground Bash pipeline waits for its stages to terminate. It does not return merely because the consumer read its first line. A producer that catches a broken pipe and continues unrelated work can delay completion. Treat a pipeline as several processes with connected descriptors and separate outcomes, rather than one command that necessarily succeeds or stops as a unit.

Check it quickly

Run the following Bash block with cat, yes, and head available. It compares default and pipefail statuses using a producer that deliberately exits seven. It then demonstrates two failing stages and an early-closing consumer. Errexit is disabled only while collecting expected nonzero results, then restored before assertions.

bash --noprofile --norc <<'BASH'
set -e
set +o pipefail
set +e
(printf 'partial\n'; exit 7) | cat >/dev/null
plain=("$?" "${PIPESTATUS[@]}")
set -e
[[ ${plain[*]} == '0 7 0' ]]
set -o pipefail
set +e
(printf 'partial\n'; exit 7) | cat >/dev/null
strict=("$?" "${PIPESTATUS[@]}")
set -e
[[ ${strict[*]} == '7 7 0' ]]
set +e
(exit 7) | (exit 4)
multiple=("$?" "${PIPESTATUS[@]}")
set -e
[[ ${multiple[*]} == '4 7 4' ]]
set +e
yes | head -n 1 >/dev/null
early=("$?" "${PIPESTATUS[@]}")
set -e
[[ ${early[0]} -ne 0 && ${early[1]} -ne 0 && ${early[2]} -eq 0 ]]
printf 'default: %s\npipefail: %s\nrightmost: %s\nearly close: %s\n' \
  "${plain[*]}" "${strict[*]}" "${multiple[*]}" "${early[*]}"
printf 'PASS: aggregate and individual stage outcomes\n'
BASH

Each reported array starts with the aggregate status, followed by the stages in source order. Expect 0 7 0, 7 7 0, and 4 7 4 for the first three. The final consumer succeeds while yes returns nonzero; the test deliberately does not require a platform-specific signal number. A production pipeline must distinguish this intentional early closure from lost required output. Measure throughput and memory separately; these status assertions do not measure buffering performance.

Sources