A bloom filter answers maybe at low cost

Wed, 15 Dec 2027 00:00:00 +0000

A Bloom filter can cheaply rule out membership in the set it represents, while a positive answer means only “possibly present.” That asymmetry depends on correct updates and unchanged hash rules. A stale or corrupted filter can reject a key that exists in the underlying database, even though the ordinary append-only algorithm has no false negatives for inserted items.

Bits represent candidates, not records

Start with a zeroed array of m bits and k hash-derived positions per key. Insertion sets all those positions. Lookup returns negative if any required bit is zero; otherwise it returns maybe. Different keys can share positions, so a positive answer is not proof that the queried key was inserted.

Inserting an existing key again leaves its bits unchanged. The filter does not retain the original keys or values, and cannot enumerate them. An exact set or database remains authoritative when a false positive would affect correctness. Do not use “maybe” alone to grant access, charge a customer, or discard required work.

Size for an expected population

Under the usual approximately uniform, independent-hashing model, n inserted distinct items give an approximate false-positive probability of p = (1 − exp(−kn/m))^k. For fixed m and n, k near (m/n) ln(2) minimizes that approximation. More hash operations are not always better: they also fill bits faster.

With an optimal hash count, a one-percent target needs roughly 9.6 bits per item, before implementation overhead. Ten million items therefore need about twelve million bytes of bitmap, not merely a few million bits. State whether a size is bits, bytes, payload alone, or total process memory.

The false-positive rate concerns queries for absent items. It is not the fraction of all positive answers that are wrong; that fraction also depends on how often callers ask about present items. Measure against known nonmembers, and do not interpret a finite sample as an exact probability guarantee.

Keep the represented set synchronized

For a negative answer to justify skipping the database, the filter must include all relevant database insertions. Publishing a new row before updating the filter creates a visibility gap. Rebuilds likewise need a defined snapshot and update handoff. A successful filter lookup does not establish that those application-level steps were atomic.

Keep the bit count, hash construction, seeds or domain prefixes, and byte encoding consistent across writers and readers. Changing those inputs changes positions. Ordinary filters cannot safely delete by clearing an item’s bits, because other items may depend on the same bits. Counting or other deletion-capable structures have different contracts.

When it breaks

Overfilling a fixed bitmap raises false positives and can erase the benefit of avoiding expensive lookups. Negatives remain sound for correctly inserted items, but fewer queries return negative. Enlarging a fixed bitmap requires rebuilding or a suitable growth design; merely changing its modulus makes existing bits meaningless.

Product behavior is not identical to this fixed-size model. Redis BF.RESERVE can create additional subfilters when scaling is enabled, while its nonscaling mode rejects insertion beyond capacity. Subfilters cost memory and lookup work. Consult the configured implementation instead of assuming it silently saturates like an unbounded insertion loop.

Lost updates, inconsistent encoding, stale snapshots, and unsafe deletion can violate the application’s expected negative-answer safety. Track insertion failures and filter freshness as well as observed false positives. The filter is an optimization whose maintenance belongs in the data lifecycle.

Check it quickly

Save the block as bloom_check.py and run python3 bloom_check.py with Python 3.10 or newer. It ran with Python 3.14.4 and uses only the standard library. Seven domain-prefixed SHA-256 digests, explicit UTF-8, and fixed key sequences make the test repeatable. No external service, network access, or file writing is required.

from hashlib import sha256
from math import exp


M, K, N = 10000, 7, 1000


def offsets(value):
 data = value.encode("utf-8")
 return tuple(int.from_bytes(sha256(index.to_bytes(2, "big") + data).digest(), "big") % M
 for index in range(K))


def add(bits, value):
 for position in offsets(value):
 bits[position // 8] |= 1 << (position % 8)


def maybe(bits, value):
 return all(bits[position // 8] & (1 << (position % 8)) for position in offsets(value))


bits = bytearray((M + 7) // 8)
inserted = [f"item:{index}" for index in range(N)]
truth = set(inserted)
for value in inserted:
 add(bits, value)
assert all(maybe(bits, value) for value in inserted)
before = bytes(bits)
add(bits, inserted[0])
assert bytes(bits) == before

absent = [f"absent:{index}" for index in range(10000)]
assert truth.isdisjoint(absent)
false_positives = [value for value in absent if maybe(bits, value)]
assert 0 < len(false_positives) < len(absent)
estimate = (1 - exp(-K * N / M)) ** K
print(f"{N} inserted; bitmap={len(bits)} bytes; estimate={estimate:.4%}")
print(f"absent probes: {len(false_positives)}/{len(absent)} false positives")
assert false_positives[0] not in truth
print("a maybe still needs exact lookup:", false_positives[0])

victim = inserted[0]
victim_offsets = set(offsets(victim))
retained = next(value for value in inserted[1:] if victim_offsets.intersection(offsets(value)))
broken = bytearray(bits)
for position in victim_offsets:
 broken[position // 8] &= ~(1 << (position % 8))
assert retained in truth and maybe(bits, retained) and not maybe(broken, retained)
print("clearing bits for", victim, "incorrectly rejects retained", retained)

new_value = next(f"new:{index}" for index in range(1000) if not maybe(bits, f"new:{index}"))
truth.add(new_value)
assert new_value in truth and not maybe(bits, new_value)
add(bits, new_value)
assert maybe(bits, new_value)
print("an origin insertion is invisible until the filter is also updated")

# Use a separate fixed bitmap so the capacity experiment has exactly 5000 items.
overfull = bytearray((M + 7) // 8)
for index in range(5000):
 add(overfull, f"item:{index}")
overfull_positives = sum(maybe(overfull, value) for value in absent)
assert overfull_positives > len(false_positives)
assert all(maybe(overfull, f"item:{index}") for index in range(5000))
print(f"5000 inserted in the same bit budget: {overfull_positives}/{len(absent)} false positives")

The 10,000-bit array occupies 1,250 payload bytes. After 1,000 insertions, all inserted items test positive, repeated insertion changes nothing, and 73 of 10,000 known-absent probes return maybe. The approximation predicts about 0.8194 percent; this finite fixture observes 0.73 percent.

Clearing item:0’s bits incorrectly rejects retained item:79. A new origin item is also invisible until its filter update. A separate bitmap filled with 5,000 items returns 8,016 false positives on the same absent probes, while still recognizing every inserted item.

These assertions distinguish algorithmic false positives from maintenance-induced false negatives. They are not a production memory benchmark, concurrency test, serialization-compatibility test, or execution of Redis Bloom commands.

Sources

Redis Bloom filters, probabilistic membership and authoritative follow-up lookups, checked 2026-09-08.
Redis BF.RESERVE, sizing formulas, capacity, and scaling behavior, checked 2026-09-08.
Python hashlib, deterministic SHA-256 digest construction, checked 2026-09-08.

Datastructure · XB Field Notes