Data · XB Field Notes

A Redis hash stores many fields in one key

Thu, 24 Feb 2028 00:00:00 +0000

A Redis hash is a field-addressable record under one key, not a schema-checked object or an automatically nested document. It is useful when callers need a few attributes without replacing the whole record. That advantage depends on access patterns, field sizes, and the meaning of replies, not merely on choosing the hash type.

Read the fields your caller actually needs

HGET retrieves one field. HMGET retrieves selected fields in the requested order, while HGETALL returns the entire mapping. With redis-py’s binary responses, an absent field becomes None and an existing empty value becomes empty bytes. A truthiness check incorrectly treats those two states as equivalent. An absent key also produces missing values through HMGET.

Field names and values are bytes; a name containing a dot is still a literal name. Redis does not interpret name.first as traversal through a nested object. Likewise, a string containing JSON remains a Redis string: HGET does not parse it. Decide serialization and validation in the application and preserve the distinction between an omitted attribute and an intentionally empty one.

Updates have precise return values

HSET changes the specified fields and leaves others untouched. A multi-field call is one command, but its integer reply counts newly added fields, not successful replacements. Updating an existing name can correctly return zero. Treating that zero as failure can cause needless retries or misleading error reports.

HINCRBY changes an integer field in one command and starts from zero when the field is absent. It rejects noninteger values and overflow beyond signed 64-bit range. This avoids a read-modify-write race for that increment; it does not make surrounding application checks atomic. Separate HGET, local arithmetic, and HSET can still overwrite another client’s update.

HDEL counts fields actually removed, and deleting the final field removes the key. Do not use an empty hash as a persistent placeholder for an entity. Whole-key expiration also removes its fields. Current Redis additionally supports field deadlines; the exercise below assigns no expirations and does not test lifetime behavior.

Compact representation is an implementation choice

Small hashes can use listpack encoding instead of a hash table. Redis 8.10.1’s standard thresholds are 512 entries and 64 bytes per field name or value; deployed settings can differ. Crossing an applicable limit changes the representation, not the public hash operations or the stored values.

Compact storage trades internal scanning work for reduced overhead and locality. Raising thresholds substantially requires measurement: a command’s advertised complexity is not a promise that arbitrarily large fields or changed configuration have no cost. Inspect OBJECT ENCODING when diagnosing representation, but do not make application correctness depend on that internal label or an assumed memory-saving ratio.

When it breaks

HGETALL grows with the whole hash and transfers both names and values. Fetching thousands of fields when the caller needs two wastes work, even if they share one key. HMGET’s work grows with the requested field count; it is not free either. Never use returned map iteration order as an application ordering contract.

WRONGTYPE indicates that the key holds another Redis type, not that a field is missing. Noninteger counters require an explicit data decision, not a retry loop. One huge hash also remains one key rather than automatically spreading its fields across cluster nodes. Choose record boundaries around actual access and ownership rather than packing unrelated data together.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Redis 8.10.1, redis-py 8.1.0, and Python 3.14.4. The setup builds only Redis core with libc and installs the client in a temporary virtual environment.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It starts a private Unix-socket server with TCP and persistence disabled. Only this server receives deliberately small thresholds: four entries and eight bytes. The test closes its client, terminates its own process, and removes the temporary runtime directory; it never connects to a default local Redis.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-hash-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--hash-max-listpack-entries", "4", "--hash-max-listpack-value", "8",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 assert client.hset("profile", mapping={"name": "Ada", "visits": "0", "empty": b""}) == 3
 assert client.hset("profile", "name", "Ada L") == 0
 assert client.hmget("profile", ["visits", "name", "missing", "empty"]) == [
 b"0", b"Ada L", None, b"",
 ]
 assert client.hmget("absent", ["a", "b"]) == [None, None]
 assert client.hgetall("profile") == {b"name": b"Ada L", b"visits": b"0", b"empty": b""}
 assert client.hincrby("profile", "visits", 1) == 1
 assert client.hget("profile", "name") == b"Ada L"
 print("HSET counts new fields; HMGET preserves order and distinguishes missing from empty")

 client.set("json", '{"name":"Ada"}')
 try:
 client.hget("json", "name")
 except redis.ResponseError as error:
 assert "WRONGTYPE" in str(error)
 else:
 raise AssertionError("a JSON string is not a hash")
 client.hset("literal", "name.first", b"Ada")
 assert client.hget("literal", "name") is None
 assert client.hget("literal", "name.first") == b"Ada"
 client.hset("invalid", "counter", "3.5")
 try:
 client.hincrby("invalid", "counter", 1)
 except redis.ResponseError as error:
 assert "integer" in str(error)
 else:
 raise AssertionError("an integer counter cannot contain 3.5")
 assert client.hget("invalid", "counter") == b"3.5"
 client.hset("overflow", "counter", 2**63 - 1)
 try:
 client.hincrby("overflow", "counter", 1)
 except redis.ResponseError as error:
 assert "overflow" in str(error)
 else:
 raise AssertionError("signed 64-bit integer overflow must fail")
 assert client.hget("overflow", "counter") == str(2**63 - 1).encode()
 print("wrong types and invalid or overflowing counters fail; dotted fields are literal")

 assert client.object("ENCODING", "profile") == b"listpack"
 assert client.hset("profile", "fourth", "ok") == 1
 assert client.object("ENCODING", "profile") == b"listpack"
 assert client.hset("profile", "fifth", "ok") == 1
 assert client.object("ENCODING", "profile") == b"hashtable"
 client.hset("value-limit", "field", b"abcdefgh")
 assert client.object("ENCODING", "value-limit") == b"listpack"
 client.hset("value-limit", "field", b"abcdefghi")
 assert client.object("ENCODING", "value-limit") == b"hashtable"
 assert client.hget("profile", "visits") == b"1"
 assert client.hget("value-limit", "field") == b"abcdefghi"
 print("private 4-field and 8-byte thresholds each trigger listpack to hashtable conversion")

 assert client.hdel("value-limit", "absent") == 0
 assert client.hdel("value-limit", "field") == 1
 assert client.exists("value-limit") == 0
 print("deleting the last field removes the hash key")

The assertions distinguish added fields from replacements, preserve HMGET ordering and missing-versus-empty values, and verify unchanged neighboring fields. A JSON string raises WRONGTYPE, a dotted field stays literal, and invalid or overflowing counters remain unchanged after rejection.

At four short fields the profile remains listpack; the fifth switches it to hashtable. A separate hash switches when its value grows from eight to nine bytes. Reads still return the expected values, and deleting the final field removes that key. These results demonstrate configured encoding transitions and command semantics, not production memory savings, latency, concurrent-load behavior, or cluster distribution.

Sources

Redis hashes, field-oriented operations and current expiration capabilities, checked 2026-09-08.
Redis HSET, partial updates and added-field counts, checked 2026-09-08.
Redis HMGET, requested ordering and missing-field replies, checked 2026-09-08.
Redis HGETALL, full-hash reads and complexity, checked 2026-09-08.
Redis HINCRBY, integer increments and supported range, checked 2026-09-08.
Redis HDEL, removal counts and final-field deletion, checked 2026-09-08.
Redis OBJECT ENCODING, internal representations and conversion, checked 2026-09-08.
Redis memory optimization, compact encoding thresholds and measurement guidance, checked 2026-09-08.
Redis 8.10.1 configuration, default hash thresholds, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only build and allocator selection, checked 2026-09-08.

UTF-8 preserves text only when byte boundaries and decoding rules agree

Wed, 02 Dec 2026 00:00:00 +0000

UTF-8 encodes Unicode scalar values using one to four bytes. ASCII characters retain their familiar one-byte representation, while many other characters need longer sequences. This makes UTF-8 a useful interchange format, but it does not make arbitrary bytes valid text or every byte offset a character boundary. Correct handling requires an agreed encoding, strict validity rules where appropriate, and a clear distinction between bytes and decoded text.

When it breaks

A file or network read can end halfway through a multibyte sequence. Decoding each chunk independently may reject a perfectly valid complete stream. Replacing errors separately in every chunk is worse when it destroys bytes that would have formed a character with the next read. A streaming decoder needs to retain incomplete input and validate it when more bytes arrive or the stream ends.

UTF-8’s recognizable leading and continuation-byte patterns do not make arbitrary slicing safe. A program that truncates an encoded byte string can cut a sequence in half. Nor does successful decoding under another codec prove that the chosen interpretation is correct. Latin-1 maps every byte, so wrongly decoding UTF-8 as Latin-1 can produce valid Unicode text whose content is nevertheless mojibake.

Encoding validity is separate from text equivalence and display width. A single visible accented letter can use one code point or a base letter plus a combining mark. Normalization can make selected forms comparable, but decoding UTF-8 does not automatically normalize them. Similarly, a byte-order mark may become a leading text character unless the selected codec intentionally consumes it.

Check it quickly

Save this as check-utf8.py and run python3 check-utf8.py with Docker. It uses standard-library codecs and Unicode operations in a pinned Python 3.11.16 container without network access. All examples are fixed in-memory bytes and strings. The script performs no database migration or file conversion and does not depend on the terminal correctly displaying an emoji:

import json,subprocess,uuid
image="python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84"
program=r"""
import codecs,json,unicodedata

text='A中🙂';encoded=text.encode('utf-8');assert len(text)==3 and len(encoded)==8
assert encoded.hex()=='41e4b8adf09f9982' and encoded.decode('utf-8')==text
pieces=[encoded[:2],encoded[2:5],encoded[5:]]
split_errors=0
for piece in pieces:
 try:piece.decode('utf-8')
 except UnicodeDecodeError:split_errors+=1
assert split_errors==3
decoder=codecs.getincrementaldecoder('utf-8')(errors='strict');decoded=[]
for piece in pieces:decoded.append(decoder.decode(piece,final=False))
decoded.append(decoder.decode(b'',final=True));assert ''.join(decoded)==text
rejected={}
for label,raw in {'overlong':bytes.fromhex('c0af'),'surrogate':bytes.fromhex('eda080'),
 'too_large':bytes.fromhex('f4908080'),'truncated':bytes.fromhex('e4b8')}.items():
 try:raw.decode('utf-8');raise AssertionError('invalid UTF-8 accepted')
 except UnicodeDecodeError as error:rejected[label]=error.reason
bad_tail=codecs.getincrementaldecoder('utf-8')();assert bad_tail.decode(bytes.fromhex('e4b8'),final=False)==''
try:bad_tail.decode(b'',final=True);raise AssertionError('incomplete final sequence accepted')
except UnicodeDecodeError:final_rejected=True
mojibake=encoded.decode('latin-1');assert mojibake!=text and mojibake.encode('latin-1')==encoded
decomposed='e\u0301';composed='é';assert decomposed!=composed and unicodedata.normalize('NFC',decomposed)==composed
bom=b'\xef\xbb\xbf'+encoded;assert bom.decode('utf-8').startswith('\ufeff') and bom.decode('utf-8-sig')==text
replacement=bytes.fromhex('ff41').decode('utf-8',errors='replace');assert replacement=='\ufffdA'
print(json.dumps({'text':text,'code_points':len(text),'bytes':len(encoded),'hex':encoded.hex(),
 'chunk_hex':[x.hex() for x in pieces],'independent_chunk_failures':split_errors,'incremental_output':decoded,
 'invalid_rejections':rejected,'final_incomplete_rejected':final_rejected,'latin1_code_points':[hex(ord(c)) for c in mojibake],
 'normalization_lengths':[len(decomposed),len(composed)],'bom_utf8_first':hex(ord(bom.decode('utf-8')[0])),
 'utf8_sig_strips_bom':bom.decode('utf-8-sig')==text,'replacement':replacement},ensure_ascii=False))
"""
name='xbcatm-socket-'+uuid.uuid4().hex[:10]
try:
 result=subprocess.run(['docker','run','--rm','-i','--name',name,'--network','none',
 '--memory','128m','--pids-limit','64',image,'python3','-'],input=program,
 capture_output=True,text=True,timeout=60)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),ensure_ascii=False,sort_keys=True))
finally:
 subprocess.run(['docker','rm','-f',name],capture_output=True,text=True,timeout=10)

The string A中🙂 has three Python string elements representing code points and encodes to eight bytes: 41e4b8adf09f9982. ASCII contributes one byte, the Chinese character three, and the emoji four. The round trip returns the original text. Counting encoded bytes and counting these code points therefore answer different questions; neither should automatically be used as a displayed-column count.

The bytes are split into 41e4, b8adf0, and 9f9982. Each piece fails when independently decoded as complete UTF-8. An incremental decoder instead produces A, then 中, then 🙂, retaining partial sequences between calls. A final empty call with final=True completes validation. A separate decoder holding only e4b8 rejects that incomplete sequence when finalized, rather than silently losing a trailing character.

Four negative cases reject an overlong encoding, a surrogate code point, a value above the Unicode limit, and an incomplete sequence. Valid UTF-8 excludes those forms even when some bytes resemble plausible continuation patterns. The example catches UnicodeDecodeError specifically. It does not accept an unrelated exception as proof that byte validation worked, and it does not use permissive surrogate handling.

Decoding the valid UTF-8 bytes as Latin-1 succeeds but yields different code points. In this controlled case, encoding that mistaken string back as Latin-1 recovers the original bytes. That reversible example is not a universal repair method: prior replacement, normalization, or lossy conversion may have discarded information. Preserve original bytes when diagnosing an encoding mismatch before attempting a corrective conversion.

The script also compares decomposed e plus a combining acute accent with composed é. The strings initially differ and have lengths two and one; NFC normalization makes them equal. Separately, plain UTF-8 decoding keeps an initial BOM as U+FEFF, while utf-8-sig consumes it. These are two different transformations, and neither should be applied merely because a text string looks unusual.

Define the byte contract at each boundary

An HTTP charset declaration or a file-format specification tells a consumer how to interpret bytes; changing the declaration does not transcode the payload. Ensure the producing encoder, transport metadata, parser, and storage interface agree. Test a multibyte sample across the actual boundaries, especially where code truncates values, buffers partial reads, or counts limits expressed in bytes rather than characters.

Choose error handling according to the data contract. The fixture shows that errors=‘replace’ turns invalid input into U+FFFD, a valid but lossy text result. That may suit a diagnostic display, but it cannot prove preservation of identifiers or signed data. Keep rejection, replacement, BOM removal, and normalization as deliberate decisions with different observable effects.

Finally, use a character-aware operation when limiting decoded text, and a specified framing rule when limiting encoded messages. Incremental decoding solves split byte sequences; it does not define grapheme clusters, line wrapping, or database comparison semantics. Verify the property the application actually requires instead of treating “we use UTF-8” as proof that every text boundary is correct.

Sources

RFC 3629 UTF-8, valid byte sequences, scalar range and invalid encodings, checked 2026-09-12.
Unicode UTF and BOM FAQ, encoding boundaries and byte-order-mark behavior, checked 2026-09-12.
Python 3.11 codecs, incremental decoding, finalization and error policies, checked 2026-09-12.
Python 3.11 unicodedata, Unicode normalization as a separate operation, checked 2026-09-12.