UTF-8 preserves text only when byte boundaries and decoding rules agree

Wed, 02 Dec 2026 00:00:00 +0000

UTF-8 encodes Unicode scalar values using one to four bytes. ASCII characters retain their familiar one-byte representation, while many other characters need longer sequences. This makes UTF-8 a useful interchange format, but it does not make arbitrary bytes valid text or every byte offset a character boundary. Correct handling requires an agreed encoding, strict validity rules where appropriate, and a clear distinction between bytes and decoded text.

When it breaks

A file or network read can end halfway through a multibyte sequence. Decoding each chunk independently may reject a perfectly valid complete stream. Replacing errors separately in every chunk is worse when it destroys bytes that would have formed a character with the next read. A streaming decoder needs to retain incomplete input and validate it when more bytes arrive or the stream ends.

UTF-8’s recognizable leading and continuation-byte patterns do not make arbitrary slicing safe. A program that truncates an encoded byte string can cut a sequence in half. Nor does successful decoding under another codec prove that the chosen interpretation is correct. Latin-1 maps every byte, so wrongly decoding UTF-8 as Latin-1 can produce valid Unicode text whose content is nevertheless mojibake.

Encoding validity is separate from text equivalence and display width. A single visible accented letter can use one code point or a base letter plus a combining mark. Normalization can make selected forms comparable, but decoding UTF-8 does not automatically normalize them. Similarly, a byte-order mark may become a leading text character unless the selected codec intentionally consumes it.

Check it quickly

Save this as check-utf8.py and run python3 check-utf8.py with Docker. It uses standard-library codecs and Unicode operations in a pinned Python 3.11.16 container without network access. All examples are fixed in-memory bytes and strings. The script performs no database migration or file conversion and does not depend on the terminal correctly displaying an emoji:

import json,subprocess,uuid
image="python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84"
program=r"""
import codecs,json,unicodedata

text='A中🙂';encoded=text.encode('utf-8');assert len(text)==3 and len(encoded)==8
assert encoded.hex()=='41e4b8adf09f9982' and encoded.decode('utf-8')==text
pieces=[encoded[:2],encoded[2:5],encoded[5:]]
split_errors=0
for piece in pieces:
 try:piece.decode('utf-8')
 except UnicodeDecodeError:split_errors+=1
assert split_errors==3
decoder=codecs.getincrementaldecoder('utf-8')(errors='strict');decoded=[]
for piece in pieces:decoded.append(decoder.decode(piece,final=False))
decoded.append(decoder.decode(b'',final=True));assert ''.join(decoded)==text
rejected={}
for label,raw in {'overlong':bytes.fromhex('c0af'),'surrogate':bytes.fromhex('eda080'),
 'too_large':bytes.fromhex('f4908080'),'truncated':bytes.fromhex('e4b8')}.items():
 try:raw.decode('utf-8');raise AssertionError('invalid UTF-8 accepted')
 except UnicodeDecodeError as error:rejected[label]=error.reason
bad_tail=codecs.getincrementaldecoder('utf-8')();assert bad_tail.decode(bytes.fromhex('e4b8'),final=False)==''
try:bad_tail.decode(b'',final=True);raise AssertionError('incomplete final sequence accepted')
except UnicodeDecodeError:final_rejected=True
mojibake=encoded.decode('latin-1');assert mojibake!=text and mojibake.encode('latin-1')==encoded
decomposed='e\u0301';composed='é';assert decomposed!=composed and unicodedata.normalize('NFC',decomposed)==composed
bom=b'\xef\xbb\xbf'+encoded;assert bom.decode('utf-8').startswith('\ufeff') and bom.decode('utf-8-sig')==text
replacement=bytes.fromhex('ff41').decode('utf-8',errors='replace');assert replacement=='\ufffdA'
print(json.dumps({'text':text,'code_points':len(text),'bytes':len(encoded),'hex':encoded.hex(),
 'chunk_hex':[x.hex() for x in pieces],'independent_chunk_failures':split_errors,'incremental_output':decoded,
 'invalid_rejections':rejected,'final_incomplete_rejected':final_rejected,'latin1_code_points':[hex(ord(c)) for c in mojibake],
 'normalization_lengths':[len(decomposed),len(composed)],'bom_utf8_first':hex(ord(bom.decode('utf-8')[0])),
 'utf8_sig_strips_bom':bom.decode('utf-8-sig')==text,'replacement':replacement},ensure_ascii=False))
"""
name='xbcatm-socket-'+uuid.uuid4().hex[:10]
try:
 result=subprocess.run(['docker','run','--rm','-i','--name',name,'--network','none',
 '--memory','128m','--pids-limit','64',image,'python3','-'],input=program,
 capture_output=True,text=True,timeout=60)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),ensure_ascii=False,sort_keys=True))
finally:
 subprocess.run(['docker','rm','-f',name],capture_output=True,text=True,timeout=10)

The string A中🙂 has three Python string elements representing code points and encodes to eight bytes: 41e4b8adf09f9982. ASCII contributes one byte, the Chinese character three, and the emoji four. The round trip returns the original text. Counting encoded bytes and counting these code points therefore answer different questions; neither should automatically be used as a displayed-column count.

The bytes are split into 41e4, b8adf0, and 9f9982. Each piece fails when independently decoded as complete UTF-8. An incremental decoder instead produces A, then 中, then 🙂, retaining partial sequences between calls. A final empty call with final=True completes validation. A separate decoder holding only e4b8 rejects that incomplete sequence when finalized, rather than silently losing a trailing character.

Four negative cases reject an overlong encoding, a surrogate code point, a value above the Unicode limit, and an incomplete sequence. Valid UTF-8 excludes those forms even when some bytes resemble plausible continuation patterns. The example catches UnicodeDecodeError specifically. It does not accept an unrelated exception as proof that byte validation worked, and it does not use permissive surrogate handling.

Decoding the valid UTF-8 bytes as Latin-1 succeeds but yields different code points. In this controlled case, encoding that mistaken string back as Latin-1 recovers the original bytes. That reversible example is not a universal repair method: prior replacement, normalization, or lossy conversion may have discarded information. Preserve original bytes when diagnosing an encoding mismatch before attempting a corrective conversion.

The script also compares decomposed e plus a combining acute accent with composed é. The strings initially differ and have lengths two and one; NFC normalization makes them equal. Separately, plain UTF-8 decoding keeps an initial BOM as U+FEFF, while utf-8-sig consumes it. These are two different transformations, and neither should be applied merely because a text string looks unusual.

Define the byte contract at each boundary

An HTTP charset declaration or a file-format specification tells a consumer how to interpret bytes; changing the declaration does not transcode the payload. Ensure the producing encoder, transport metadata, parser, and storage interface agree. Test a multibyte sample across the actual boundaries, especially where code truncates values, buffers partial reads, or counts limits expressed in bytes rather than characters.

Choose error handling according to the data contract. The fixture shows that errors=‘replace’ turns invalid input into U+FFFD, a valid but lossy text result. That may suit a diagnostic display, but it cannot prove preservation of identifiers or signed data. Keep rejection, replacement, BOM removal, and normalization as deliberate decisions with different observable effects.

Finally, use a character-aware operation when limiting decoded text, and a specified framing rule when limiting encoded messages. Incremental decoding solves split byte sequences; it does not define grapheme clusters, line wrapping, or database comparison semantics. Verify the property the application actually requires instead of treating “we use UTF-8” as proof that every text boundary is correct.

Sources

RFC 3629 UTF-8, valid byte sequences, scalar range and invalid encodings, checked 2026-09-12.
Unicode UTF and BOM FAQ, encoding boundaries and byte-order-mark behavior, checked 2026-09-12.
Python 3.11 codecs, incremental decoding, finalization and error policies, checked 2026-09-12.
Python 3.11 unicodedata, Unicode normalization as a separate operation, checked 2026-09-12.

Encoding · XB Field Notes

UTF-8 preserves text only when byte boundaries and decoding rules agree

When it breaks

Check it quickly

Define the byte contract at each boundary

Sources