A listening socket keeps two queues, not one. Half-open connections live in the SYN queue until the final ACK arrives; established ones then wait in the accept queue until the application calls accept(). Both have a limit, and which one overflows decides what the client experiences.

SYN cookies exist for the first queue. When it is full the server stops allocating state and encodes what it needs into the sequence number, so the handshake completes with no memory held. That is a real rescue and a degradation: the connection it saves is worse than the one it would have built normally.

Two queues, two different symptoms

A SYN allocates a request socket, moves the connection to SYN_RECV and sends a SYN-ACK. The client’s ACK then moves it to the accept queue as ESTABLISHED. The SYN queue is capped per listener by tcp_max_syn_backlog — minimum 128, scaling with memory, about 304 bytes per entry. The accept queue is capped by somaxconn (4096 by default on current kernels, 128 before Linux 5.4) combined with the application’s listen() backlog.

The two overflows look nothing alike. SYN queue overflow is what syncookies were built for: the connection still completes. Accept queue overflow is worse and quieter — with tcp_abort_on_overflow at its default of 0 the server ignores the final ACK, so connect() returns success, the client sends its request, and nothing answers. Measured with backlog 1 against 400 clients: all 400 believed they connected, the server accepted 3, and the other 397 were lost with no error on either side.

With tcp_syncookies at its default of 1, the kernel sends a cookie when the SYN backlog overflows. Instead of allocating a request socket it computes the initial sequence number as a counter in the top 8 bits and a truncated hash in the low 24, the hash covering both addresses, both ports, the counter and a server secret. The counter advances every 64 seconds, so the same 4-tuple yields a different cookie each tick.

Verification reverses it. The ACK’s acknowledgement number carries the counter in its top 8 bits; the server checks it against its current and previous tick, recomputes the hash and compares the low 24 bits. Reproducing that construction: a fresh ACK verifies, one replayed 70 seconds later — a single tick — still verifies, and ACKs 200 and 600 seconds later are rejected as stale. Changing the source port or address by one is rejected outright, so the cookie cannot be lifted onto another connection. Blind forgery probability is one in 16.7 million, which is the entire security argument.

A cookie SYN-ACK carries no options. The kernel documentation is blunt: syncookies “seriously violate TCP protocol, do not allow to use TCP extensions, can result in serious degradation of some services”. The expensive omission is window scaling — without it the receive window caps at 65535 bytes instead of 8388480 at scale factor 7. On a 100 ms path that is 5.2 Mbit/s instead of 671.1 Mbit/s, a 128-fold difference from one missing option. SACK is gone too, so a sender losing several segments in one window recovers one cumulative ACK at a time.

The MSS is partly recovered: the cookie path steals three hash bits to index a small MSS table, and that is the only option that survives. Timestamps, SACK and window scale are simply absent. A server in cookie mode is not just admitting fewer connections; the ones it admits are slow, and that looks like a network problem rather than a handshake one.

When it breaks

The first failure is cookies hiding a real flood from monitoring. SyncookiesSent climbing is the signal, but by design the flood is invisible at the application layer: no half-open backlog to alert on, no failed connections, no error log, just a counter. A dashboard built on error rates or SYN_RECV depth shows a healthy server while it is under sustained attack. Watch the cookie counters and their ratio to new connections, not the application.

The second failure is the accept queue overflowing anyway, which cookies do nothing about. ListenOverflows climbs, clients see timeouts on their first request, and the bottleneck is accept(), not the SYN queue — measured here as 397 of 400 lost with every client believing it connected. The fix is to accept faster, add workers and raise somaxconn and the listen backlog, not to tune syncookies.

Check it quickly

Compare both queues and the counters:

sysctl net.ipv4.tcp_syncookies net.ipv4.tcp_max_syn_backlog net.core.somaxconn
nstat -az | grep -E 'Syncookies|ListenOverflow|ListenDrops'
ss -lnt

On a healthy server all three counters read 0, and ss -lnt shows Send-Q matching the configured backlog with Recv-Q at 0. A rising SyncookiesSent means the SYN queue overflowed; a rising ListenOverflows means the accept queue did, and cookies will not fix that one. If Recv-Q on a listening socket stays above zero, established connections are waiting for an application not calling accept() fast enough.

Sources