TCP Deep Dive

TCP (Transmission Control Protocol) is defined in RFC 793 (1981) with significant extensions in RFC 1122, RFC 2581, RFC 5681, and many others. It provides: reliability (guaranteed delivery via acknowledgment and retransmission), ordering (sequence numbers ensure bytes are delivered in transmission order), flow control (receive window prevents sender from overwhelming receiver), congestion control (adaptive sending rate prevents network saturation), and error detection (checksum over header and data).

TCP vs. UDP — When to Choose Each

TCP's reliability comes at a cost: latency, complexity, and head-of-line blocking. Choosing between TCP and UDP requires understanding these trade-offs:

• Use TCP: HTTP, HTTPS, email (SMTP/IMAP/POP), file transfers, database connections, anything where correctness matters more than latency.

• Use UDP: DNS queries (short request-response, timeout-and-retry is sufficient), video streaming (a dropped frame is better than pausing to retransmit), games (old state is worthless, just send new state), DHCP.

• Use QUIC (UDP-based with TLS): HTTP/3, modern video conferencing (WebRTC data channels). QUIC recovers TCP's reliability in user space while eliminating head-of-line blocking.

SYN Cookies and SYN Flood Defense

A SYN flood attack sends thousands of SYN packets per second with spoofed source IPs. The server responds to each with SYN-ACK and creates a half-open connection entry, consuming memory. If enough SYN-ACKs are sent with no ACK completing the handshake, the server's connection table fills and legitimate connections are rejected.

SYN cookies (RFC 4987) eliminate the need to store state for half-open connections. Instead of storing connection state after receiving a SYN, the server encodes all necessary connection information (client IP, port, ISN, MSS, timestamp) in the initial sequence number of the SYN-ACK. When the legitimate ACK arrives, the server decodes the ISN to reconstruct the connection. No state stored, no memory exhaustion — SYN flood mitigation without resource consumption.

# Check SYN cookie status (Linux)
sysctl net.ipv4.tcp_syncookies          # Should be 1 (enabled)

# Monitor SYN flood activity
netstat -an | grep SYN_RECV | wc -l    # Count half-open connections
ss -n state syn-recv | wc -l           # Alternative

# Full connection state counts
ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn

# tuning
sysctl -w net.ipv4.tcp_max_syn_backlog=65536   # Increase SYN backlog queue
sysctl -w net.ipv4.tcp_synack_retries=2         # Reduce SYN-ACK retries (flood mitigation)

TCP Flags Deep Dive

TCP flags occupy 9 bits in the header. The operationally significant flags:

• SYN: synchronize sequence numbers. Only set during connection establishment. A SYN with ACK is the server's half of the handshake.

• ACK: acknowledgment number is valid. Set in virtually every packet after the initial SYN. The absence of ACK in a non-SYN packet indicates something unusual.

• FIN: no more data from sender. Initiates graceful close. Both sides must send FIN to fully close.

• RST: reset — abort connection immediately. No graceful close. Used when a packet arrives for a closed port, when the connection is aborted due to error, or explicitly by applications using SO_LINGER with timeout=0.

• PSH: push data to application immediately without buffering. Used for interactive applications (SSH, telnet) where each keystroke should be delivered immediately, not wait for a full buffer.

• URG: urgent pointer field is significant. Rarely used in modern protocols — superceded by application-layer priority mechanisms. Old telnet break signal used this.

• ECE + CWR: Explicit Congestion Notification (ECN). When a router experiences congestion, it sets the ECN codepoint in the IP header. The receiver echoes this to the sender via ECE. The sender confirms action taken via CWR. This avoids packet loss as the congestion signal, improving performance.

The Initial Sequence Number (ISN) is the starting point for each direction's byte numbering. Modern OSes choose ISNs using a time-based pseudo-random algorithm (RFC 6528: ISN = MD5(src_ip, src_port, dst_ip, dst_port, secret) + clock_offset). This prevents TCP sequence prediction attacks where an attacker could inject data into an existing connection by guessing the sequence number.

Cumulative vs. Selective Acknowledgment

Basic TCP uses cumulative acknowledgment: ACK=N means "I have received all bytes up to N-1 successfully." If segment 1001–2000 arrives but 2001–3000 is lost, ACK=1001 is sent. When 3001–4000 arrives (out of order), ACK=1001 is still sent (three duplicate ACKs). The sender must retransmit from 2001 onward — even though 3001–4000 was received.

SACK (Selective Acknowledgment, RFC 2018) allows the receiver to inform the sender exactly which segments are received and which are missing. The SACK option contains block pairs (left_edge, right_edge) for each out-of-order segment received. The sender can retransmit only the specific missing segments — not everything after the loss. This dramatically improves performance over lossy links (Wi-Fi, satellite, mobile).

# Check TCP options negotiated in a connection (Linux)
ss -ti dst 8.8.8.8       # Show TCP internals: cwnd, ssthresh, retrans, RTT, MSS, SACK

# Example output from ss -ti:
# cubic wscale:7,7 rto:204 rtt:4.121/1.052 ato:40 mss:1448 pmtu:1500 rcvmss:1448
# rcvbuf:131072 sndbuf:87380 lastsnd:68 lastrcv:68 lastack:68
# pacing_rate 36.8Mbps delivery_rate 25.2Mbps unacked:0 retrans:0/0 dsack_dups:0
# rcv_space:14480 rcv_ssthresh:64448 minrtt:3.5

# Check SACK is enabled
sysctl net.ipv4.tcp_sack           # Should be 1

# Capture SACK options in tcpdump
tcpdump -i eth0 'tcp[tcpflags] & tcp-ack != 0' -vvv | grep SACK

The receive window (rwnd) is a 16-bit field in the TCP header — originally limiting maximum window size to 65,535 bytes (65 KB). On modern networks with 100+ ms round-trip times, the bandwidth-delay product can be hundreds of megabytes — far exceeding 65 KB. Enter TCP Window Scaling (RFC 7323): negotiated during the handshake via a scale factor (0–14 bits), making the effective window up to 1 GB (65535 × 2¹⁴).

The Zero Window condition occurs when rwnd = 0. The sender pauses. The receiver sends a Window Update (pure ACK with non-zero rwnd) when space becomes available. If this update is lost, the sender waits indefinitely — deadlock. TCP prevents this with the Persist Timer: the sender periodically sends a Window Probe to check if the window has reopened.

Nagle Algorithm and PSH

The Nagle algorithm (RFC 896) buffers small writes: if there is unacknowledged data in flight, hold small new segments until either the buffer fills to MSS or all previous data is acknowledged. This coalesces many small writes (interactive typing) into fewer larger segments, dramatically improving efficiency. Side effect: latency. For interactive applications (SSH, gaming), disable Nagle with TCP_NODELAY socket option.

# Disable Nagle algorithm in code (Go example)
conn, _ := net.Dial("tcp", "server:port")
tcpConn := conn.(*net.TCPConn)
tcpConn.SetNoDelay(true)   // TCP_NODELAY — disables Nagle

# Python
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

# Check Nagle on existing connections
ss -ti | grep nodelay   # Shows if TCP_NODELAY is set

# Diagnose delayed ACK + Nagle interaction (a common performance problem)
# Symptom: 40ms delays on small write-read interactions
# Cause: sender holds small write (Nagle), receiver holds ACK (delayed ACK timer 40ms)
# Fix: TCP_NODELAY on sender OR TCP_QUICKACK on receiver (or both)

TCP congestion control is the mechanism by which senders adapt their transmission rate to avoid overloading the network. It uses packet loss and ECN as congestion signals, and maintains a congestion window (cwnd) that limits how much data can be in flight. The actual sending rate is limited by min(cwnd, rwnd).

Congestion Control Phases

• Slow Start: begins at cwnd=1 MSS. For each ACK received, cwnd increases by 1 MSS (exponential growth). Continues until cwnd reaches ssthresh (slow start threshold) or packet loss occurs. Despite the name, this is the fastest growth phase.

• Congestion Avoidance: once cwnd reaches ssthresh, growth becomes additive — 1 MSS per RTT (linear). This is the AIMD (Additive Increase, Multiplicative Decrease) algorithm: cautious probing for available bandwidth.

• Fast Retransmit / Fast Recovery: three duplicate ACKs signal a lost segment (not timeout — the network is still delivering later segments). ssthresh = cwnd/2. cwnd = ssthresh + 3. Retransmit the lost segment. Resume from Congestion Avoidance at ssthresh — not Slow Start. This avoids the performance penalty of dropping to cwnd=1.

• Timeout: retransmission timer expires — much more severe signal. ssthresh = cwnd/2. cwnd = 1 MSS. Restart Slow Start from scratch.

Modern Congestion Control Algorithms

Classic Reno and CUBIC are loss-based: they reduce cwnd only when packet loss occurs. This works well on wired networks but is aggressive on shared links and slow to converge on high-bandwidth links (100G+ WANs):

• CUBIC (Linux default): uses a cubic function to grow cwnd, allowing faster recovery from loss events on high-bandwidth-delay product (BDP) networks. Standard on Linux since kernel 2.6.19.

• BBR (Bottleneck Bandwidth and RTT): Google's delay-based algorithm. Instead of reacting to loss, BBR models network state (bandwidth and RTT) and sends at the estimated optimal rate. Dramatically improves performance on lossy links (mobile, intercontinental). Enabled on YouTube's servers since 2016.

• QUIC's congestion control: QUIC (HTTP/3) implements congestion control in user space, allowing per-connection algorithm selection. Different connections from the same app can use different algorithms simultaneously.

# Check and change congestion control algorithm (Linux)
sysctl net.ipv4.tcp_congestion_control       # Show current algorithm
sysctl net.ipv4.tcp_available_congestion_control  # Show available algorithms

# Switch to BBR
sysctl -w net.ipv4.tcp_congestion_control=bbr
sysctl -w net.core.default_qdisc=fq          # BBR works best with fair queueing

# Per-connection in code (Linux):
# setsockopt(fd, IPPROTO_TCP, TCP_CONGESTION, "bbr", strlen("bbr") + 1)

# Monitor congestion window live
watch -n 0.5 'ss -ti | grep -A1 ESTABLISHED'

The Retransmission Timeout (RTO) is computed using the Jacobson algorithm:

• SRTT = (1 - α) × SRTT + α × RTTsample (α = 1/8)

• RTTVAR = (1 - β) × RTTVAR + β × |RTTsample - SRTT| (β = 1/4)

• RTO = SRTT + max(G, 4 × RTTVAR) (G = clock granularity, typically 1ms)

On each retransmission, RTO is doubled (exponential back-off) up to a maximum (typically 60–120 seconds). This prevents retransmission storms during severe congestion. The RTO resets when a segment is successfully acknowledged.

TIME_WAIT serves two purposes:

1. Delayed segment absorption: delayed segments from the closed connection could arrive after a new connection reuses the same 4-tuple. TIME_WAIT (2 × MSL) ensures all delayed segments from the old connection have expired before the 4-tuple can be reused, preventing them from corrupting the new connection's data stream.

2. Reliable FIN-ACK delivery: the final ACK may be lost. The passive closer (server) retransmits its FIN. The active closer must be in TIME_WAIT to respond with ACK — if it were in CLOSED, it would send RST, confusing the server.

TIME_WAIT Reduction Techniques

• TCP_REUSE (net.ipv4.tcp_tw_reuse=1): allow reuse of TIME_WAIT sockets for new outbound connections when safe (requires TCP Timestamps to disambiguate segments). Safe to enable for outbound connections on busy servers.

• SO_REUSEADDR: allows binding to a port that has sockets in TIME_WAIT. Necessary for server restart without waiting for TIME_WAIT expiry.

• Architectural: use connection pooling so connections are reused rather than closed and reopened. The best fix is fewer connection close events.

TCP throughput is bounded by the bandwidth-delay product (BDP): the maximum data "in flight" at any moment. For a 1 Gbps link with 100ms RTT, BDP = 1,000,000,000 bits/second × 0.1 seconds = 100 Mb = 12.5 MB. The TCP window must be at least 12.5 MB to fully utilize the link. Default Linux socket buffers (4 MB) cannot saturate a 1 Gbps intercontinental path.

# TCP buffer tuning for high-BDP paths (Linux)
# View current settings
sysctl net.ipv4.tcp_rmem              # [min, default, max] receive buffer
sysctl net.ipv4.tcp_wmem              # [min, default, max] send buffer

# Increase for high-bandwidth, high-latency paths
sysctl -w net.ipv4.tcp_rmem="4096 131072 67108864"   # max 64 MB receive
sysctl -w net.ipv4.tcp_wmem="4096 65536 67108864"    # max 64 MB send
sysctl -w net.core.rmem_max=67108864
sysctl -w net.core.wmem_max=67108864

# Enable auto-tuning (default on modern Linux — should already be on)
sysctl -w net.ipv4.tcp_moderate_rcvbuf=1

# Enable TCP window scaling (should be default)
sysctl -w net.ipv4.tcp_window_scaling=1

# Calculate required buffer for your BDP:
# bandwidth_bits/s × RTT_seconds / 8 = bytes
# Example: 10 Gbps × 0.1s / 8 = 125 MB minimum buffer for full utilization

TCP Offload — Moving Work to Hardware

Modern NICs offload TCP processing from the CPU:

• TSO (TCP Segmentation Offload): the kernel hands the NIC a large buffer; the NIC splits it into MSS-sized segments and computes checksums. Saves CPU cycles for segmentation and checksum calculation.

• GRO (Generic Receive Offload): the NIC aggregates small incoming segments into larger buffers before passing to the kernel. Reduces per-packet interrupt overhead.

• RSS (Receive Side Scaling): distributes incoming connections across multiple CPU cores using hardware hashing, enabling multi-core TCP processing.

Key TCP options and their operational importance:

• MSS (Maximum Segment Size, Option 2): each side advertises the maximum segment it can receive in the SYN. Default TCP MSS = 536 bytes; Ethernet default = 1460 bytes (1500 MTU - 20 IP - 20 TCP). MSS is NOT negotiated — each side independently declares its limit; the sender uses the minimum.

• Window Scale (Option 3): scale factor for the window field. Negotiated in SYN/SYN-ACK only. If one side doesn't include it, window scaling is disabled for the connection. Always present on modern systems.

• SACK (Option 4 — SACK Permitted) + Option 5 (SACK Blocks): SACK Permitted advertised in SYN/SYN-ACK. SACK blocks (up to 4 ranges) carried in ACKs to report out-of-order receipt. Critical for performance over lossy links.

• Timestamps (Option 8): TSval (timestamp value) and TSecr (timestamp echo reply). Enables precise RTT measurement, PAWS protection, and improved retransmission decisions.

• TCP Fast Open, TFO (Option 34): allows data to be sent in the SYN packet on subsequent connections, eliminating one RTT of setup latency. Uses a cookie mechanism to prevent SYN data amplification.

• Multipath TCP, MPTCP (RFC 8684): multiple subflows over different paths (e.g., WiFi + cellular), transparent to applications. Used on iOS for Siri and FaceTime for seamless handoff between networks.

# TCP connection debugging toolkit

# 1. Check connection state
ss -tn dst 10.0.0.5:5432               # Specific connection state
ss -tn state established               # All established connections
ss -tn state syn-sent                  # Connections waiting for SYN-ACK (connect timeout)
ss -tn state time-wait | wc -l         # TIME_WAIT count (high = rapid connection cycling)

# 2. Check TCP retransmissions (high = packet loss or MTU issue)
ss -ti dst 10.0.0.5 | grep retrans
netstat -s | grep -i retran
cat /proc/net/snmp | grep Tcp

# 3. Capture the problem
tcpdump -i eth0 -w /tmp/capture.pcap 'host 10.0.0.5 and port 5432'
# Look for: retransmissions (same seq twice), zero window, RSTs

# 4. Check MTU on the path
ping -M do -s 1452 10.0.0.5           # Test 1480 byte packets
tracepath 10.0.0.5                     # Shows MTU changes at each hop

# 5. Check kernel TCP error counters
netstat -s | grep -E "failed|reset|error|timeout"

# 6. Watch real-time TCP events (Linux ftrace)
echo 1 > /sys/kernel/debug/tracing/events/tcp/enable

TCP Head-of-Line Blocking

TCP delivers bytes in order. If packet N is lost, packets N+1, N+2, ... are buffered and not delivered to the application until N is retransmitted and received. This is TCP head-of-line blocking: a single lost packet holds up everything behind it in the stream.

HTTP/2 multiplexes multiple request/response streams over one TCP connection. If one stream's data is lost, TCP holds up ALL streams — including those with no data loss. A 1% packet loss that only affects one stream stalls all 30 streams in an HTTP/2 connection. HTTP/3 / QUIC solves this by implementing independent stream delivery in user space: a lost packet only blocks the one QUIC stream that contained it, not others.

TCP Fast Open

Standard TCP requires 1 RTT for handshake + 1 RTT minimum for the first request. On a 100ms path, that is 200ms before the server processes the first byte of the request. TCP Fast Open (TFO, RFC 7413) allows data in the SYN packet on repeat connections, reducing first-request latency to 1 RTT. Chrome and iOS use TFO for performance-sensitive connections.

# Enable TCP Fast Open (Linux)
sysctl -w net.ipv4.tcp_fastopen=3      # 1=client, 2=server, 3=both

# Verify TFO in connection
ss -ti | grep tfo                       # Look for "fastopen" in output

# TFO in server code (Python)
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_TCP, socket.TCP_FASTOPEN, 5)   # 5 = backlog for TFO
s.bind(('0.0.0.0', 8080))
s.listen()