NTP — Network Time Protocol

Time is not a peripheral concern in networking — it is foundational. TLS certificates have validity windows defined by NotBefore and NotAfter timestamps. A clock that is wrong by one day can render all certificates invalid or allow expired ones to pass. Kerberos enforces a 5-minute clock skew tolerance — beyond that, ticket validation fails and authentication breaks. Distributed databases use timestamps for conflict resolution. Log correlation requires synchronized clocks across all devices. Financial systems require sub-millisecond accuracy for regulatory audit trails.

NTP (Network Time Protocol) is the protocol that keeps these clocks synchronized. Version 1 was published in 1985 by David Mills. Today, NTPv4 (RFC 5905) is the standard, with NTS (Network Time Security, RFC 8915) adding cryptographic authentication. NTP operates across the public internet, achieving millisecond-level synchronization over heterogeneous paths with variable delays.

The NTP Pool Project

The NTP Pool Project (pool.ntp.org) is a large virtual cluster of volunteer-run NTP servers. DNS round-robin returns different server IPs based on the client's region. Over 4,000 servers participate globally. Most operating systems default to pool.ntp.org or vendor-specific pools (time.apple.com, time.google.com, time.windows.com).

For production infrastructure, use 4+ servers from diverse sources: 2 from the NTP pool + 2 from your cloud provider. Using servers from different autonomous systems ensures a single network event doesn't affect all your NTP sources simultaneously.

Public Stratum 1 Servers

time.nist.gov: NIST (National Institute of Standards and Technology) — U.S. government atomic standard.

time.cloudflare.com: Cloudflare's Stratum 1 service with NTS support. Uses their anycast network.

time.google.com: Google's time servers with "smeared" leap seconds (distribute the second across 20 hours rather than inserting a step).

time.apple.com: Apple's NTP infrastructure, used by macOS/iOS by default.

The Four Timestamps

Each NTP exchange involves four timestamps:

T1: Time the client sent the request (client's clock)

T2: Time the server received the request (server's clock)

T3: Time the server sent the reply (server's clock)

T4: Time the client received the reply (client's clock)

Offset and Round-Trip Delay Calculation

From these four timestamps, NTP computes:

# Round-trip delay (RTD):
delay = (T4 - T1) - (T3 - T2)
# = total elapsed time - time server spent processing
# This is the network transmission time for both directions

# Clock offset:
offset = ((T2 - T1) + (T3 - T4)) / 2
# = average of (forward delay skew) and (backward delay skew)
# Positive offset: client clock is slow; Negative: client clock is fast

# Example:
# T1 = 10:00:00.000 (client sent)
# T2 = 10:00:00.030 (server received, 30ms after T1 by server clock)
# T3 = 10:00:00.031 (server sent, 1ms processing)
# T4 = 10:00:00.062 (client received)
#
# delay = (0.062 - 0.000) - (0.031 - 0.030) = 0.062 - 0.001 = 0.061s
# offset = ((0.030 - 0.000) + (0.031 - 0.062)) / 2 = (0.030 - 0.031) / 2 = -0.0005s
# Client clock is 0.5ms fast

Asymmetric Delays

NTP assumes symmetric network delay (forward ≈ reverse). This assumption breaks when paths are asymmetric — for example, satellite uplinks (low latency one way, high latency other way) or asymmetric DSL. Asymmetry introduces a systematic offset proportional to half the asymmetry. NTP cannot detect or compensate for asymmetry without external information.

NTP Timestamp Encoding

NTP timestamps are 64-bit fixed-point numbers: 32 bits for seconds since January 1, 1900 (UTC) and 32 bits for fractional seconds. The fractional part represents 1/2^32 seconds ≈ 233 picoseconds per LSB — theoretically sub-nanosecond precision, though practical precision is limited by hardware clock resolution and network jitter.

# NTP timestamp representation:
# 64 bits = [32-bit seconds | 32-bit fraction]
# Seconds since 1900-01-01T00:00:00Z

# Converting NTP timestamp to Unix time:
# Unix epoch = 1970-01-01 = 70 years after NTP epoch
# Offset = 70 years in seconds = 2208988800

unix_time = ntp_seconds - 2208988800

# Example NTP timestamp: 0xE9944000.0x00000000
# 0xE9944000 = 3919880192 seconds since 1900
# Unix time = 3919880192 - 2208988800 = 1710891392
# = 2024-03-19T18:43:12Z

Intersection Algorithm

Each NTP server provides not just a time estimate, but a confidence interval (the offset ± a maximum error bound calculated from dispersion and round-trip delay). The intersection algorithm finds the smallest interval that intersects with the maximum number of server confidence intervals. Servers whose intervals fall outside the intersection are rejected as "falsetickers."

This algorithm guarantees that as long as more than half your NTP servers are accurate (honest), the selected time will be correct — the minority of bad servers cannot override the majority. This is why you need at least 3 NTP servers (1 bad out of 3 is still a minority) and ideally 5+ for robustness.

Clustering and Selection

After the intersection algorithm selects "truechimers" (servers that agree), NTP further filters by jitter and synchronization distance. A server with high jitter (variable delay) is penalized even if its offset is small. The final clock source is called the system peer — the single server driving the local clock correction.

Slewing vs. Stepping

When NTP finds an offset between the local clock and the correct time, it corrects it one of two ways:

Slewing: slowly adjusting the clock frequency to drift toward the correct time. The kernel's adjtime()/adjtimex() syscall adjusts the clock rate by up to 500 parts per million (0.5ms/s). For a 100ms offset, slewing takes ~200 seconds. This preserves time monotonicity — clocks never go backwards.

Stepping: an immediate jump to the correct time. NTP uses stepping only for large initial offsets (128ms threshold by default in ntpd). Stepping can cause problems: log timestamps appear to jump, lease timers may expire prematurely, Kerberos tickets may invalidate.

ntpd Configuration

# /etc/ntp.conf
# At least 4 servers for Marzullo algorithm to work properly
server time1.google.com iburst prefer
server time2.google.com iburst
server time.cloudflare.com iburst
server 0.pool.ntp.org iburst

# iburst: send 8 requests on first contact for faster initial sync

# Security: restrict access
restrict default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict ::1
restrict 10.0.0.0 mask 255.0.0.0 nomodify notrap  # allow internal clients

# Drift file: stores clock frequency correction
driftfile /var/lib/ntp/ntp.drift

# Log file
logfile /var/log/ntp.log
logconfig =syncall +sysall

# Disable monlist (amplification attack vector)
disable monitor

chrony Configuration

# /etc/chrony.conf (chrony, preferred for modern Linux)
server time.cloudflare.com iburst nts    # NTS = Network Time Security (encrypted)
server time.google.com iburst
server 0.pool.ntp.org iburst
server 1.pool.ntp.org iburst

# Allow large initial step (for VM or container first boot)
makestep 1.0 3    # step up to 1s for first 3 clock updates, then slew only

# Hardware timestamping (requires supported NIC)
hwtimestamp *

# Drift file
driftfile /var/lib/chrony/drift

# Security
allow 10.0.0.0/8    # allow clients in 10.0.0.0/8 to sync from this server
deny all            # deny everything else

# Log
logdir /var/log/chrony
log tracking measurements statistics

Checking Synchronization Status

# chrony
chronyc tracking              # current time source and offset
chronyc sources -v            # all configured sources with status
chronyc sourcestats           # statistics for each source
chronyc ntpdata               # detailed NTP data for each source

# ntpd
ntpq -p                       # peer status table
ntpstat                       # brief sync status
timedatectl status            # systemd view of time sync

# Example chronyc sources output:
# MS Name/IP address   Stratum Poll Reach LastRx Last sample
# ^* time.cloudflare.com    1    6   377    23   +0.123ms[+0.234ms]  ±  0.456ms
# ^  time.google.com        1    6   377    24   -0.089ms[-0.189ms]  ±  0.567ms
# ^ = server, * = selected source, + = acceptable but not selected

Symmetric Key Authentication (RFC 1305 / RFC 5905)

Both client and server pre-share a secret key. Each NTP packet includes a key ID and a MAC (MD5 or SHA-1 HMAC of the packet). The receiver verifies the MAC. This prevents modification of NTP packets in transit.

Limitations: requires pre-shared key distribution (does not scale to public pool servers), uses MD5/SHA-1 (aging), does not prevent replay attacks without additional measures.

# ntpd symmetric key example
# /etc/ntp.keys
1 MD5 MySecretKey123

# /etc/ntp.conf
server 192.168.1.1 key 1
trustedkey 1
requestkey 1
controlkey 1

NTS: Network Time Security (RFC 8915)

NTS is the modern, TLS-based solution for secure NTP. The NTS Key Establishment (NTS-KE) protocol runs over TLS 1.3 on TCP port 4460, establishing shared keys and providing server authentication via PKI certificates. Subsequent NTP exchanges use UDP/123 with extension fields carrying the NTS authentication material.

NTS advantages over symmetric keys: server authentication (no MITM on first contact), per-packet authentication with fresh keys, no manual key distribution, compatible with existing PKI infrastructure.

# chrony with NTS (Network Time Security)
server time.cloudflare.com iburst nts
server ntppool1.time.nl iburst nts

# chrony verifies the server's TLS certificate
# using the system CA store (/etc/ssl/certs/ca-certificates.crt)

# Check NTS status:
chronyc ntpdata  # shows NTS-KE and NTS cookie status

How NTP Handles Leap Seconds

NTP servers set the Leap Indicator (LI) bits in the NTP packet header to warn clients of an impending leap second: LI=1 means the last minute of the day will have 61 seconds; LI=2 means 59 seconds. Clients see this warning and adjust their behavior.

The problem: the kernel and operating system must handle the actual insertion. Linux kernels historically "froze" the clock for one second (two consecutive identical seconds). Many applications that use gettimeofday() without checking for LI would see time go backwards by one second, triggering negative time deltas, spin loops, or crashes.

Leap Second Smearing

Google pioneered leap second smearing: instead of inserting a sharp 1-second step, the extra second is distributed across 20 hours (±10 hours around midnight UTC). During those 20 hours, Google's time servers run at a slightly different rate (1.0001157 instead of 1.0), effectively spreading the leap over time so no application ever sees a 23:59:60 or a repeated second.

Cloudflare, Amazon AWS, and Microsoft Azure also use smearing. The tradeoff: during smearing, Google's clocks disagree with non-smearing NTP servers by up to 0.5 seconds. Do not mix smearing and non-smearing sources in the same NTP configuration.

The IERS (International Earth Rotation and Reference Systems Service) announced in 2022 that leap seconds will be discontinued by 2035, replacing them with a larger accumulated correction every century. Until then, leap second handling remains a risk in production systems.

Best Practices for NTP Security

1. Disable monlist (ntpd): add disable monitor to ntp.conf. Already disabled in chrony by default.

2. Restrict NTP server access: if running an NTP server, use restrict directives to limit which hosts can query it.

3. Block port 123 UDP at internet border for servers that should not receive internet queries.

4. Use NTS for NTP sources that authenticate: configure nts option in chrony.conf for public NTS servers.

5. Use 4+ diverse sources: multiple sources from different ASes make falseticker attacks much harder.

6. Monitor clock offset: alert on offset > 100ms (possible MITM or broken server). Alert on stratum jumps (server losing its reference).

PTP (IEEE 1588) vs NTP

PTP operates over Ethernet (or IP multicast) using hardware timestamping at the NIC and network switch level. The key difference from NTP: hardware timestamps are added at the physical layer — the actual moment a packet's first bit departs or arrives at the wire. Software-based NTP timestamps include the OS scheduling jitter (microseconds to milliseconds). Hardware-based PTP timestamps are accurate to nanoseconds.

PTP boundary clocks and transparent clocks in managed switches compensate for switch queuing and forwarding delays, eliminating the variable delay that limits NTP accuracy.

# Linux PTP (linuxptp) configuration
# ptp4l.conf
[global]
tx_timestamp_timeout 10
logAnnounceInterval 1
logSyncInterval 0        # 1 message/second
logMinDelayReqInterval 0

# Hardware timestamping
[eth0]
# Enable hardware timestamping on the interface
# ethtool -T eth0 → shows hardware-timestamping capability

# Start PTP daemon
ptp4l -i eth0 -f /etc/linuxptp/ptp4l.conf

# Synchronize OS clock to PTP hardware clock
phc2sys -s eth0 -c CLOCK_REALTIME -n 16 -O 0

Grandmaster Clock

In PTP, the master clock is called the Grandmaster Clock — the network's primary time reference, typically disciplined by GPS. The Best Master Clock Algorithm (BMCA) automatically selects the grandmaster based on clock class, clock accuracy, offset scaled log variance, and priority fields in Announce messages.

The Ordering Problem

In a distributed system, two events happening simultaneously on different machines will have different wall-clock timestamps unless clocks are perfectly synchronized. If Clock A is 100ms ahead of Clock B, an event on A at 10:00:00.100 looks like it happened before an event on B at 10:00:00.150 — even if B's event actually caused A's event. This breaks causality.

Solutions: Logical clocks (Lamport timestamps, Vector clocks) establish causal ordering without wall-clock sync. For external ordering (clients need to know real-world order), physical clock synchronization (NTP, PTP) is required. Google Spanner uses TrueTime to bound physical clock uncertainty and ensure real-time ordering guarantees.

Certificate Validity and Clock Drift

TLS certificate NotBefore and NotAfter fields use UTCTime or GeneralizedTime. A certificate's validity window is checked against the local clock. If a server's clock is wrong by even one day:

— Clock too early: certificates appear not-yet-valid. TLS handshakes fail with "certificate not yet valid."

— Clock too late: expired certificates appear valid. Security checks fail silently.

— OCSP stapling: OCSP responses expire after hours to days. If the server's clock is wrong, stapled responses appear expired and clients reject them.

VMs and Clock Discipline

Virtualization adds complexity to timekeeping. A VM's clock is emulated — the hypervisor periodically synchronizes the VM's software clock to the host's clock. During high CPU load or VM migration, the VM clock can drift significantly. VMware recommends either using VMware Tools' time synchronization or configuring the VM to use an NTP server, but not both simultaneously.

Container best practice: disable NTP inside containers; rely on the host's NTP-synchronized clock. If a container needs a specific timezone, set the TZ environment variable — this does not affect the clock, only how times are displayed.

AWS Time Sync Service

AWS provides a local NTP endpoint at 169.254.169.123 (link-local, always reachable within AWS VPC). Amazon Time Sync Service uses a fleet of atomic clocks in each AWS region. Amazon Linux 2 and AL2023 configure chrony to use this endpoint by default. AWS also offers a PTP hardware clock via 169.254.169.253 for instances with enhanced networking — providing sub-microsecond accuracy without leaving the datacenter.

# /etc/chrony.conf (Amazon Linux 2)
server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4

# For PTP hardware clock (requires enhanced networking)
refclock PHC /dev/ptp0 poll 2 dpoll -2 offset 0