Networking Fundamentals — Module 28

SSH — Secure Shell

From the terminal of desperation to the cryptographic bedrock of modern infrastructure: how SSH works, why it replaced everything else, and how to use it without shooting yourself in the foot.

30–42 min May 2026

Chapter 1

The Night Tatu Ylönen Wrote SSH

February 1995. Helsinki. A Finnish researcher named Tatu Ylönen watches his university network get sniffed. The attacker harvested thousands of usernames and passwords from unencrypted Telnet and rlogin sessions crossing the wire. Ylönen has a decision: patch the protocol or replace it. Within three months he ships SSH-1 — an encrypted, authenticated replacement for Telnet, rlogin, and rsh — and releases it free on the internet. Within a year, 2 million users have it. SSH changes the rules of remote administration forever.

Before SSH, remote shell access was a horror show. Telnet sent everything — credentials, keystrokes, output — in plaintext. rlogin and rsh trusted hostnames that could be spoofed. Network sniffers were trivial. The 1995 Helsinki incident was not unique; it was simply the one that produced a solution.

SSH-1 was good but had design flaws. In 2006, the IETF standardized SSH-2 (RFC 4251–4254), which fixed the cryptographic weaknesses, separated authentication from transport, and introduced multiplexed channels. Today SSH-2 is the only acceptable version; SSH-1 must be disabled everywhere.

WOW: SSH is used for more than remote shells. Every time you run git push to GitHub over SSH, every time your CI/CD pipeline deploys code via rsync or scp, every time a Kubernetes operator syncs a secret — SSH is working underneath.

This module covers the SSH protocol from the TCP handshake through the cryptographic key exchange, all authentication methods, channel multiplexing, port forwarding, the agent, certificates, and the security hardening practices that separate a properly locked-down server from a breach waiting to happen.

Chapter 2

The Protocol Stack: Transport, Auth, Connection

Most protocols are monolithic — one specification, one wire format. SSH is different. The IETF deliberately split it into three independent layers, each specified in its own RFC. This layering is why SSH can be extended (new auth methods, new channel types) without touching the cryptographic core.

SSH-2 is composed of three protocol layers stacked on top of TCP:

SSH Transport Layer Protocol (RFC 4253)

The transport layer handles algorithm negotiation, key exchange, encryption, integrity protection, and compression. After the transport layer finishes its work, every subsequent packet is encrypted and MAC-protected. The transport layer produces a session identifier — a hash of values exchanged during key exchange — that is used by the upper layers to bind authentication to a specific session.

Key exchange produces two shared secrets: the session key material and the exchange hash H. Six symmetric keys are derived from these: client-to-server encryption key, server-to-client encryption key, client-to-server MAC key, server-to-client MAC key, client-to-server IV, server-to-client IV. This direction-split design means compromising one direction's key does not reveal the other.

SSH Authentication Protocol (RFC 4252)

Once the transport layer is established, the client authenticates against the ssh-userauth service. The authentication protocol is pluggable: publickey, password, keyboard-interactive, hostbased, and gssapi-with-mic are all defined methods. The server advertises which methods it accepts; clients try them in order.

SSH Connection Protocol (RFC 4254)

After authentication, the connection protocol multiplexes logical channels over the single encrypted TCP connection. A channel can be a shell session, a port-forward, an X11 connection, or any custom application type. Each channel has a sender and recipient channel number, and a window size — SSH implements its own flow control independent of TCP.

SSH Packet Structure (after key exchange):
  uint32   packet_length      (length of payload + padding + padding_length)
  byte     padding_length     (random padding to enforce block boundary)
  byte[n]  payload            (SSH message)
  byte[m]  random padding     (m = padding_length)
  byte[k]  mac                (HMAC of seqno + plaintext, then packet encrypted)

WARN: SSH packet sequence numbers are maintained per direction and reset to 0 after a rekey. If you write custom SSH tooling, failing to handle rekey sequence number resets is a common bug that breaks MAC verification.

Chapter 3

Key Exchange: How Two Strangers Agree on a Secret

Imagine you have never met a server before. You have no shared secret. You need to agree on an encryption key — but everything you send can be observed by an attacker. This is the fundamental problem of key exchange, solved by Diffie-Hellman in 1976 and extended by elliptic-curve variants. SSH uses this mathematics every time a new connection is made.

Elliptic Curve Diffie-Hellman (ECDH) with Curve25519

Modern OpenSSH defaults to curve25519-sha256 for key exchange. Both sides generate an ephemeral EC key pair. The client sends its ephemeral public key. The server sends its ephemeral public key plus its host key signature. Both sides independently compute the same ECDH shared secret. The critical word is ephemeral: these keys are thrown away after the session ends.

The mathematical basis: choose a random scalar a (private key) and compute A = a × G where G is the curve base point. The server does the same with scalar b and point B. The shared secret is K = a × B = b × A = a × b × G. An eavesdropper sees A and B but cannot compute K — this is the elliptic curve discrete logarithm problem.

The Exchange Hash and Host Key Verification

SSH computes an exchange hash H = SHA-256(client_version || server_version || client_KEXINIT || server_KEXINIT || server_host_key || client_ephemeral_pub || server_ephemeral_pub || K). The server signs H with its host private key. The client verifies this signature using the server's public key — which it must already know or trust-on-first-use.

This signature binds the key exchange to the specific server identity. An attacker who can observe but not intercept gets nothing. An attacker who intercepts would need the server's host private key to forge the signature.

Trust-on-First-Use (TOFU) and known_hosts

The first time a client connects to a server, OpenSSH asks: "Are you sure you want to continue connecting? The authenticity of host X can't be established." If you type yes, the host public key fingerprint is stored in ~/.ssh/known_hosts. On all future connections, OpenSSH verifies the server presents the same key — this catches MITM attacks.

WARN: The TOFU model has a fatal weakness: the first connection is unverified. If an attacker intercepts your first-ever SSH connection to a server, they can insert themselves undetected. For critical servers, verify the host key fingerprint out-of-band (e.g., via cloud console) before first connection.

WOW: Curve25519 was designed by Daniel J. Bernstein to be impossible to misimplement — fixed-time scalar multiplication, no weak cofactor issues, no special-case edge inputs. Compare that to NIST P-256, which has a complex cofactor and special-case handling that has caused numerous implementation bugs. Bernstein specifically designed the constants so they could not have been chosen to embed a backdoor.

SSH Connection Handshake

Click any phase to see what is exchanged and why.

ClientServerBoth

1TCP SYN / SYN-ACK / ACKBoth1 RTT

2Protocol Version ExchangeBoth0.5 RTT

3SSH_MSG_KEXINITBoth0.5 RTT

4Key Exchange (ECDH)Both1 RTT

5SSH_MSG_NEWKEYSBoth0 RTT

6Service Request: ssh-userauthClient0.5 RTT

7User AuthenticationClient1 RTT

8SSH_MSG_CHANNEL_OPENClient0.5 RTT

9PTY / Shell RequestClient0.5 RTT

Chapter 4

Host Keys: Server Identity

A host key is the server's identity — the long-term asymmetric key pair that proves "I am the server you connected to last time, not an impostor." Unlike user keys (which identify humans), host keys identify machines. Every SSH server generates them on install and stores them in /etc/ssh/. Losing a host private key forces all clients to re-verify (and accept a "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!" alarm).

Host Key Types

OpenSSH supports multiple host key algorithms:

ed25519 — Preferred. EdDSA over Curve25519. Fast, small key (32 bytes), constant-time, no weak parameter risk. Generate: ssh-keygen -t ed25519

ecdsa — ECDSA over NIST P-256/P-384/P-521. Faster than RSA, smaller than RSA, but NIST curves have theoretical parameter-selection concerns. Still widely used.

rsa — Classic RSA. Minimum 3072 bits for new keys (NIST recommendation). Still compatible with all clients. Slower keygen and signing. Key size: 4096-bit RSA ≈ 256-bit ECC security.

dsa — Deprecated. Fixed 1024-bit key size (by FIPS 186), broken. Never generate DSA host keys.

The known_hosts File Format

# ~/.ssh/known_hosts
github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... (base64 pubkey)
|1|abc123=|def456= ssh-rsa AAAAB3NzaC1yc2EAAAA...   (hashed hostname)

# Hashed format: ssh-keygen -H hashes the hostname for privacy
# The hash is SHA1(hostname) with a random salt — prevents leaking
# which hosts you connect to if your known_hosts is stolen

WARN: Run ssh-keygen -R hostname to cleanly remove a stale host key rather than manually editing known_hosts — the hash entries need proper removal. Never use StrictHostKeyChecking no in production; it defeats the entire host-key trust model.

Chapter 5

User Authentication Deep Dive

Authentication is where most SSH security incidents happen. Not in the crypto — modern AES-CTR with HMAC-SHA2 or ChaCha20-Poly1305 are solid. The failures happen in authentication: reused keys, unprotected private key files, password auth on internet-facing servers, authorized_keys files containing stale or unauthorized entries. This chapter covers every auth method and its real-world trade-offs.

SSH Authentication Methods

Select a method to compare security, usability, and trade-offs.

Security

Very High

Usability

Medium

How It Works

Server stores user public key. Client proves private key possession by signing a challenge derived from session ID.

Best For

All production SSH, automation

Risk

Key theft if private key unprotected

Public Key Authentication: The Mechanics

The client wants to prove it holds the private key corresponding to a public key listed in the server's ~/.ssh/authorized_keys. The protocol:

1. Client sends: public key algorithm name + public key bytes (SSH_MSG_USERAUTH_REQUEST with method=publickey, signed=false) — asking "would you accept this key?"

2. Server checks authorized_keys. If the key is listed, replies SSH_MSG_USERAUTH_PK_OK.

3. Client signs: SHA-256(session_id || "publickey" || username || service || algorithm || pubkey) with its private key.

4. Server verifies the signature. On success: SSH_MSG_USERAUTH_SUCCESS.

The session ID binds the signature to this specific session — a signature from a legitimate user cannot be replayed against a different session with a different exchange hash.

The authorized_keys File

# ~/.ssh/authorized_keys (on the server)
# Each line: [options] keytype base64key [comment]

# Basic entry
ssh-ed25519 AAAAC3NzaC... user@laptop

# Restricted entry: force a specific command, no PTY, no forwarding
command="/usr/bin/rsync --server -avz . /backup/",no-pty,no-agent-forwarding,no-port-forwarding ssh-ed25519 AAAAC3NzaC...

# Restrict to source IP
from="10.0.1.0/24",no-x11-forwarding ssh-rsa AAAAB3NzaC...

Generating Keys with ssh-keygen

# Best practice: ed25519 with passphrase
ssh-keygen -t ed25519 -C "user@hostname-$(date +%Y)" -f ~/.ssh/id_ed25519

# Generates:
#   ~/.ssh/id_ed25519     (private key, encrypted with passphrase)
#   ~/.ssh/id_ed25519.pub (public key, copy to server's authorized_keys)

# View fingerprint
ssh-keygen -lf ~/.ssh/id_ed25519.pub
# 256 SHA256:abc123... user@hostname (ED25519)

# Convert old RSA to modern format
ssh-keygen -p -f ~/.ssh/id_rsa -m RFC4716

WARN: Always protect your private key with a passphrase. An unprotected private key on a stolen laptop is an immediate full compromise of every server that key authorizes. Use the SSH agent to avoid typing the passphrase repeatedly while keeping the file encrypted.

Chapter 6

SSH Certificates: The Enterprise Answer to authorized_keys Sprawl

A company has 500 engineers and 2,000 servers. The naive approach: each engineer has a key pair, and their public key is added to the authorized_keys file of every server they need access to. That is 500 × 2,000 = 1,000,000 authorized_keys entries to manage. When an engineer leaves, you have to hunt down and remove their key from every server. SSH certificates solve this at the infrastructure level.

How SSH CAs Work

An SSH CA is just another SSH key pair — but it is trusted to sign user and host certificates. The workflow:

1. Each server has TrustedUserCAKeys /etc/ssh/ca_user_key.pub in sshd_config — it trusts any user key signed by this CA.

2. When a user needs access, an admin (or automated system like Vault) signs their public key: ssh-keygen -s ca_key -I "user@corp" -n "john,deploy" -V "+8h" user_key.pub

3. The user presents their certificate during authentication. The server verifies the CA signature and checks principals, validity period, and critical options.

4. No authorized_keys entry needed on any server. To revoke: let the certificate expire, or add the key to RevokedKeys.

Certificate Fields

ssh-keygen -Lf ~/.ssh/id_ed25519-cert.pub

# id_ed25519-cert.pub:
#         Type: ssh-ed25519-cert-v01@openssh.com user certificate
#         Public key: ED25519-CERT SHA256:abc123
#         Signing CA: ED25519 SHA256:ca_fingerprint (using rsa-sha2-512)
#         Key ID: "john@corp"
#         Serial: 42
#         Valid: from 2026-05-24T10:00:00 to 2026-05-24T18:00:00
#         Principals:
#                 john
#                 deploy
#         Critical Options: (none)
#         Extensions:
#                 permit-agent-forwarding
#                 permit-port-forwarding
#                 permit-pty
#                 permit-user-rc

WOW: HashiCorp Vault has an SSH Secrets Engine that functions as a certificate authority. Engineers authenticate to Vault (LDAP, OIDC, etc.), request a signed SSH certificate valid for 30 minutes, and connect to the server. No static authorized_keys. No long-lived keys. Certificates expire automatically. This is the gold standard for enterprise SSH access management.

Chapter 7

The SSH Agent: Unlocking Keys Once

You protect your private key with a passphrase. Good. But typing a 30-character passphrase every time you SSH somewhere would drive you insane. The SSH agent is the solution: a background process that holds your decrypted private key in memory and performs signing operations on behalf of SSH clients. The key never leaves the agent — client applications ask the agent to sign challenges.

How the Agent Works

ssh-agent listens on a Unix domain socket (path stored in $SSH_AUTH_SOCK). The OpenSSH client, when it needs to authenticate, connects to the agent socket and sends a "please sign this challenge" request. The agent returns the signature. The private key material never crosses the socket.

# Start agent and add key
eval "$(ssh-agent -s)"        # sets SSH_AUTH_SOCK and SSH_AGENT_PID
ssh-add ~/.ssh/id_ed25519     # prompts for passphrase ONCE, decrypts key

# Add with expiry (remove key after 8 hours)
ssh-add -t 28800 ~/.ssh/id_ed25519

# List keys in agent
ssh-add -l

# macOS Keychain integration (adds passphrase to Keychain for persistence)
ssh-add --apple-use-keychain ~/.ssh/id_ed25519

Agent Forwarding: Power and Risk

With agent forwarding (ssh -A or ForwardAgent yes), a remote SSH session on server A can use your local agent to authenticate to server B. This enables bastion-host workflows without copying private keys to intermediate hosts.

WARN: Agent forwarding is dangerous on untrusted servers. Root on the remote server can interact with your forwarded agent socket and impersonate you to any server your key authorizes — without ever seeing your private key. Prefer ProxyJump over agent forwarding. If you must forward, only do so to hosts you fully trust.

SSH_AUTH_SOCK Hijacking

On shared servers, if another user has sudo/root, they can find your agent socket (/tmp/ssh-XXXX/agent.YYYY), set SSH_AUTH_SOCK to that path, and use your agent. This is why you should not forward your agent to shared multi-user servers. The socket permissions (user-only) stop other non-root users, but root can bypass them.

Chapter 8

SSH Port Forwarding and Tunneling

SSH can do much more than remote shells. Its channel multiplexing capability lets it act as a general-purpose secure tunnel — forwarding TCP ports, proxying arbitrary protocols, even running VPN-like setups. Network engineers who understand SSH tunneling can reach any resource on a private network through a single SSH-accessible bastion.

SSH Port Forwarding Explorer

Select a forwarding type to see the command, data flow, and use case.

Local Port Forwarding (-L)

COMMAND

ssh -L 8080:internal.corp:80 user@bastion

DATA FLOW

Browser:8080 → [SSH Client] ──SSH tunnel──→ [SSH Server] → internal.corp:80

USE CASE

Access an internal web server through a bastion host. Your local port 8080 maps to the remote network resource.

WARN: Opens a local port — any process on localhost can use the tunnel unless you also pass -o GatewayPorts=no (default).

The Mechanics of Local Forwarding

When you run ssh -L 8080:internal:80 user@bastion, OpenSSH:

1. Opens a listening socket on localhost:8080 (or 0.0.0.0:8080 if GatewayPorts yes).

2. When a local connection arrives on port 8080, opens a new SSH channel of type direct-tcpip with destination internal:80.

3. The SSH server receives the channel-open request and makes a TCP connection to internal:80 on behalf of the channel.

4. Data flows bidirectionally through the channel, encrypted end-to-end in the SSH session.

SOCKS5 Proxy: The Swiss Army Knife

Dynamic forwarding (ssh -D 1080 user@jump) spawns a SOCKS5 server locally. Configure your browser or curl --socks5 to route through it. The SSH client dynamically opens direct-tcpip channels for each SOCKS connection — no fixed destination required. This is essentially a poor-man's VPN for TCP applications.

# Use SOCKS proxy with curl
curl --socks5 localhost:1080 https://internal-service.corp/api

# Use SOCKS proxy in git
git config --global core.gitProxy "nc -x localhost:1080"

# Use SOCKS proxy in Firefox:
# Network Settings → Manual proxy → SOCKS Host: localhost, Port: 1080, SOCKS v5

ProxyJump: The Right Way to Reach Internal Hosts

ProxyJump is the modern, safe way to connect through a bastion. It works by opening a channel to the destination using the bastion as a relay, but the bastion never sees plaintext — it merely relays TCP bytes. Your credentials authenticate to the destination directly.

# ~/.ssh/config
Host bastion
    HostName bastion.corp.example.com
    User admin
    IdentityFile ~/.ssh/corp_ed25519

Host internal-*
    ProxyJump bastion
    User deploy
    IdentityFile ~/.ssh/corp_ed25519

# Now:  ssh internal-app1  connects via bastion automatically

Chapter 9

The SSH Config File: Taming Complexity

Most sysadmins who are new to SSH type the full ssh -i ~/.ssh/specific_key -p 2222 -l deploy internal.corp every time. Experienced engineers write a ~/.ssh/config that reduces that to ssh internal. The config file is one of SSH's most powerful and underused features.

config File Syntax and Match Logic

# ~/.ssh/config
# ---- Global defaults (apply to all hosts) ----
Host *
    ServerAliveInterval 60
    ServerAliveCountMax 3
    AddKeysToAgent yes
    IdentityFile ~/.ssh/id_ed25519

# ---- GitHub ----
Host github.com
    IdentityFile ~/.ssh/github_ed25519
    User git

# ---- Work bastion ----
Host bastion
    HostName bastion.corp.example.com
    User admin
    Port 22
    ControlMaster auto
    ControlPath ~/.ssh/cm-%r@%h:%p
    ControlPersist 10m

# ---- Internal hosts (via bastion) ----
Host *.corp.internal
    ProxyJump bastion
    User deploy
    StrictHostKeyChecking yes
    UserKnownHostsFile ~/.ssh/known_hosts_corp

Connection Multiplexing with ControlMaster

ControlMaster allows multiple SSH sessions to share a single TCP connection. The first connection creates a control socket. Subsequent connections to the same host reuse the socket without TCP handshake or key exchange. This dramatically speeds up repeated operations (git push, scp, ansible).

ControlPersist 10m keeps the master connection alive for 10 minutes after the last session ends, so the next ssh command is near-instant.

WARN: ControlMaster sockets on shared filesystems (NFS, SMB) are dangerous — other users on the same share could hijack the socket. Keep ControlPath on local storage, ideally in /tmp or a user-only directory with mode 700.

Chapter 10

sshd: Hardening the Server

A default OpenSSH install is reasonably secure but not paranoid. An internet-facing SSH server will receive thousands of brute-force attempts per day. Hardening sshd is not optional — it is the difference between an annoying background noise and a breached server.

Essential sshd_config Hardening

# /etc/ssh/sshd_config

# Disable password authentication - keys only
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM no           # or: keep UsePAM yes but disable ChallengeResponse

# Disable root login
PermitRootLogin no
# (or PermitRootLogin prohibit-password if root must be accessible)

# Allow only specific users / groups
AllowUsers deploy ansible
AllowGroups ssh-access

# Modern algorithms only
KexAlgorithms curve25519-sha256,diffie-hellman-group16-sha512
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com

# Disable old protocol features
X11Forwarding no
AllowAgentForwarding no    # unless needed
AllowTcpForwarding no      # unless needed
PrintMotd no
Banner none

# Limit connection rate (supplement with fail2ban)
MaxAuthTries 3
MaxSessions 10
LoginGraceTime 30

# Log level for auditing
LogLevel VERBOSE

Fail2ban and Port Knocking

Fail2ban parses auth log files and bans IPs that exceed failed login thresholds. Install it and configure it to watch /var/log/auth.log (Ubuntu/Debian) or /var/log/secure (RHEL). A typical rule: 5 failures in 10 minutes → ban for 1 hour.

Port knocking hides the SSH port: firewall drops all port-22 traffic until a client sends packets to a secret sequence of ports in order. After the knock sequence, the firewall temporarily opens port 22 for that source IP. This makes the server invisible to port scanners.

Moving SSH to a non-standard port (e.g., 2222) reduces log noise but provides no real security — port scanners find it in seconds. It is security theater, not defense-in-depth.

WOW: Shodan, the internet-connected device search engine, indexes all internet-accessible SSH servers. If your server has PasswordAuthentication yes and is reachable from the internet, automated credential stuffing bots will find it within hours. The average internet-facing server receives 1,000–5,000 SSH brute-force attempts per day.

Chapter 11

SCP, SFTP, and rsync-over-SSH

SSH's channel multiplexing lets it carry more than shell sessions. File transfer protocols built on SSH take advantage of the same authenticated, encrypted transport — without needing a separate security layer. SCP, SFTP, and rsync are the three workhorses.

SCP: Simple Copy

SCP (Secure Copy) uses the SSH connection to transfer files. Modern OpenSSH (9.0+) replaced the legacy SCP protocol (which had path injection vulnerabilities) with SFTP under the hood while keeping the SCP command interface.

# Copy local file to remote
scp file.txt user@server:/remote/path/

# Copy remote file to local
scp user@server:/remote/file.txt ./local/

# Copy recursively
scp -r local_dir/ user@server:/remote/dir/

# Use specific key and port
scp -i ~/.ssh/deploy_key -P 2222 artifact.tar.gz deploy@server:/releases/

SFTP: The Right Tool for File Operations

SFTP is a full file transfer protocol defined in RFC draft-ietf-secsh-filexfer. Unlike SCP, which is a one-shot copy, SFTP is stateful: you open a connection, navigate directories, read/write files, stat metadata. OpenSSH's sftp-server subsystem runs on the remote as a child process.

sftp user@server
# Connected to server.
sftp> ls -la /remote/path
sftp> get remote_file.txt
sftp> put local_file.txt /remote/
sftp> mkdir /remote/new_dir
sftp> chmod 644 /remote/new_dir/file
sftp> bye

rsync over SSH: The Best of Both

rsync over SSH combines rsync's delta-transfer algorithm with SSH's security. Only changed blocks of files are transferred — essential for large files or slow links. The -e ssh flag (default in modern rsync) routes traffic through SSH.

# Sync local to remote (archive mode: preserves permissions, timestamps, symlinks)
rsync -avz --delete local_dir/ user@server:/remote/dir/

# Use specific SSH key and port
rsync -avz -e "ssh -i ~/.ssh/deploy_key -p 2222" dist/ deploy@server:/var/www/

# Dry run: show what would change without doing it
rsync -avzn local/ user@server:/remote/

WARN: The --delete flag removes files on the destination that don't exist on the source. Always do a dry run with -n before a destructive rsync. Forgetting --delete on a backup job means stale deleted files accumulate; including it without -n first can wipe destination-only files.

Chapter 12

SSH in Automation: CI/CD, Ansible, and Deploy Keys

Humans type passphrases. Machines cannot. Automation — CI/CD pipelines, configuration management, deployment tools — needs SSH keys that can authenticate without human interaction. The security challenge: how do you store long-lived SSH credentials without putting a target on your infrastructure?

Deploy Keys (GitHub / GitLab)

A deploy key is an SSH key pair associated with a single repository rather than a user account. The public key is registered in the repository settings with read-only (or read-write) access. The private key is placed in the CI runner or deployment server.

# Generate a dedicated deploy key (no passphrase for automation)
ssh-keygen -t ed25519 -f ~/.ssh/deploy_key_myrepo -C "deploy@myrepo" -N ""

# Add public key to GitHub repo Settings → Deploy Keys
# Store private key in CI/CD secret: SECRET_DEPLOY_KEY

# Use in CI
eval "$(ssh-agent -s)"
echo "$SECRET_DEPLOY_KEY" | ssh-add -
git clone git@github.com:org/repo.git

Ansible and SSH

Ansible manages remote hosts exclusively over SSH. The control node needs key-based authentication to all managed hosts. Key practices:

1. Create a dedicated ansible service account on managed hosts with a locked password (only SSH key auth).

2. Store the Ansible private key in your secrets manager (Vault, AWS Secrets Manager).

3. Use ansible_ssh_common_args: '-o StrictHostKeyChecking=yes' — never disable host key checking in production.

4. Use Ansible Vault to encrypt sensitive variables — SSH keys in playbook repos must be encrypted, not committed in plaintext.

Short-Lived Certificates for Automation

The gold standard: automation requests a signed SSH certificate from Vault (valid 15–60 minutes) at the start of each job. The certificate expires before the job could be replayed. No long-lived secrets on disk. This is available via Vault's SSH Secrets Engine and is the pattern used by large engineering organizations.

Chapter 13

Misconceptions About SSH

MISCONCEPTION: "SSH encrypts everything, so my server is secure." — Encryption is transport security, not access control. If password authentication is enabled and someone brute-forces your credentials, encryption provides zero protection. The attacker's session is encrypted too. Disable password auth.

MISCONCEPTION: "Moving SSH to port 2222 protects against attacks." — This is security through obscurity, not security. Masscan can scan the entire IPv4 internet for all 65535 ports in under 6 minutes. Attackers find non-standard SSH ports within hours. Change the port to reduce log noise if that helps operationally, but never rely on it for security.

MISCONCEPTION: "My private key is safe because it has a passphrase." — A passphrase encrypts the key file on disk. If you add it to an SSH agent on a compromised machine, or if you forward the agent to an untrusted host, your private key is accessible to anyone who can interact with the agent socket — regardless of the passphrase.

MISCONCEPTION: "SSH agent forwarding is safe because the private key never leaves my machine." — True: the key doesn't cross the socket. But the signing oracle does. Root on the remote server can connect to your forwarded agent and sign arbitrary challenges — authenticating as you to any host your key accesses. The key never moves, but the capability does.

MISCONCEPTION: "Root login is only dangerous if password auth is enabled." — Even with key-only auth, permitting root login means a compromised key grants immediate root everywhere that key is authorized. Use a non-root account with sudo access instead. Privilege escalation stays audited in sudo logs; direct root login does not.

MISCONCEPTION: "known_hosts protects me from MITM attacks." — It protects you on the second and subsequent connections. The first connection (TOFU) is unverified. If an attacker intercepts your very first SSH connection to a new server and you type "yes", you have accepted their host key and subsequent connections will verify against the attacker's key, not the real server's.

Chapter 14

IQ Depth Check: How Deep Does Your SSH Knowledge Go?

Beginner

What does SSH stand for and what does it replace?
SSH stands for Secure Shell. It replaces Telnet (unencrypted remote shell), rlogin (trusted-hostname authentication), rsh (remote shell execution), and rcp (remote file copy) — all of which transmitted credentials and data in plaintext over the network.

Intermediate

Explain the difference between -L, -R, and -D port forwarding.
-L (local): Listens on a local port, forwards connections through the SSH tunnel to a destination reachable by the SSH server. Good for accessing internal services through a bastion. -R (remote): Listens on a port on the SSH server, forwards connections back through the tunnel to a destination reachable by your local machine. Good for exposing local services. -D (dynamic): Creates a SOCKS5 proxy locally; the SSH client dynamically opens channels to destinations requested by SOCKS clients — no fixed destination. Good for routing arbitrary traffic.

Senior

How does SSH certificate authentication differ from public key authentication, and when should you use certificates?
Public key auth requires the server to have the user's public key in authorized_keys — managed per-server. Certificates add a CA layer: the server trusts a CA key; users present keys signed by the CA with embedded principals, validity period, and critical options. Benefits: no per-server key distribution, automatic expiry, centralized revocation, auditable serial numbers. Use certificates when managing more than a handful of servers or engineers, or when you need short-lived access (ephemeral certs from Vault with 30-minute TTLs).

PhD

Explain the cryptographic binding in SSH public key authentication — why can't a valid signature from one session be replayed against another?
During public key authentication, the data signed by the client is: SHA-256(session_id || "publickey" || username || service_name || algorithm || public_key_blob). The session_id is the exchange hash H from the key exchange phase — computed as SHA-256(client_version || server_version || client_KEXINIT || server_KEXINIT || host_public_key || client_ephemeral_pub || server_ephemeral_pub || shared_secret K). Since the ephemeral keys are random per session, H is unique per session. Including H in the signed data means a valid signature is bound to a specific session's H — it cannot be reused in a session with a different H. This prevents signature replay attacks across sessions.

🎯 Key Takeaways

✓SSH-2 (RFC 4251–4254) is the only acceptable version; SSH-1 has cryptographic flaws and must be disabled.
✓The SSH protocol has three layers: Transport (encryption, key exchange), Authentication (publickey/password/cert), and Connection (multiplexed channels).
✓Curve25519 ECDH key exchange provides perfect forward secrecy — session keys are ephemeral and not derivable from the host key even if the host key is later compromised.
✓Public key authentication uses challenge-response with the session ID embedded in the signed data, preventing cross-session replay.
✓SSH certificates (SSH CA) eliminate authorized_keys management at scale and enable short-lived, automatically expiring access.
✓Agent forwarding grants root-on-remote the ability to use your agent as a signing oracle — prefer ProxyJump over -A.
✓ControlMaster/ControlPersist multiplexes multiple SSH sessions over one TCP connection, dramatically speeding up repeated operations.
✓Disable PasswordAuthentication, PermitRootLogin, and legacy algorithms in sshd_config; complement with fail2ban.
✓Deploy keys and short-lived certificates from Vault are the right approach for CI/CD and automation — long-lived unprotected private keys in CI secrets are a liability.
✓Trust-on-first-use (TOFU) means the first connection to a new server is the attack window; verify host key fingerprints out-of-band for critical systems.

Discussion

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub