SNMP and Syslog

Network management has two fundamental problems: observability (what is the current state of everything?) and alerting (when does something change that I need to know about?). SNMP and Syslog address these from different angles. SNMP is a structured, typed, polled protocol for querying and modifying device state. Syslog is an asynchronous, text-based stream of events emitted by devices.

Together, they form the monitoring backbone of most enterprise and carrier networks. Understanding both — their design, their security properties, and their limitations — is essential for anyone operating or securing network infrastructure.

The SNMP Model

Manager (NMS): the Network Management System — Nagios, Zabbix, PRTG, Prometheus with SNMP exporter. Sends GET, GETNEXT, GETBULK, SET requests to agents. Receives Trap/Inform notifications.

Agent: software running on the managed device (router, switch, server, UPS). Listens on UDP/161. Responds to manager requests. Sends Traps/Informs on UDP/162 when events occur.

MIB (Management Information Base): a definition file written in SMI (Structure of Management Information) syntax that defines what OIDs exist and what types they have. MIBs are compiled by management software to translate numeric OIDs to human-readable names. Standard MIBs (MIB-II, IF-MIB, HOST-RESOURCES-MIB) apply to all devices; vendor-specific enterprise MIBs extend them.

OID Namespace

OIDs form a global tree rooted at ISO (1). Every device's manageable variables are leaves in this tree:

OID structure:
1 (ISO)
└── 3 (org)
    └── 6 (dod)
        └── 1 (internet)
            ├── 2 (mgmt)
            │   └── 1 (mib-2)
            │       ├── 1 (system)
            │       │   ├── 1.0 = sysDescr (device description)
            │       │   ├── 3.0 = sysUpTime (uptime in 1/100s)
            │       │   └── 5.0 = sysName (hostname)
            │       └── 2 (interfaces)
            │           └── 2.1 (ifTable)
            │               ├── 2.1.2.N = ifDescr.N
            │               ├── 2.1.8.N = ifOperStatus.N
            │               └── 2.1.10.N = ifInOctets.N
            └── 4.1 (enterprise)
                ├── 9 = Cisco
                ├── 2636 = Juniper
                └── 8072 = Net-SNMP

GET and GETNEXT

GET: retrieve the value of a specific OID. The manager sends a GetRequest PDU with one or more OID bindings. The agent replies with a GetResponse containing the values.

GETNEXT: retrieve the next OID in the MIB tree after the specified OID. Used to walk the MIB sequentially — useful for exploring what an agent supports or enumerating table entries.

GETBULK (SNMPv2c/v3)

GetBulk retrieves multiple values in a single request, reducing round-trips when enumerating tables. Parameters: non-repeaters (how many OIDs to GET once) and max-repetitions (how many times to GETNEXT the remaining OIDs). Essential for polling large routing tables or interface tables efficiently.

# snmpwalk: GETNEXT traversal
snmpwalk -v2c -c public 192.168.1.1 1.3.6.1.2.1.2.2.1.8
# → lists ifOperStatus for all interfaces

# snmpget: precise OID retrieval
snmpget -v2c -c public 192.168.1.1 sysUpTime.0
# → SNMPv2-MIB::sysUpTime.0 = Timeticks: (4328100) 5 days, 0:13:21.00

# snmpbulkwalk: GetBulk traversal (faster for large tables)
snmpbulkwalk -v2c -c public 192.168.1.1 ifTable

SET

SET writes a value to a writable OID on the agent. Used for remote configuration: setting interface admin status (ifAdminStatus = 2 to shut down an interface), VLAN assignments, SNMP community string rotation. Requires the community string to have write privileges (SNMPv1/v2c) or a user with write access (SNMPv3).

TRAP vs. INFORM

Trap: agent sends a one-way UDP notification to the manager. No acknowledgement. If the manager is down or the UDP packet is lost, the trap is gone forever. Fast and simple.

Inform (SNMPv2c/v3): the agent sends the notification and waits for the manager to acknowledge. If no ACK, the agent retransmits. More reliable but requires the manager to respond promptly. Most production monitoring uses Informs for critical alerts.

Community Strings

A community string is a text string that must match between manager and agent for the exchange to succeed. SNMPv1 and v2c carry it in plaintext in every PDU. Three typical community strings:

— public: default read-only. Present in default configurations of most network devices. Should be changed before deployment.

— private: default read-write. Even more dangerous. Must be changed.

— Custom: use a long random string, treat it like a password, store in secrets manager.

SNMPv2c Improvements

SNMPv2c (RFC 1901) added GetBulk (critical for performance), 64-bit counters (Counter64 — essential for 1Gbps+ interfaces where 32-bit counters wrap in seconds), and the Inform PDU. Security remained community-string-based.

The Counter Wrap Problem

SNMPv1 uses Counter32 (32-bit) for interface octets. A 32-bit counter wraps at 2^32 bytes = 4.29 GB. On a 1 Gbps interface running at full speed, that is 34 seconds to wrap. A monitoring system polling every 5 minutes cannot distinguish "counter wrapped once" from "zero traffic." SNMPv2c's Counter64 wraps at 2^64 bytes — a 10 Gbps interface running flat out would take 46 years to wrap.

SNMPv3 Security Model (USM)

SNMPv3 uses the User-based Security Model (USM) with three security levels:

noAuthNoPriv: no authentication, no encryption. As insecure as SNMPv1/v2c but with a username. Only acceptable for isolated lab networks.

authNoPriv: authenticated (HMAC), no encryption. Prevents tampering and replay attacks but SNMP data is visible on the wire.

authPriv: authenticated + encrypted. The correct setting for production. Uses HMAC-SHA-256 (or stronger) for auth and AES-256 for encryption.

SNMPv3 Configuration

# Cisco IOS SNMPv3 configuration
snmp-server group MONITORING-GROUP v3 priv read MONITORING-VIEW
snmp-server user monitor MONITORING-GROUP v3 auth sha-256 Auth$ecret123 priv aes 256 Priv$ecret456
snmp-server view MONITORING-VIEW internet included

# Net-SNMP agent (Linux) /etc/snmp/snmpd.conf
createUser monitor SHA-256 "Auth$ecret123" AES "Priv$ecret456"
rouser monitor priv

# Query with SNMPv3
snmpget -v3 -l authPriv -u monitor   -a SHA-256 -A "Auth$ecret123"   -x AES -X "Priv$ecret456"   192.168.1.1 sysUpTime.0

VACM: View-based Access Control

SNMPv3 includes the View-based Access Control Model (VACM), which controls which OIDs each user or group can access (read/write/notify). A monitoring user can be restricted to read-only access on specific MIB subtrees — they cannot SET, and they cannot read sensitive enterprise MIBs containing credentials.

MIB-II (RFC 1213): The Universal Foundation

MIB-II defines the minimum set of objects required for all TCP/IP managed nodes. Groups within MIB-II:

system (1.3.6.1.2.1.1): sysDescr, sysUpTime, sysContact, sysName, sysLocation, sysObjectID.

interfaces (1.3.6.1.2.1.2): ifTable — one row per interface with ifDescr, ifType, ifSpeed, ifOperStatus, ifInOctets, ifOutOctets, error counters.

ip (1.3.6.1.2.1.4): IP forwarding tables, ARP cache, IP statistics.

tcp (1.3.6.1.2.1.6): TCP connection table, statistics.

udp (1.3.6.1.2.1.7): UDP statistics.

IF-MIB (RFC 2863): Modern Interface Monitoring

IF-MIB extends the interfaces group with 64-bit counters (ifHCInOctets, ifHCOutOctets — the HC stands for High Capacity) and ifAlias (operator-set description). Always use IF-MIB's 64-bit counters for modern high-speed interfaces.

HOST-RESOURCES-MIB (RFC 2790)

Provides operating system-level data: hrProcessorLoad (CPU%), hrStorageTable (disk/memory), hrSWRunTable (running processes), hrSWInstalled (installed software). Works on Linux, Windows, BSD, and any OS with a conformant SNMP agent.

Enterprise MIBs

Vendor MIBs live under 1.3.6.1.4.1.ENTERPRISE_ID. You must obtain the vendor's MIB files, compile them into your NMS, and then you can query vendor-specific data: Cisco's per-interface QoS policy stats, Juniper's routing engine temperature, HP's iLO power consumption. Enterprise MIBs are the richest source of data but require vendor-specific effort.

Syslog Origins: RFC 3164 (BSD Syslog)

The original syslog protocol was not standardized — it was the convention used by BSD Unix in the early 1980s. RFC 3164 (2001) documented the existing practice without truly standardizing it. The "format" was loose: a priority value encoding facility and severity, an optional timestamp, a hostname, and a message. No structured fields, no defined escaping, no versioning.

Modern Syslog: RFC 5424

RFC 5424 (2009) defined a proper syslog format with a structured header: VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID [STRUCTURED-DATA] MSG. The structured data section allows key-value pairs, enabling machines to parse fields without string matching.

# RFC 5424 syslog message format:
<priority>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID [SD] MSG

# Example:
<165>1 2026-05-24T10:00:00.123Z fw01.corp.com sshd 12345 - - Failed password for alice from 203.0.113.5 port 49832 ssh2

# Priority = (Facility * 8) + Severity
# Facility 20 (local4) = 20*8 = 160
# Severity 5 (notice) = 5
# Priority = 165

# Structured data example:
<134>1 2026-05-24T10:01:00Z webserver apache 9801 - [request@12345 method="POST" uri="/api/login" status="401" bytes="230"] authentication failure

Syslog Facility Codes

The facility identifies the source of the message. 24 defined facilities:

0 kern (kernel), 1 user, 2 mail, 3 daemon, 4 auth/security, 5 syslog, 6 lpr, 7 news, 8 uucp, 9 cron, 10 authpriv, 16-23 local0–local7 (for application use).

Security-relevant events (authentication, authorization, audit) should use facility authpriv (10) to separate them from general system logs — most log servers can route authpriv to a separate, access-controlled log file.

UDP/514: The Default (and Its Problems)

UDP/514 is the legacy syslog transport. Advantages: zero configuration on most devices, no connection management overhead, simple firewall rules. Disadvantages: no delivery guarantee — during high-traffic periods or network congestion, log messages are silently dropped. No sequence numbers means drops are invisible. No auth means anyone can inject fake log entries.

TCP/514: Reliable Delivery

RFC 6587 defines syslog over TCP. TCP provides delivery guarantees — the sender knows if the receiver got the message. Messages are framed with octet-counting (preferred) or newline-delimited. TCP/514 is unencrypted but reliable. Better than UDP for environments where lost logs are unacceptable but encryption is not required.

TLS/6514: Encrypted and Authenticated

RFC 5425 defines syslog over TLS (port 6514). Provides: delivery guarantees (TCP), encryption (TLS), and mutual authentication (client and server certificates). This is the correct transport for security-critical syslog in production environments.

# rsyslog TLS client configuration (/etc/rsyslog.conf)
global(
  defaultNetstreamDriver="gtls"
  defaultNetstreamDriverCAFile="/etc/ssl/certs/siem-ca.pem"
  defaultNetstreamDriverCertFile="/etc/ssl/certs/client-cert.pem"
  defaultNetstreamDriverKeyFile="/etc/ssl/private/client-key.pem"
)

*.* action(
  type="omfwd"
  target="siem.corp.example.com"
  port="6514"
  protocol="tcp"
  StreamDriver="gtls"
  StreamDriverMode="1"
  StreamDriverAuthMode="x509/name"
  StreamDriverPermittedPeers="siem.corp.example.com"
)

The ELK/EFK Stack

Elasticsearch: distributed search and analytics engine. Indexes log data as JSON documents. Supports full-text search, aggregations, and time-series queries. Used to store and search logs.

Logstash / Fluentd / Fluent Bit: log collection and processing agents. Accept logs from syslog, file, beats, or API; parse structured fields (regex, Grok patterns); filter, transform, and route to output destinations. Fluent Bit is preferred for containers (low memory footprint).

Kibana: visualization and dashboard layer for Elasticsearch. Builds time-series graphs, geo maps, alert rules, and SIEM-style investigation workflows.

Loki + Grafana: Lightweight Alternative

Grafana Loki indexes only log labels (timestamp, host, service, level), not the full log text — making it much cheaper to store and index than Elasticsearch. Queries are fast for label filtering but slower for full-text search across all log content. Ideal for infrastructure logs where you know what you're looking for.

Structured Logging vs. Unstructured Syslog

Traditional syslog messages are unstructured text — "Failed password for alice from 203.0.113.5". Extracting "alice" and "203.0.113.5" requires fragile regex patterns that break when message formats change. Modern applications output structured JSON logs:

# Unstructured (legacy syslog):
May 24 10:00:00 server1 sshd[12345]: Failed password for alice from 203.0.113.5 port 49832 ssh2

# Structured (modern JSON log):
{
  "timestamp": "2026-05-24T10:00:00Z",
  "hostname": "server1",
  "service": "sshd",
  "event": "auth_failure",
  "username": "alice",
  "src_ip": "203.0.113.5",
  "src_port": 49832,
  "protocol": "ssh2"
}

Structured logs are directly indexable, queryable without regex, and consistent across software versions. The tradeoff: higher log volume (JSON overhead) and requires application-level changes. For new applications, always emit structured logs.

What Is a Flow?

A flow is a unidirectional sequence of packets sharing the same 5-tuple: source IP, destination IP, source port, destination port, IP protocol. NetFlow exports flow records when flows expire (TCP FIN/RST, or timeout). Records do not contain payload — just metadata about the conversation.

NetFlow Architecture

Exporter: the router or switch that observes traffic and creates flow records. Sends them via UDP/2055 to the collector.

Collector: receives and stores flow records. Examples: Ntopng, ElastiFlow, nfdump, Grafana + flow exporters.

Analyzer: queries the collector to answer questions: who is my top talker? what is the protocol breakdown? is there anomalous traffic to unusual destinations?

IPFIX: The Standard

IPFIX (IP Flow Information Export, RFC 7011) is the IETF standardization of NetFlow v9. It adds a flexible template system allowing exporters to define any set of fields per flow record — not just the standard 5-tuple + counters. IPFIX is the modern standard; NetFlow v5 and v9 remain common in legacy infrastructure.

Essential SNMP Hardening Steps

1. Disable SNMPv1 and SNMPv2c entirely if possible. Use SNMPv3 authPriv only.

2. Change default community strings on all devices. Use long random strings; treat them as passwords.

3. Restrict SNMP access to management IPs with ACLs. SNMP should only be reachable from the NMS IP(s), not from the internet or untrusted VLANs.

4. Disable SNMP SET if not needed for active management. Read-only for monitoring is sufficient.

5. Block port 161 and 162 at the internet border. No SNMP should be reachable from untrusted networks.

6. Audit SNMP community strings regularly. They often persist for years after deployment without rotation.

# iptables: allow SNMP only from NMS
iptables -A INPUT -p udp --dport 161 -s 10.0.0.5/32 -j ACCEPT
iptables -A INPUT -p udp --dport 161 -j DROP

# Cisco IOS: restrict SNMP to NMS IP
access-list 10 permit 10.0.0.5
access-list 10 deny any log
snmp-server community RANDOM_COMPLEX_STRING ro 10
no snmp-server community public
no snmp-server community private

Prometheus and SNMP Exporter

The SNMP Exporter bridges the old and new worlds: it queries devices via SNMP and exposes the results as Prometheus metrics. Prometheus scrapes the exporter on a schedule; Grafana visualizes the time-series data. The SNMP exporter uses a YAML configuration file generated from MIB definitions (generator tool).

# prometheus.yml scrape config
scrape_configs:
  - job_name: 'snmp'
    static_configs:
      - targets:
          - 192.168.1.1  # router to monitor
    metrics_path: /snmp
    params:
      auth: [snmpv3_auth]
      module: [if_mib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: snmp-exporter:9116

OpenTelemetry: The Future of Observability

OpenTelemetry (OTel) defines vendor-neutral APIs and protocols for traces, metrics, and logs. Network devices are beginning to export telemetry via gRPC/protobuf streaming (gNMI — gRPC Network Management Interface), which replaces SNMP polling with push-based high-frequency streaming. Cisco's Model-Driven Telemetry and Juniper's Junos Telemetry Interface stream operational data at sub-second intervals — not possible with poll-based SNMP.

YANG and gNMI: The Future of Device Management

YANG (RFC 6020) is a data modeling language that replaces MIBs. NETCONF and RESTCONF expose YANG-modeled device state and configuration via XML/JSON over SSH or HTTPS. gNMI (gRPC Network Management Interface) uses Protocol Buffers for compact binary encoding with streaming subscriptions. Together these form the foundation of programmable, model-driven network management.