IDS and IPS

Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS) are security tools that monitor network traffic for malicious activity. An IDS detects and alerts; an IPS also takes automated action (blocks, resets connections, drops packets). Both are only as good as their rules, their tuning, and — crucially — the humans who respond to their alerts.

Understanding IDS/IPS requires understanding both the technology (how detection works, what rules look like, how placement affects visibility) and the operational reality (alert fatigue, false positives, tuning, analyst workflow). The technology is learnable in a day. The operational excellence takes years.

Network IDS/IPS (NIDS/NIPS)

NIDS: passively monitors network traffic. Receives traffic via a network tap or span port (mirrored copy). Since it receives copies of packets, it cannot block traffic — it can only detect and alert. If it generates a false positive, no traffic is disrupted. If it misses an attack, the attack succeeds.

NIPS: sits inline in the traffic path. Traffic must pass through the IPS before reaching its destination. The IPS can drop, modify, or reset packets. False positive = legitimate traffic blocked. True positive = attack stopped before reaching the target. The inline placement introduces latency and creates a potential network availability risk (if the IPS fails, does traffic continue?).

Host-Based IDS/IPS (HIDS/HIPS)

HIDS: an agent running on a specific host that monitors system calls, file access, process behavior, and log events. Can detect attacks that bypass network detection (e.g., an attack originating from a permitted connection, insider threats, local privilege escalation). Examples: OSSEC, Wazuh, Falco (containers), auditd.

HIPS: same as HIDS but can take blocking actions (kill processes, quarantine files, block specific system calls via seccomp). Modern endpoint protection platforms (CrowdStrike Falcon, SentinelOne) are HIPS with behavioral detection and ML.

Deployment Modes

Inline (IPS mode): traffic passes through. Can block. Single point of failure. Introduces latency. Requires bypass/fail-open capability for HA.

Passive tap (IDS mode): receives a copy of traffic. Cannot block. No single point of failure. Zero latency impact. Can send TCP RSTs to terminate detected sessions.

Span/mirror port: traffic is mirrored from a switch port to the IDS. No physical inline risk. But switch CPU overhead can cause dropped packets in mirroring, leading to missed detections.

The Detection Accuracy Matrix

IDS detection has four outcomes for any event:

— True Positive (TP): the system correctly identifies an attack. This is what we want.

— False Positive (FP): the system alerts on legitimate traffic. Alert fatigue, wasted analyst time, potential blocking of legitimate activity.

— True Negative (TN): the system correctly allows legitimate traffic. Silent success — most IDS actions.

— False Negative (FN): the system misses an attack. Silent failure — the attack succeeds undetected. The most dangerous outcome.

The tradeoff between FP and FN is the core challenge of IDS tuning. More sensitive detection → more TPs but also more FPs. Less sensitive → fewer FPs but more FNs. The right balance depends on the environment: a high-security financial network tolerates more FPs to minimize FNs; a high-availability e-commerce site might prioritize FP reduction to minimize disruption.

Suricata Rule Structure

Suricata rules follow the format: action protocol src_ip src_port direction dst_ip dst_port (options)

Key Suricata Features

Multi-threaded: each CPU core handles separate packet streams. Can process 10-40 Gbps on commodity hardware.

Protocol parsers: understands HTTP, DNS, TLS, SMTP, FTP, SSH at the application layer. Rules can match specific HTTP headers, DNS query types, TLS certificates.

File extraction: can extract files from HTTP, FTP, SMTP streams and submit to antivirus or sandbox.

Flowbits: track state across multiple packets — set a flag on packet 1, check it on packet 3. Enables multi-stage attack detection.

Lua scripting: write detection logic in Lua for complex conditions that rules can't express.

# Suricata YAML configuration (suricata.yaml)
vars:
  address-groups:
    HOME_NET: "[10.0.0.0/8,172.16.0.0/12,192.168.0.0/16]"
    EXTERNAL_NET: "!$HOME_NET"
    HTTP_SERVERS: "$HOME_NET"
    DNS_SERVERS: "$HOME_NET"

# Rule sources
default-rule-path: /etc/suricata/rules
rule-files:
  - suricata.rules
  - emerging-attack_response.rules
  - emerging-malware.rules
  - local.rules   # your custom rules

# Output: eve.json for SIEM integration
outputs:
  - eve-log:
      enabled: yes
      filetype: regular
      filename: /var/log/suricata/eve.json
      types:
        - alert
        - http
        - dns
        - tls

# AF_PACKET for high-performance capture
af-packet:
  - interface: eth0
    cluster-id: 99
    cluster-type: cluster_flow
    defrag: yes
    threads: auto

Snort Rule Format

# Snort/Suricata compatible rule format
# action proto src_ip src_port dir dst_ip dst_port (options)

# Example: detect Metasploit meterpreter HTTP reverse shell
alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (
  msg:"MALWARE-CNC Win.Trojan.Meterpreter HTTP variant outbound connection";
  flow:to_server,established;
  urilen:>600;
  http_uri;
  content:"/C_YYYYYYY";
  classtype:trojan-activity;
  sid:5001;
  rev:2;
)

# Detect cleartext FTP credentials
alert tcp any any -> any 21 (
  msg:"FTP Password Transmitted in Cleartext";
  flow:to_server,established;
  content:"PASS ";
  nocase;
  classtype:policy-violation;
  sid:5002;
  rev:1;
)

Emerging Threats Rule Sets

Emerging Threats (ET) provides free and commercial rule sets:

ET Open: free, community rules. Updated several times per day. Covers: malware, exploit kits, C2, policy, scanning, web attacks.

ET Pro: commercial rules with faster updates and broader coverage.

Rules are distributed as .rules files that are downloaded and referenced in the IDS configuration. Rule updates should be automated (daily or hourly for active threat environments).

Internet Edge (Outside Firewall)

Sees all internet traffic, including traffic the firewall will block. Provides insight into the threat landscape ("are we being scanned?", "are there attacks against a port we have open?"). High noise level — internet-facing sensors see enormous volumes of background scanning and probing. Useful for: threat intelligence, firewall policy validation, understanding your external attack surface.

DMZ Segment

Sees traffic that passed the perimeter firewall. Alerts here indicate either a firewall policy gap or an attack against explicitly permitted services. Essential for monitoring web servers, email gateways, and other internet-exposed hosts. Lower volume than edge sensors, higher signal.

Internal Segments

Sees east-west traffic between internal VLANs. This is where lateral movement, internal reconnaissance, and data exfiltration from compromised internal hosts are visible. Critical for detecting compromised internal systems. Most organizations have no internal IDS visibility — their IDS is only at the perimeter, which is why lateral movement goes undetected for months.

Tap vs. SPAN Port

Network tap: hardware device that passively copies all traffic. Transparent to the network. No packet loss. Preserves electrical signal independently. More expensive but more reliable.

SPAN port: switch feature that mirrors traffic from one or more ports to a dedicated monitor port. Free (software feature). Risk: SPAN ports can drop packets under high load, creating IDS blind spots. Also: some switches cannot SPAN their own management traffic.

Tuning to Reduce False Positives

1. Suppress rules for known-good traffic: add suppress rules that prevent alerting from specific source IPs or to specific destinations that you know generate FPs.

2. Threshold rules: require N occurrences in X seconds before alerting. Eliminates one-off matches that are almost always coincidental.

3. Pass rules: explicitly allow traffic that matches an attack signature but is known-good. Pass rules have higher priority than alert rules.

4. Context enrichment: tag alerts with asset context (is the destination a web server? is the source an internal server?). An XSS alert against a database server is different from one against a web application server.

# Suricata: suppress FP for known-good internal scanner
suppress gen_id 1, sig_id 1002, track by_src, ip 10.0.5.10/32
# This suppresses rule SID 1002 when source is the internal vulnerability scanner

# Threshold: alert only after 10 hits in 5 minutes
threshold gen_id 1, sig_id 1001, type limit, track by_src, count 10, seconds 300

# Pass rule (higher priority than alert rules)
pass tcp 10.0.3.5 any -> any 443 (msg:"Known good backup agent"; sid:9001;)

Fail-Open vs. Fail-Closed

An inline IPS must handle hardware or software failure. Two modes:

Fail-open: on IPS failure, traffic bypasses the IPS and flows normally. Network stays available. Security gap during failure.

Fail-closed: on IPS failure, all traffic is blocked. Network goes down. Maximum security, zero availability. Appropriate only for the most critical paths where the risk of a breach exceeds the risk of an outage.

Most production IPS deployments use fail-open with out-of-band alerting on IPS failure, so network operations know immediately when the IPS is bypassed.

IPS Modes for New Deployments

Best practice for deploying a new IPS:

1. Start in IDS mode (detection only). Collect 2-4 weeks of data. Analyze FP rate for each rule.

2. Tune high-FP rules: add suppressions, tune thresholds, disable rules that never TP.

3. Move high-confidence, low-FP rules to IPS mode first (known malware signatures, CVE-specific rules).

4. Gradually move more rules to IPS mode as confidence builds.

5. Keep anomaly-detection rules in IDS mode indefinitely — they have inherently high FP rates and should not block.

Packet Fragmentation

IP fragmentation splits a packet across multiple IP fragments. The IDS must reassemble them to inspect the complete payload. Attackers can overlap fragments (second fragment overlaps bytes from first) where different OS implementations handle the overlap differently. The IDS may reconstruct one version; the target OS another — the exploit is in the version the target sees.

TCP Segmentation

Attackers split signatures across TCP segments. "UNION SELECT" becomes "UNI" in one TCP segment and "ON SELECT" in the next. A naive IDS checks only individual segments. A stream-aware IDS must reassemble the TCP stream and then check — but must handle all the edge cases of TCP (retransmissions, out-of-order delivery, overlapping data).

Encoding and Obfuscation

URL encoding: UNION%20SELECT — the %20 is decoded to space by the web server but may not match a raw-bytes signature. Unicode encoding. Double encoding. Null bytes in payloads. Character case variations. IDS must normalize the traffic before matching signatures.

Protocol Compliance Exploitation

Sending protocol violations that the IDS drops but the target accepts. For example, HTTP requests with invalid content-length headers that confuse stream reassembly. The Ptacek-Newsham paper identified 15+ such techniques. Modern IDS engines address these with normalization passes before signature matching.

Encryption

TLS encryption completely hides payload from network IDS. Without TLS inspection, an IDS can only see: TLS SNI (hostname), certificate details, connection timing, traffic volume patterns. Behavioral detection on encrypted traffic (detecting C2 beaconing by timing patterns) is the primary mechanism for encrypted traffic analysis.

NDR vs. Traditional IDS

Traditional IDS: pattern-matching against known signatures. Fast, precise, zero-day blind.

NDR: machine learning on network traffic patterns, connection metadata, protocol timing, flow statistics. Detects anomalous behavior even from novel attacks. Higher FP rate, requires analyst investigation, but finds what signatures miss.

NDR Data Sources

NetFlow/IPFIX: connection metadata. Who talked to whom, how much, when. No payload. Enables lateral movement detection (internal hosts that suddenly start connecting to many other internal hosts).

Full packet capture (PCAP): complete packet contents for post-incident investigation. Extremely storage-intensive. Usually selective (capture only from suspicious hosts or segments).

DNS logs: all DNS queries and responses. DGA (domain generation algorithm) domain detection, DNS tunneling detection, C2 domain lookups.

TLS metadata: certificate details, cipher suites, JA3 fingerprints (TLS fingerprinting based on ClientHello parameters). Identifies malware that uses distinctive TLS configurations even in encrypted traffic.

JA3 TLS Fingerprinting

JA3 creates a fingerprint from the TLS ClientHello: SSL/TLS version + cipher suites + extensions + elliptic curves + elliptic curve point formats, all concatenated and MD5-hashed. Many malware families have consistent JA3 fingerprints regardless of the destination — the malware's TLS implementation is fingerprinted, not the content. Known malware JA3 fingerprints are published and can be used as IDS signatures for encrypted C2 traffic.

The Alert Pipeline

IDS (Suricata) → eve.json → log shipper (Filebeat/Fluent Bit) → SIEM (Elasticsearch/Splunk) → correlation rules → enrichment (GeoIP, threat intel, asset data) → prioritized alert queue → analyst workflow → incident response.

SIEM Correlation Rules

SIEM correlation rules aggregate multiple events into higher-confidence alerts:

# Elasticsearch SIEM detection rule (EQL)
# Detect: port scan followed by successful authentication within 10 minutes
sequence by source.ip with maxspan=10m
  [network where event.type=="connection" and destination.port < 1024
   and count(*) > 50]
  [authentication where event.outcome=="success"]

# This creates a high-priority alert that requires both:
# 1. A port scan (>50 connections to ports <1024) from a source IP
# 2. AND a successful auth from the same IP within 10 minutes
# The sequence is much more suspicious than either event alone

Threat Intelligence Integration

MISP, Recorded Future, VirusTotal, AlienVault OTX provide threat intelligence feeds: known malicious IPs, domains, file hashes, attack patterns (MITRE ATT&CK TTPs). IDS rules can reference these feeds. SIEM correlation can auto-classify alerts against TI.

AWS GuardDuty

AWS GuardDuty is a managed threat detection service that analyzes: VPC Flow Logs (connection metadata), DNS query logs (within VPC), CloudTrail API logs (AWS API calls), and EKS audit logs. GuardDuty uses ML and threat intelligence to detect: compromised EC2 instances communicating with known C2, unusual API calls (credential misuse), port scanning from EC2 instances, cryptocurrency mining.

AWS Network Firewall + Suricata

AWS Network Firewall uses Suricata rules for stateful deep packet inspection. Compatible with ET rule sets. Deployed as a managed VPC inspection endpoint. Inspect traffic between subnets, to internet, or from AWS Transit Gateway.

VPC Traffic Mirroring

AWS VPC Traffic Mirroring copies ENI traffic to a target (another ENI running an IDS appliance or a Network Load Balancer for scale). This is the cloud equivalent of a SPAN port — allows deploying a traditional IDS (Suricata, Zeek) in AWS without modifying application architecture.