Stream Processing with Apache Flink
Understand Flink's event-time model, watermarks, stateful windows, checkpointing, and Kafka source-to-sink pipelines through a worked streaming example.
Chapter 01
Where Flink Fits
Prerequisites: topics, partitions, consumer groups, and the difference between event time and processing time.
Learning objective: design a Kafka-to-Flink job that handles event time, bounded state, checkpoints, and an explicit output contract.
Kafka stores and distributes an ordered log. It does not calculate a continuously updated answer by itself. Apache Flink is a distributed processing engine that reads events, maintains state, handles event-time windows, and emits derived results. Pairing Kafka with Flink is useful when an order, delivery, or payment stream must be aggregated or joined as events arrive rather than in a later batch.
Imagine a delivery platform wants a five-minute total of order value by region. Kafka topics carry order events; Flink reads them, filters invalid records from the paid-order stream, groups them by region, updates a windowed total, and writes results to a separate topic for dashboards or downstream services. The stream is still unbounded: the job stays running, and its state evolves as new records and late records arrive.
Kafka Streams or Flink?
Chapter 02
Time, Windows, and Watermarks
Event time is not the time your code sees a record
Processing time is when a task handles a record. Event time is when the business event happened, usually carried in the event itself. Networks, mobile devices, retries, and upstream queues reorder arrival: a phone may upload a delivery scan after connectivity returns. If the question is “orders placed between 10:00 and 10:05,” processing-time windows can place a late-arriving 10:03 event in the wrong window. Event-time processing uses the event timestamp.
A watermark is the engine's progress estimate for event time. It does not prove that no older event will ever arrive. A bounded-out-of-orderness strategy says, in effect, “expect records to be up to this far behind.” When the watermark passes a window boundary, Flink can produce the result; later records may be dropped or handled by an allowed-lateness strategy. Watermark delay is a latency-versus-completeness tradeoff that must be chosen from observed source behavior.
| Concept | Question it answers | Example choice |
|---|---|---|
| Event time | When did the business event happen? | Timestamp attached by the order service. |
| Processing time | When did this task handle the event? | Useful for operational latency, not historical event grouping. |
| Watermark | How far has event-time progress advanced? | Allow bounded disorder based on measured late arrivals. |
| Window | Which events form one result? | Five-minute tumbling revenue per region. |
A watermark is not a guarantee
Chapter 03
A Kafka-to-Flink Job
The example uses line-delimited order records: region,amountCents,eventTimeEpochMs.
Use a dedicated input topic and consumer group for the job. The source should preserve the event timestamp so Flink can assign timestamps before windowing. The example uses tuples to keep parsing visible; production events should use a versioned schema and a serializer such as Avro or Protobuf where appropriate.
Build this example with the Kafka source and sink connector artifacts compatible with the Flink release you deploy; connector artifacts are separate from some Flink distributions. The parser assumes clean CSV for teaching. A production job should use a schema-aware format and route malformed records to a recoverable side output. The sink is explicitly at-least-once, so downstream consumers must handle retries idempotently.
Do not confuse output aggregation with an upsert
Chapter 04
State and Checkpoints Make Recovery Possible
State is the information a job keeps between records: a running total for each key, a deduplication set, or a partial join buffer. Flink partitions state with the computation so tasks can process many keys in parallel. State grows with key cardinality and retention, so a seemingly small aggregation can become a storage problem when keys never expire.
Checkpointing periodically captures a consistent snapshot of operator state and source progress. After a failure, Flink can restore a checkpoint and resume from corresponding source positions. For Kafka, this works best when the job has stable source identity, a configured checkpoint storage location, and a sink whose delivery semantics match the recovery model. Checkpoints are not backups of every external system and do not automatically make arbitrary side effects exactly once.
The numbers above are illustrative starting points, not universal tuning values. Measure checkpoint duration, alignment time, state size, backpressure, consumer lag, and recovery time with representative data. Tune the interval against the freshness objective and the cost of replay.
Chapter 05
Choose the Right Output and Delivery Contract
A stream job may write an enriched event, a continuously updated aggregate, or a correction event. Those are different contracts. For an order pipeline, one topic can preserve normalized order facts while a separate topic carries the region-and-window aggregate. If consumers require upsert or compaction semantics, also set the Kafka record key to the region/window identity. Make the key, schema, and update semantics explicit so readers do not sum replacement values twice.
| Design decision | Why it matters | Production check |
|---|---|---|
| Input offset policy | Earliest replays retained history; latest starts from new records. | Document bootstrap and recovery behavior. |
| Event-time delay | More delay can capture late events but increases result latency. | Use observed lateness and business tolerance. |
| State retention | Longer state supports deduplication and joins but costs more. | Bound state and test cleanup behavior. |
| Sink semantics | At-least-once can produce duplicates after retries. | Make consumers idempotent or configure transactional output. |
Exactly-once has a boundary
Chapter 06
Common Mistakes and Debugging
- ✓Using processing time for a business question defined by when the event occurred; fix it by carrying and validating event timestamps.
- ✓Setting an arbitrary watermark delay; inspect lateness distributions and test the tradeoff with replayed data.
- ✓Keeping unbounded state for every key; define state TTL or window boundaries and monitor state growth.
- ✓Treating checkpoints as a substitute for sink idempotency; test duplicate behavior after a forced restart.
- ✓Ignoring backpressure; inspect busy time, checkpoint health, source lag, and sink throughput together.
Chapter 07
Practice and Takeaways
Use a local Kafka cluster or a disposable test environment. Keep real customer data out of sample topics; synthetic orders are enough to test parsing, windows, restart behavior, and output contracts.
- ✓Create a synthetic order stream with event timestamps arriving out of order. Explain which five-minute window receives each event.
- ✓Change the watermark delay and compare result latency with the number of late events.
- ✓Add a malformed record path and record the topic, partition, offset, and parse error for each rejected event.
- ✓Force a job restart and verify that outputs are not double-counted by a downstream consumer.
- ✓Compare the same simple aggregation in Kafka Streams and Flink. Write down operational and feature tradeoffs for your workload.
🎯 Key Takeaways
- ✓Kafka stores and distributes event logs; Flink executes distributed, stateful computations over them.
- ✓Event time, watermarks, and windows define how late and reordered events affect results.
- ✓Stateful jobs need bounded state, checkpoints, and a tested recovery path.
- ✓Output keys and update semantics are part of the public data contract.
- ✓Exactly-once claims must name the source, state, sink, and external-effect boundaries.
HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
