Stream Processing: Kafka Streams vs Flink vs Spark Structured Streaming
Fast recall
Quick revision
Core answer
Knowledge of modern stream processing, event-time semantics, and framework trade-offs
Confusing stream processing with message queues; ignoring event-time vs processing-time; thinking Spark Streaming is real real-time
Discuss event time vs processing time, watermarks, exactly-once semantics, state backends, checkpointing, windowing types, and operational cost
Interview-ready guide
Interview answer
Stream processing transforms real-time data as it arrives: filter, enrich, aggregate, join. Unlike batch, streams are unbounded, so the framework must handle state, out-of-order events, failures, and time itself.
Three dominant frameworks in 2026.
Kafka Streams
- A Java library, not a cluster
- Embedded in your Spring Boot or plain Java app
- Reads from Kafka topics, writes to Kafka topics
- Supports windowed aggregations, joins, interactive queries
- State in embedded RocksDB; backed up to Kafka changelog topic
- Exactly-once via Kafka transactions
- Scales by running multiple app instances (consumer group)
- Perfect for Kafka-centric processing inside microservices
Apache Flink
- State-of-the-art streaming engine
- Dedicated cluster: JobManager plus TaskManagers
- Native event-time processing; watermarks are first-class
- True exactly-once via distributed snapshots (Chandy-Lamport)
- Sub-second to millisecond latency
- Rich windowing: tumbling, sliding, session, global, custom
- State backends: RocksDB on local disk, incremental checkpoints to S3
- Complex Event Processing (CEP) library for pattern matching
- Used by Uber, Netflix, Alibaba, LinkedIn at scale
Spark Structured Streaming
- Evolution of Spark Streaming (DStreams)
- Treats stream as unbounded table incrementally appended
- Micro-batch execution by default (seconds latency)
- Continuous mode for milliseconds but limited operations
- Rich ecosystem: Spark SQL, MLlib, batch integration
- Best when you already run Spark and want unified code
Key concepts to explain
Event time vs processing time
- Events have timestamps (event time); arrive at processor later (processing time)
- A 5-minute window on event time must wait for late events
- Watermarks mark 'all events up to time T have arrived'
- Flink has best event-time support
Watermarks
- Signal event-time progress
- Too low: drops correct events; too high: adds latency
- Tune based on observed lag distribution
State management
- Kafka Streams: RocksDB locally; changelog topic for recovery
- Flink: RocksDB, incremental checkpoints to S3; savepoints for upgrades
- Spark: state store, checkpoints to HDFS/S3
Exactly-once
- Within Kafka: Kafka Streams and Flink both support via transactions
- End-to-end: Flink with transactional sinks (Kafka, JDBC XA, Iceberg)
- External non-transactional systems: still need idempotent sinks or dedupe
When to pick each
Pick Kafka Streams when
- Data already in Kafka, output to Kafka
- Embedded in a microservice
- Moderate state (fits on local SSD)
- No cluster ops capacity; simple is good
- Typical: event enrichment, real-time aggregations in Spring Boot
Pick Flink when
- Complex stateful processing with huge state
- Event-time semantics critical; lots of out-of-order data
- Sub-second latency required
- Heterogeneous sources: Kafka, Kinesis, Pulsar, files
- Team has engineering capacity for cluster ops
- Typical: fraud detection, real-time analytics, CEP
Pick Spark Structured Streaming when
- Already running Spark for batch; unified code wins
- Latency needs are seconds, not milliseconds
- Data lake integration: Delta Lake, Iceberg, Hudi
- ML inference on streams via MLlib or MLflow
- Typical: streaming ETL into data lake
Operational trade-offs
- Kafka Streams: no extra infrastructure; just apps
- Flink: dedicated cluster with HA JobManager; non-trivial ops
- Spark: Spark cluster; DIY complexity
Latency regimes
- Sub-10ms: Flink
- 10–100ms: Kafka Streams, Flink
- Seconds: Spark micro-batch
Scaling models
- Kafka Streams: add app instances; consumer group assigns partitions
- Flink: change parallelism; rescaling via savepoint plus restore
- Spark: add executors; dynamic allocation supported
Windowing types
- Tumbling: non-overlapping fixed (every 5 minutes)
- Sliding: overlapping fixed (every 5 minutes, sliding every 1 minute)
- Session: gap-based (events within 30 seconds of each other)
- Global: single window with custom triggers
- Flink richest; Kafka Streams and Spark cover common cases
Unified batch and streaming
- Beam (Google's unified model) runs on Flink, Spark, Dataflow
- Spark natively unified batch and streaming
- Flink is streaming-first but supports batch
Managed options
- AWS Kinesis Data Analytics: managed Flink
- Confluent Cloud: managed Kafka + ksqlDB (SQL on Kafka Streams)
- Databricks: managed Spark
- Google Dataflow: managed Beam
The strategic answer
- Kafka Streams for embedded simple streaming in microservices
- Flink for serious stateful streaming at scale
- Spark Structured Streaming when already on Spark
- Event-time semantics and exactly-once are non-negotiable for correctness
- Operational cost often the deciding factor — managed options shift the calculus
Stream Processing: Kafka Streams vs Flink vs Spark Structured Streaming
Direct Answer
Three dominant real-time stream processors, each with different sweet spots: Kafka Streams is a Java library (no cluster, runs inside your app) — best for Kafka-centric, embeddable processing with stateful operations (joins, aggregations, windowing); limited to Kafka as source/sink. Apache Flink is a dedicated stream engine with true event-time processing, low latency (sub-second), rich state management, and checkpoint-based exactly-once — best for complex stateful pipelines, CEP, and heavy windowing. Spark Structured Streaming extends Spark's batch model to streams (micro-batches) — best when you already use Spark for batch, want unified code, and sub-second latency isn't required. Rule of thumb: Kafka Streams for simple Kafka → Kafka topologies embedded in apps; Flink for serious stateful streaming; Spark when you're already on Spark.
Common Mistakes & Pitfalls
Confusing stream processing with message queues; ignoring event-time vs processing-time; thinking Spark Streaming is real real-time
Check your understanding
Answer these without looking back at the article.
- How do watermarks work in Flink and what are the trade-offs in configuring them?
- What is a Flink savepoint and how does it differ from a checkpoint?
- How does Kafka Streams handle rebalancing when app instances come and go?
- When would you use ksqlDB vs Kafka Streams directly?
Related interview prep categories
2Want a guided curriculum order?
These modules are also sequenced in our flagship backend interview track — open it when you prefer an ordered path instead of browsing by topic.