KEDA: Event-Driven Autoscaling for Kafka Consumers and Queue Workers

Intermediate·3 min read·

Fast recall

Quick revision

Core answer

KEDA (Kubernetes Event-Driven Autoscaler) scales workloads based on external event sources — Kafka consumer lag, SQS queue depth, RabbitMQ messages, Prometheus queries, CRON expressions, Redis streams, and 60+ built-in scalers. It can scale from zero to N and back to zero when idle — something HPA cannot do because HPA requires minReplicas >= 1. KEDA internally creates an HPA under the hood for the scaling-from-1-to-N range, plus its own controller for 0↔1. Use KEDA for consumers and batch workers; keep HPA for request-driven services.

Interview Answer3 min

KEDA is Kubernetes Event-Driven Autoscaling — it scales workloads based on external event sources like Kafka consumer lag, SQS queue depth, RabbitMQ messages, or Prometheus queries. It composes with HPA rather than replacing it.

Why it exists

HPA requires resource utilization to be the scaling signal and a minimum of 1 replica always running.
A Kafka consumer at 15% CPU with lag climbing to 100,000 messages is exactly the scenario HPA cannot handle — CPU never crosses threshold, lag keeps growing.
KEDA makes queue depth and consumer lag first-class scaling metrics, and scales from zero for idle workloads.

Architecture

KEDA runs three components: operator, metrics adapter, admission webhooks.
A ScaledObject CRD points at a Deployment and declares triggers — Kafka lag, SQS depth, whatever.
KEDA owns the 0↔1 transition — when activation threshold is crossed, it scales the Deployment from zero to one.
For 1→N scaling, KEDA auto-generates an HPA that reads metrics from KEDA's metrics adapter; developers don't touch this HPA, edits happen on the ScaledObject.

ScaledObject vs ScaledJob

ScaledObject wraps a long-running Deployment — right choice for Kafka consumers, HTTP workers, anything that runs continuously and handles many messages per replica.
ScaledJob creates a Job per event batch — right for per-message isolated work like ML inference, image processing, one-off ETL; each message spawns a fresh pod that exits when done.

Scale-to-zero considerations

Cold start latency is the main cost — 30 to 60 seconds for a Spring Boot JVM; for latency-critical paths keep minReplicaCount at 1.
Cooldown period controls flapping — 5 minutes of idle before scaling back to zero is a good default.
For bursty workloads like nightly ETL or batch ingestion, scale-to-zero saves 40 to 80 percent on compute vs leaving one replica running 24/7.

Production gotchas

Kafka consumer groups rebalance on every scale event — tune session.timeout.ms and handle rebalance listener events gracefully.
Graceful shutdown is mandatory — KEDA sends SIGTERM, the consumer must commit in-flight messages before exiting or messages are re-processed on the next scale-up.
PDB with minAvailable blocks scale-to-zero — use maxUnavailable instead for PDB on KEDA-managed Deployments.
Don't edit the auto-generated HPA; it is owned by KEDA and reconciles on every change; edit the ScaledObject instead.

Interview-ready guide

Interview answer

Why it exists

HPA requires resource utilization to be the scaling signal and a minimum of 1 replica always running.
A Kafka consumer at 15% CPU with lag climbing to 100,000 messages is exactly the scenario HPA cannot handle — CPU never crosses threshold, lag keeps growing.
KEDA makes queue depth and consumer lag first-class scaling metrics, and scales from zero for idle workloads.

Architecture

KEDA runs three components: operator, metrics adapter, admission webhooks.
A ScaledObject CRD points at a Deployment and declares triggers — Kafka lag, SQS depth, whatever.
KEDA owns the 0↔1 transition — when activation threshold is crossed, it scales the Deployment from zero to one.
For 1→N scaling, KEDA auto-generates an HPA that reads metrics from KEDA's metrics adapter; developers don't touch this HPA, edits happen on the ScaledObject.

ScaledObject vs ScaledJob

ScaledObject wraps a long-running Deployment — right choice for Kafka consumers, HTTP workers, anything that runs continuously and handles many messages per replica.
ScaledJob creates a Job per event batch — right for per-message isolated work like ML inference, image processing, one-off ETL; each message spawns a fresh pod that exits when done.

Scale-to-zero considerations

Cold start latency is the main cost — 30 to 60 seconds for a Spring Boot JVM; for latency-critical paths keep minReplicaCount at 1.
Cooldown period controls flapping — 5 minutes of idle before scaling back to zero is a good default.
For bursty workloads like nightly ETL or batch ingestion, scale-to-zero saves 40 to 80 percent on compute vs leaving one replica running 24/7.

Production gotchas

Kafka consumer groups rebalance on every scale event — tune session.timeout.ms and handle rebalance listener events gracefully.
Graceful shutdown is mandatory — KEDA sends SIGTERM, the consumer must commit in-flight messages before exiting or messages are re-processed on the next scale-up.
PDB with minAvailable blocks scale-to-zero — use maxUnavailable instead for PDB on KEDA-managed Deployments.
Don't edit the auto-generated HPA; it is owned by KEDA and reconciles on every change; edit the ScaledObject instead.

03 · Deep dive