Kafka Consumer Lag Monitoring and Alerting

Direct Answer

consumer lag (difference between latest offset and consumer position) indicates how far behind consumers are. Monitor via Kafka AdminClient, Confluent Control Center, or export to Prometheus with kafka-lag-exporter. Alert when lag consistently grows.

Code Example

1) Maven dependencies:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

2) Bind Kafka client metrics into Micrometer (Spring Boot does this automatically when actuator + a Kafka client are on the classpath, but you can wire it explicitly):

@Configuration
public class KafkaMetricsConfig {

    // Spring Kafka auto-registers `KafkaClientMetrics` per consumer/producer factory.
    // For raw clients, bind manually:
    @Bean
    public KafkaClientMetrics consumerKafkaMetrics(ConsumerFactory<?, ?> cf,
                                                   MeterRegistry registry) {
        // The factory exposes its underlying KafkaConsumer's metrics() map;
        // KafkaClientMetrics translates them to Micrometer gauges.
        Consumer<?, ?> probe = cf.createConsumer();
        KafkaClientMetrics metrics = new KafkaClientMetrics(probe);
        metrics.bindTo(registry);
        return metrics;
    }
}

3) application.yml — expose Prometheus scrape endpoint:

management:
  endpoints:
    web:
      exposure:
        include: health, info, prometheus, metrics
  metrics:
    tags:
      application: ${spring.application.name}
      environment: ${ENV:dev}

4) The metrics that matter (exposed at /actuator/prometheus):

• kafka_consumer_fetch_manager_records_lag — current lag for one (client-id, topic, partition).
• kafka_consumer_fetch_manager_records_lag_max — highest lag across all assigned partitions for this consumer; the headline metric for alerting.
• kafka_consumer_fetch_manager_records_consumed_rate — throughput; useful to distinguish a slow consumer from one that simply has nothing to read.
• kafka_consumer_coordinator_rebalance_total / kafka_consumer_coordinator_rebalance_latency_avg — rebalance frequency and pause duration.

5) Prometheus alert rules (kafka-consumer-alerts.yml):

groups:
  - name: kafka-consumer-lag
    interval: 30s
    rules:
      # Headline alert: any consumer's max-lag is high AND still growing.
      - alert: KafkaConsumerLagHighAndGrowing
        expr: |
          kafka_consumer_fetch_manager_records_lag_max > 10000
          and
          deriv(kafka_consumer_fetch_manager_records_lag_max[5m]) > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'Consumer {{ $labels.client_id }} lag is {{ $value }} and growing'
          description: |
            Group {{ $labels.client_id }} on topic {{ $labels.topic }}
            partition {{ $labels.partition }} has lag of {{ $value }} messages
            and the trend over 5m is positive. Likely a slow consumer or
            insufficient concurrency.

      # Spike absorbed: lag is high but actively shrinking — informational.
      - alert: KafkaConsumerLagShrinking
        expr: |
          kafka_consumer_fetch_manager_records_lag_max > 50000
          and
          deriv(kafka_consumer_fetch_manager_records_lag_max[5m]) < 0
        for: 5m
        labels: { severity: info }

      # Rebalance churn — usually means session.timeout.ms is too tight
      # or processing exceeds max.poll.interval.ms.
      - alert: KafkaConsumerRebalanceChurn
        expr: increase(kafka_consumer_coordinator_rebalance_total[10m]) > 5
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: '{{ $labels.client_id }} rebalanced {{ $value }} times in 10m'

      # Consumer is alive but processing nothing — possible deadlock or
      # listener exception in a tight loop.
      - alert: KafkaConsumerStalled
        expr: |
          rate(kafka_consumer_fetch_manager_records_consumed_total[5m]) == 0
          and
          kafka_consumer_fetch_manager_records_lag_max > 0
        for: 5m
        labels: { severity: critical }

Critical points:

• Alert on lag trend (deriv) + threshold, not absolute lag alone — a steady lag of 50K during a known burst is fine; a growing lag of 5K is a problem.
• Tag metrics with application, environment, client_id, topic, partition so dashboards can break down by team/topic and per-partition skew shows up.
• For Kubernetes, hook the same kafka_consumer_fetch_manager_records_lag_max metric into KEDA's kafka scaler to autoscale consumer pods on lag.
• Always also monitor producer-side metrics (kafka_producer_record_send_rate, kafka_producer_outgoing_byte_rate) — a 'consumer lag' alert often turns out to be a producer-side spike.

Common Mistakes & Pitfalls

Alerting on absolute lag value instead of lag trend — a constant lag is acceptable, growing lag is not