Consumer Group Rebalancing — Causes and Mitigation

Direct Answer

Rebalancing is triggered when consumers join or leave the group (scaling, restarts, crashes) or when a consumer misses heartbeats. During rebalancing, all consumers in the group stop consuming. Minimize with static membership, session.timeout tuning, and cooperative rebalancing.

Code Example

A rebalance reassigns partitions among consumers in a group. While it runs, all consumers in the group pause — every rebalance is downtime. The triggers are predictable and most are avoidable.

spring:
  kafka:
    consumer:
      group-id: orders-processor
      properties:
        # Heartbeat / liveness — KEEP THESE TUNED
        heartbeat.interval.ms: 3000              # consumer sends heartbeat every 3s
        session.timeout.ms: 45000                # broker considers consumer dead after 45s of no heartbeat
        max.poll.interval.ms: 300000             # max gap between poll() calls (5 min)
        max.poll.records: 100                    # smaller batches → faster poll cycle
        # Cooperative rebalancing (lower impact than the legacy stop-the-world)
        partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor

@KafkaListener(topics = "orders", groupId = "orders-processor", concurrency = "4")
public void onOrder(OrderEvent ev, Acknowledgment ack) {
    // Long-running calls inside the listener push max.poll.interval.ms toward its limit
    // — offload to async work and ack quickly, OR raise max.poll.interval.ms.
    processor.process(ev);          // must complete inside max.poll.interval.ms
    ack.acknowledge();
}

// Rebalance lifecycle hooks
@Component
class RebalanceObserver implements ConsumerAwareRebalanceListener {

    @Override
    public void onPartitionsRevokedBeforeCommit(Consumer<?, ?> consumer,
                                                 Collection<TopicPartition> parts) {
        log.warn("Partitions REVOKED: {}", parts);   // commit in-flight work here
    }

    @Override
    public void onPartitionsAssigned(Consumer<?, ?> consumer,
                                     Collection<TopicPartition> parts) {
        log.info("Partitions ASSIGNED: {}", parts);
    }
}

• Cooperative Sticky Assignor is the modern default — only the moving partitions pause, not the entire group. Switch off the legacy RangeAssignor / StickyAssignor for new deployments.
• Static membership (group.instance.id) makes pod restarts a no-op for the rebalance protocol — the same pod ID rejoins and reclaims its partitions without triggering a full rebalance.
• Keep listener processing time well under max.poll.interval.ms. If a single message can take 5 minutes (e.g. PDF generation), offload to a thread pool and ack the offset early; track completion separately.
• A noisy rebalance metric (rebalance-rate-per-hour > a few) usually means GC, container CPU throttling, or processing time is the real problem — fix the root cause, not the timeouts.

Common Mistakes & Pitfalls

Not knowing that long processing times (exceeding max.poll.interval.ms) trigger rebalance