Managing Consumer Group Rebalancing for Sensor Topics #

A rebalance is Kafka redistributing a topic’s partitions across the live members of a consumer group, and left at its defaults it is a stop-the-world event: every consumer pauses, drops its partitions, and waits for a new assignment before processing resumes. For an environmental telemetry group that is mid-window on a five-minute PM2.5 average, an ill-timed rebalance means a stalled dashboard and, without idempotent output, a duplicated result. The cure is configuration, not code: the cooperative-sticky assignor, static membership, and timeouts matched to reality turn rebalancing from a disruptive reshuffle into a quiet, incremental hand-off. This page is the configuration companion to building an at-least-once faust agent for sensor streams, and both sit under Faust and Kafka consumer patterns for sensor streams.

What Triggers a Rebalance and Why It Hurts #

A consumer group rebalances whenever its membership or the topic’s partition count changes: an instance joins, an instance leaves, an instance is presumed dead because it stopped heartbeating, or a partition is added. In an autoscaled Kubernetes deployment these events are routine — a Horizontal Pod Autoscaler adds a pod during a smoke event, a node is drained for maintenance, a rolling deploy cycles every pod in turn. Each of those, under the legacy eager assignors, revokes all partitions from all consumers and reassigns from scratch.

The damage is threefold. First, throughput stops: during the revoke-and-reassign no consumer processes anything, so lag spikes across the whole group. Second, in-flight state is disrupted: an agent accumulating a windowed aggregate in a faust Table has its partition yanked, and the new owner must rebuild that state from the changelog before it can emit. Third, uncommitted offsets are replayed: any events processed since the last commit on a moved partition are redelivered to the new owner, and without idempotent output those become duplicate rows. The goal of every setting below is to shrink or eliminate one of these three costs.

For sensor networks the frequency of these triggers is easy to underestimate. A regional deployment that autoscales on consumer lag will add and remove pods several times a day as diurnal traffic rises and falls; a weekly rolling deploy cycles every pod; spot-instance reclamation on cloud infrastructure can evict a node with two minutes’ notice. Each is a membership change, and a group that treats every one of them as a stop-the-world reshuffle will spend a meaningful fraction of its uptime paused. The distinction that matters is not whether rebalances happen — they always will — but whether each one moves a handful of partitions quietly or freezes the entire group. That distinction is entirely a matter of configuration.

The Three Levers That Fix It #

Cooperative-sticky assignment changes the rebalance protocol from eager to incremental. Instead of every consumer revoking everything, only the specific partitions that must move to rebalance the group are revoked; every other partition keeps its owner and keeps processing. A single pod joining a 12-partition, 3-instance group moves perhaps 3 partitions rather than all 12, and the other 9 never pause.

Static membership removes the most common rebalance trigger entirely. By assigning each instance a stable group.instance.id, a consumer that disappears and returns within session.timeout.ms is recognized as the same member and simply reclaims its old partitions — no reassignment occurs. This is what makes a rolling deploy invisible to the group: each pod restarts inside the timeout window and picks up exactly where it left off.

Right-sized timeouts stop false rebalances. session.timeout.ms is how long the broker waits for a heartbeat before declaring a consumer dead; max.poll.interval.ms is how long a consumer may spend processing a batch before the broker assumes it hung. Set them below your real-world jitter and processing time and you get phantom evictions — a healthy consumer declared dead, evicted, then readmitted, each cycle a full rebalance. Set them sensibly and only genuine failures trigger reassignment.

# python 3.11 · faust-streaming==0.11.1 · confluent-kafka==2.3.0 · pydantic==2.6.4
import os
import faust

app = faust.App(
    "env-telemetry-consumer",
    broker="kafka://broker:9092",
    processing_guarantee="at_least_once",
    # Incremental rebalancing: only moving partitions are revoked.
    broker_client_id="env-consumer",
    consumer_group_instance_id=os.environ["POD_NAME"],   # static membership
    # Underlying confluent-kafka consumer settings.
    consumer_kwargs={
        "partition.assignment.strategy": "cooperative-sticky",
        "session.timeout.ms": 30000,
        "heartbeat.interval.ms": 10000,     # ~1/3 of session.timeout
        "max.poll.interval.ms": 300000,     # above worst-case batch time
    },
)

Deriving consumer_group_instance_id from a stable pod identity (a StatefulSet ordinal, or POD_NAME injected via the downward API) is what makes static membership durable across restarts. A random UUID would defeat it — the “same member” recognition depends on the id surviving the restart.

The three levers reinforce one another rather than substituting for each other. Static membership prevents the avoidable rebalances — deploys and restarts — from happening at all. The cooperative-sticky assignor makes the unavoidable ones — a genuine scale event or node failure — incremental rather than global. And correctly sized timeouts ensure that transient network hiccups are not misread as failures in the first place, so the assignor is only invoked when something has truly changed. Configure only one and the other two gaps remain: static membership alone still suffers a stop-the-world reshuffle on a real scale-out, while cooperative-sticky alone still reassigns needlessly on every rolling deploy. The robust posture applies all three together.

Consumer Configuration Reference #

Every setting below directly shapes rebalance behavior. The columns give starting points for three consumer-infrastructure profiles; note these describe where the consumers run, not where the sensors are.

heartbeat.interval.ms should sit near one-third of session.timeout.ms. Heartbeats are how a consumer proves liveness between polls; at one-third the timeout, a consumer can miss two heartbeats to transient jitter and still not be evicted. Set it too close to the session timeout and a single delayed heartbeat kills the member.

max.poll.interval.ms is a correctness bound, not a performance knob. It must be strictly larger than the longest time any batch can legitimately take to process. If a rare large batch or a slow sink write can take four minutes, a 300-second (5-minute) interval is safe; a 60-second one guarantees periodic false evictions. When you find yourself raising it past ten minutes, the real problem is a blocking call — fix that instead.

rebalance.timeout.ms guards rolling deploys. It is how long the group leader waits for all members to rejoin during a rebalance. Set it comfortably above your pod startup time so a slow-booting replacement is waited for rather than excluded, which would otherwise trigger a second rebalance when it finally arrives.

Verification and Testing #

Rebalance behavior is only trustworthy once observed under the events that trigger it. Exercise each path deliberately.

# Drive a controlled scale event and assert no duplicate windows land.
import asyncpg
import pytest


@pytest.mark.asyncio
async def test_no_duplicate_windows_across_rebalance(pg_pool: asyncpg.Pool):
    """
    Precondition: publish a fixed set of readings, scale the consumer group
    from 2 -> 3 instances mid-stream, then assert the windowed output table
    holds exactly one row per (zone_id, window_start).
    """
    dupes = await pg_pool.fetch(
        """
        SELECT zone_id, window_start, COUNT(*) AS n
        FROM windowed_pm25
        GROUP BY zone_id, window_start
        HAVING COUNT(*) > 1
        """
    )
    assert dupes == [], f"rebalance produced duplicate windows: {dupes}"

Beyond the automated check, run three manual drills. Scale-out: add an instance under steady load and confirm from the broker’s consumer-group metadata that only a minority of partitions changed owner (cooperative-sticky working). Rolling restart: cycle every pod one at a time and confirm the group generation id increments minimally, ideally not at all if restarts land within session.timeout.ms (static membership working). Kill-and-wait: SIGKILL a pod and confirm reassignment fires only after session.timeout.ms elapses, not instantly — proving liveness detection is timeout-driven, not eager.

Expected healthy signal: on a 12-partition group of 3 instances, a scale-out to 4 should move roughly 3 partitions and leave 9 undisturbed, with a lag blip measured in seconds, not the whole-group stall of an eager assignor.

Gotchas #

Mixing assignors across a rolling upgrade breaks the group. All members must agree on the assignment strategy. If you deploy cooperative-sticky to some pods while others still run the range assignor, the group cannot form a valid assignment and rebalances in a loop. Roll out the assignor change in a single coordinated deploy, or use the documented two-step upgrade path where members advertise both strategies transiently.

Static membership without a stable id is worse than none. If group.instance.id is derived from something ephemeral — a container id, a random UUID — each restart presents a new static member while the old one lingers until session.timeout.ms expires. The group briefly believes it has extra members and its partitions sit unowned. Bind the id to a durable identity like a StatefulSet ordinal.

A blocked poll loop defeats every setting here. The elegant assignor and generous timeouts are worthless if a synchronous sink write freezes the consumer past max.poll.interval.ms. The consumer is evicted, rejoins, is evicted again — a rebalance storm no configuration will calm. Keep agents fully async, exactly as the at-least-once agent build does. This is the single most common root cause of chronic rebalancing.

Adding partitions reshuffles keys mid-flight. Increasing a topic’s partition count is itself a rebalance trigger, and worse, it remaps the key-to-partition hash so a device’s future readings may land on a different partition than its history. Any per-key state keyed on partition locality is disrupted. Size partitions for long-term peak at creation time rather than growing them on a live sensor topic.

Related #

• Faust and Kafka Consumer Patterns for Sensor Streams — parent overview of agents, Tables, and partition assignment
• Building an At-Least-Once Faust Agent for MQTT Sensor Streams — the agent whose idempotent sink makes any residual rebalance replay harmless