Choosing Partition Keys for Sensor Topics

The partition key is the only ordering guarantee Kafka gives you, and it is easy to spend it on the wrong thing. Kafka promises that messages with the same key arrive in the order they were produced; it promises nothing across keys. So the question “what should the key be?” is really “what sequence must never be reordered?” For environmental telemetry the answer is almost always the sensor — and almost every alternative key that seems more natural, especially region or metric, produces a partition distribution that stalls one consumer while the rest idle. This guide works through that choice, how to measure whether you got it right, and what to do when a key genuinely outgrows a single partition. It refines the Kafka stream synchronization workflows stage.

Ordering Is Per Key, Skew Is Per Key

Two properties follow from the same mechanism, and they pull in opposite directions.

Ordering is what you want: every reading from sensor eui-70b3d5 lands on one partition, is consumed by one worker, and arrives in production order. A rolling baseline, a last-value cache, a gap detector — all of them are correct only under that guarantee.

Skew is what you get: the distribution of messages across partitions is the distribution of your key’s cardinality and frequency. A key with 500 roughly-equal values across 12 partitions spreads evenly. A key with 6 values, one of which carries 40% of the traffic, does not — and no amount of consumer scaling fixes it, because that one value’s traffic is pinned to one partition by definition.

This is why region is such a tempting and such a poor key. It is how humans think about the network, it is what reports group by, and it has a cardinality of maybe eight with a heavy skew toward the dense urban region. Keying on it hands 40% of the load to one worker permanently. The grouping you want for reporting is not the grouping you need for partitioning, and conflating them is the single most common Kafka mistake in sensor pipelines.

Partition load under four candidate keys Grouped bar chart of the skew ratio — busiest partition divided by the median — for keys on device identifier, site, region and metric, showing that low-cardinality keys concentrate load on one worker. Partition load under four candidate keys 0 2.5 5 7.5 1.1 device_id 2.4 site_id 6.8 region 4.1 metric skew ratio
Above about 2.0 the busiest consumer is doing twice the median work; above 4.0 adding consumers achieves nothing, because the bottleneck is one partition.

Production-Ready Implementation

The default key is the device identifier, encoded consistently:

# python 3.11 · confluent-kafka==2.4.0
from confluent_kafka import Producer

def partition_key(reading: dict) -> bytes:
    """Sensor identity is the ordering unit — encode it once, here, and nowhere else."""
    return reading["device_id"].encode("utf-8")


def publish(producer: Producer, topic: str, reading: dict) -> None:
    producer.produce(
        topic=topic,
        key=partition_key(reading),
        value=serialize(reading),
        on_delivery=_on_delivery,
    )

Encoding the key in exactly one function matters more than it looks. A producer that sends device_id.encode() in one path and str(device_id).encode("utf-8") in another will usually agree — until a device identifier arrives as an integer, at which point the same sensor hashes to two partitions and its ordering silently breaks.

Measuring skew is a query against the topic’s own metadata rather than a guess:

from collections import Counter
from confluent_kafka import Consumer, TopicPartition


def partition_volumes(consumer: Consumer, topic: str) -> dict[int, int]:
    """Messages per partition, from the broker's watermarks — no consumption needed."""
    md = consumer.list_topics(topic, timeout=10).topics[topic]
    volumes = {}
    for pid in md.partitions:
        low, high = consumer.get_watermark_offsets(TopicPartition(topic, pid), timeout=10)
        volumes[pid] = high - low
    return volumes


def skew_ratio(volumes: dict[int, int]) -> float:
    """Busiest partition divided by the median. Above ~2.0, one worker is carrying the topic."""
    counts = sorted(volumes.values())
    median = counts[len(counts) // 2] or 1
    return max(counts) / median

A skew ratio near 1.0 is a healthy key. Above 2.0, the busiest consumer is doing twice the median work and will be the first to fall behind under load; above 4.0, adding consumers achieves nothing because the bottleneck is a single partition.

When one key genuinely is too hot — a reference-grade instrument reporting at 10 Hz while the fleet reports every minute — salting spreads it across partitions at the cost of that key’s global ordering:

def salted_key(reading: dict, hot: set[str], fanout: int = 4) -> bytes:
    """Spread a hot sensor across `fanout` partitions.

    Ordering is preserved within each salted key, not across them, so only do
    this for sensors whose downstream processing is order-independent — counts,
    sums, and per-window aggregates are fine; a rolling baseline is not.
    """
    device_id = reading["device_id"]
    if device_id not in hot:
        return device_id.encode("utf-8")
    bucket = hash((device_id, reading["observed_at"])) % fanout
    return f"{device_id}#{bucket}".encode("utf-8")

The docstring carries the constraint that makes this safe or unsafe. Salting is correct for aggregations that are commutative and wrong for anything that walks a sensor’s history in order — which includes most of sensor drift correction.

What each key preserves and what it costs Matrix of five keying strategies against the ordering guarantee they provide, their typical cardinality, and the downstream operations that depend on that ordering. What each key preserves and what it costs Ordering Cardinality Safe for device_id per sensor 100–50 000 everything site_id per site 10–500 site rollups region per region 3–20 nothing — skewed null (round robin) none n/a order-independent sinks device_id#salt within a bucket key × fanout sums and counts only
Salting buys throughput for a genuinely hot sensor and gives up its global ordering — fine for counts, wrong for a rolling baseline.

Parameter Tuning Guide

Candidate key Cardinality Ordering preserved Typical skew Verdict
device_id 100–50 000 per sensor 1.0–1.2 the default
site_id 10–500 per site 1.5–3.0 acceptable if sites are balanced
region 3–20 per region 3.0–8.0 never — report by it, do not key by it
metric 4–10 per metric 2.0–5.0 never — one metric always dominates
null (round-robin) n/a none 1.0 only for order-independent sinks
device_id#salt key × fanout within a salt bucket 1.0–1.2 for a genuinely hot device only
Fleet size Partitions Consumers Headroom
Under 500 sensors 6 2–3 3 years of growth
500–5 000 12 4–6 comfortable
5 000–50 000 24–48 8–16 watch rebalance duration
Over 50 000 48–96 16–32 consider a topic per metric family
What changing the partition count does to existing keys Flow diagram of a partition count increase: the default partitioner hashes the key modulo the count, so roughly half of all keys map to a different partition and one sensor briefly exists on two. What changing the partition count does to existing keys 12 partitions hash(key) % 12 steady state Increase to 24 ~50 % of keys re-map ordering breaks Transition one sensor on two partitions bounded window 24 partitions hash(key) % 24 steady state again Partitions are cheap to add and impossible to remove, so size for years of growth rather than repartitioning annually.
Idempotent writes and event-time windows absorb the transition; a rolling baseline computed during it does not.

Verification and Testing

The property to assert is that one sensor’s readings never span partitions. It is a two-line test and it catches every encoding inconsistency:

# python 3.11 · confluent-kafka==2.4.0 · pytest==8.2.0
def test_one_sensor_maps_to_exactly_one_partition(producer, consumer, topic):
    readings = [
        {"device_id": "eui-70b3d5", "observed_at": f"2026-08-01T10:{m:02d}:00Z", "value": 20 + m}
        for m in range(30)
    ]
    for r in readings:
        publish(producer, topic, r)
    producer.flush(10)

    partitions = {m.partition() for m in drain(consumer, topic, expect=len(readings))}
    assert len(partitions) == 1


def test_skew_stays_within_budget(consumer, topic):
    assert skew_ratio(partition_volumes(consumer, topic)) < 2.0

Run the second one against production periodically rather than only in CI. Skew is not a static property: it changes as the fleet grows, as sensors are added unevenly across sites, and dramatically during an incident when one area’s sensors start reporting at their maximum rate. The hot-cell handling in spatial windowing is the same problem one stage downstream.

Gotchas

Changing the partition count re-maps every key. The default partitioner is hash(key) % partitions, so going from 12 to 24 partitions moves roughly half of all keys. Do it during a quiet window and expect a bounded period where one sensor’s readings appear on two partitions.

A key that includes the timestamp. It looks like it improves distribution, and it does — by destroying ordering completely, since every reading gets a unique key. If you find yourself reaching for this, what you actually want is a null key and an order-independent sink.

Serializing the key with a JSON encoder. json.dumps("eui-1") produces "eui-1" with quotes, which hashes differently from the raw bytes. Keys are opaque bytes; encode them as UTF-8 and stop.

Assuming consumers scale past partition count. A consumer group with more members than partitions leaves the extras idle forever. If throughput is short and partitions are maxed, the partition count is the thing to change — not the replica count.

FAQ

What actually breaks if I use a random or null key?

Ordering. With no key, the producer round-robins across partitions, so two readings from the same sensor can land on different partitions and be consumed in either order. Every stateful operation downstream — drift baselines, windowed means, last-value caches — assumes it sees a sensor’s readings in time order, and a null key quietly removes that guarantee.

How many partitions should a sensor topic have?

Enough that your peak throughput divided by per-consumer throughput fits, with headroom to add consumers later — and no more. Partitions are cheap to add and impossible to remove, and every one costs broker file handles and adds to rebalance time. For most environmental networks, twelve to twenty-four partitions handles years of growth.

Does adding partitions later break ordering?

Yes, for the duration of the change. The default partitioner hashes the key modulo the partition count, so increasing the count re-maps existing keys to different partitions and readings from one sensor briefly exist on two. Drain the topic, or accept a bounded window of reordering and let idempotent writes and event-time windows absorb it.