Sizing RocksDB State Stores for Sensor Aggregates #

State size in a streaming aggregation is not an emergent property — it is an arithmetic consequence of four numbers you already know, and it can be computed before a line of code runs. Teams discover this after a state store grows to 40 GB overnight and a rebalance takes twenty minutes to restore it. This guide does the arithmetic first, sets the RocksDB knobs that follow from it, and covers the one failure that breaks the model: a key space that is not bounded. It extends stateful IoT stream processing patterns.

The Four Numbers #

State size is keys × live_windows × bytes_per_entry × overhead, and each term has a natural source.

Keys is the cardinality of your grouping. For per-sensor aggregates it is the fleet size, which the device registry knows exactly. For per-sensor-per-metric it is fleet size times metrics. For per-H3-cell it is the occupied cell count from spatial windowing.

Live windows is retention divided by window advance. A tumbling five-minute window kept for one hour is 12; the same window kept for a day is 288. A sliding window advancing every 30 seconds with a five-minute span multiplies that by ten, which is the single largest lever on this page.

Bytes per entry is the serialized aggregate. A count-sum-min-max tuple is about 40 bytes; a reservoir sample for percentiles is kilobytes. Measure it rather than guessing — the difference between those two is three orders of magnitude in total state.

Overhead is RocksDB’s index, bloom filters and space amplification. A factor of 2 to 3 on the raw payload is a safe planning assumption at default settings.

Production-Ready Implementation #

Compute the estimate explicitly, in code, and assert on it in CI so a schema change that multiplies state fails the build rather than the cluster:

# python 3.11
from dataclasses import dataclass


@dataclass(frozen=True)
class StateEstimate:
    keys: int
    window_seconds: int
    advance_seconds: int          # == window_seconds for tumbling windows
    retention_seconds: int
    bytes_per_entry: int
    overhead_factor: float = 2.5   # RocksDB index, bloom filters, space amplification

    @property
    def live_windows(self) -> int:
        return max(1, self.retention_seconds // self.advance_seconds)

    @property
    def entries(self) -> int:
        return self.keys * self.live_windows

    @property
    def bytes_total(self) -> float:
        return self.entries * self.bytes_per_entry * self.overhead_factor

    def summary(self) -> str:
        gb = self.bytes_total / 1e9
        return (f"{self.keys:,} keys x {self.live_windows:,} live windows "
                f"= {self.entries:,} entries ~= {gb:.2f} GB")


# A city network: 2 000 sensors x 4 metrics, 5-minute tumbling windows kept for a day
city = StateEstimate(
    keys=2_000 * 4,
    window_seconds=300,
    advance_seconds=300,
    retention_seconds=86_400,
    bytes_per_entry=48,
)
assert city.bytes_total < 4e9, city.summary()     # 8 000 x 288 = 2.3 M entries ~= 0.28 GB

The same fleet with a sliding window advancing every thirty seconds:

sliding = StateEstimate(
    keys=2_000 * 4,
    window_seconds=300,
    advance_seconds=30,            # ten overlapping windows live at once
    retention_seconds=86_400,
    bytes_per_entry=48,
)
print(sliding.summary())           # 8 000 x 2 880 = 23 M entries ~= 2.8 GB

Ten times the state for the same data, from one parameter. That is the number to have in hand before choosing a window type in the sliding versus tumbling comparison.

RocksDB settings then follow from the estimate rather than from a blog post:

# faust==1.10.4 (rocksdb backend) — options passed through to python-rocksdb
ROCKSDB_OPTIONS = {
    # Block cache holds hot blocks. Aim at 25-30% of the working set, capped by
    # what the container can spare after the JVM-free Python heap and page cache.
    "max_open_files": 4096,
    "write_buffer_size": 64 << 20,        # 64 MB memtable
    "max_write_buffer_number": 3,
    "target_file_size_base": 64 << 20,
    "block_cache_size": 512 << 20,        # 512 MB for a ~2 GB working set
    "bloom_filter_size": 10,              # bits per key; cuts negative-lookup IO sharply
}

Parameter Tuning Guide #

The last row is the one that decides architecture. A restore time of forty minutes means every rebalance is an outage, and at that point an external state store — or a much shorter retention — is the answer rather than a bigger cache.

Verification and Testing #

Assert the estimate in CI, and verify the real store against it in staging.

# python 3.11 · pytest==8.2.0
def test_state_estimate_stays_within_the_node_budget():
    """Fails the build if a grouping or retention change blows the state budget."""
    est = StateEstimate(keys=8_000, window_seconds=300, advance_seconds=300,
                        retention_seconds=86_400, bytes_per_entry=48)
    assert est.bytes_total < 4e9, est.summary()


def test_measured_state_matches_the_estimate(rocksdb_path, replay_one_day):
    """Within a factor of two of the model — beyond that, an assumption is wrong."""
    replay_one_day()
    on_disk = sum(f.stat().st_size for f in rocksdb_path.rglob("*.sst"))
    est = StateEstimate(keys=8_000, window_seconds=300, advance_seconds=300,
                        retention_seconds=86_400, bytes_per_entry=48)
    assert 0.5 * est.bytes_total < on_disk < 2.0 * est.bytes_total

If the measured size exceeds the estimate by more than about double, the usual cause is not the estimate but expiry: check that old windows are actually being deleted, which is a watermark question rather than a storage one.

Gotchas #

An unbounded key space. Keying by something that grows without limit — a message id, a raw coordinate, a session token — makes every term in the model meaningless because keys never stops increasing. If you cannot state the cardinality of your key, that is the bug.

Retention set to “forever” by omission. Many frameworks default windowed tables to unlimited retention. State then grows linearly with uptime, which looks fine for a fortnight and is a 40 GB store by the end of the quarter.

Percentiles by reservoir per key per window. A t-digest or reservoir is kilobytes where a count-sum-min-max is tens of bytes. If you need percentiles per sensor per five-minute window, compute state size before committing — it is frequently the term that dominates everything else.

Ignoring space amplification during compaction. RocksDB temporarily needs room for the files it is merging. A disk sized exactly to the steady-state estimate will fill during a large compaction. Provision at least twice the estimate.

FAQ #

Related #

• Stateful IoT Stream Processing Patterns — the stateful processing stage this sizing belongs to
• Checkpointing and Recovering Stream State After a Crash — what happens to this state when the process dies
• Sliding vs Tumbling Windows for Sensor Rollups — the window choice that multiplies or divides everything on this page