Flagging Imputed Values Through the Pipeline #

An imputed value that loses its label becomes a measurement. Not officially, not deliberately — it simply passes through one aggregation that drops the flag column, or one export that selects value and nothing else, and from that point on nothing in the dataset distinguishes it from something an instrument actually recorded. This guide covers the plumbing that prevents that: a flag vocabulary rich enough to be useful, provenance that survives aggregation, and assertions that stop imputed data reaching the places it must never reach. It is the bookkeeping half of gap filling and imputation strategies.

What Has to Survive, and How Far #

Three facts must travel with a filled value, and each survives a different distance.

That it was filled must reach the final consumer — the export, the API response, the map. This is non-negotiable and is what the flag column exists for.

How it was filled must reach anyone deciding whether to use it. Interpolated across two minutes and modelled from a neighbour are different products with different uncertainties, so the flag vocabulary needs to distinguish them rather than collapsing both to “not measured”.

What it was filled from must reach an auditor. The neighbour sensor, the model version, the residual standard deviation — enough to reproduce the value. This can live in a provenance column that most consumers never read.

The mistake almost every pipeline makes is designing for the first and assuming the others follow. They do not: the moment an aggregation runs, a flag has to be combined across rows, and the naive combination — take the maximum, or the modal value — destroys the distinction.

Production-Ready Implementation #

The vocabulary first. It extends the QC codes used across the pipeline rather than inventing a parallel scheme:

# python 3.11 · pandas==2.2.2 · numpy==1.26.4
from __future__ import annotations

import numpy as np
import pandas as pd

# CF-convention aligned; 7 and 8 are the imputation codes
QC_GOOD = 1            # measured, passed all checks
QC_SUSPECT = 2         # measured, questionable
QC_ANOMALOUS = 3       # measured, statistically unusual
QC_OUT_OF_RANGE = 4    # measured, physically impossible — no value published
QC_MODELLED = 7        # NOT measured: neighbour regression
QC_INTERPOLATED = 8    # NOT measured: temporal interpolation
QC_MISSING = 9         # NOT measured: no value at all

IMPUTED = {QC_MODELLED, QC_INTERPOLATED}
MEASURED = {QC_GOOD, QC_SUSPECT, QC_ANOMALOUS}

Provenance rides alongside as structured data, written at the moment of imputation:

def record_imputation(
    df: pd.DataFrame, mask: pd.Series, *, method: str, source: str,
    residual_sd: float, model_version: str,
) -> pd.DataFrame:
    """Attach flag, uncertainty and provenance to the rows this fill created."""
    out = df.copy()
    out.loc[mask, "qc_flag"] = QC_MODELLED if method == "neighbour" else QC_INTERPOLATED
    out.loc[mask, "uncertainty"] = residual_sd
    out.loc[mask, "provenance"] = [
        {"method": method, "source": source, "residual_sd": residual_sd,
         "model_version": model_version}
    ] * int(mask.sum())
    return out

Aggregation is where the flag has to be combined rather than carried, and the rule is that counts travel separately from the value:

def aggregate_with_provenance(df: pd.DataFrame, every: str = "1h") -> pd.DataFrame:
    """Windowed aggregate that reports how much of it was actually measured.

    n_measured is the number that a data-capture rule applies to; n_total is what
    the mean was computed over. Publishing only the mean makes a window built
    from two measurements indistinguishable from one built from sixty.
    """
    df = df.copy()
    df["is_measured"] = df["qc_flag"].isin(MEASURED)

    agg = (
        df.set_index("observed_at")
        .groupby("device_id")
        .resample(every, closed="left", label="left", origin="epoch")
        .agg(
            value=("value", "mean"),
            n_total=("value", "count"),
            n_measured=("is_measured", "sum"),
            worst_flag=("qc_flag", "max"),
        )
        .reset_index()
    )
    agg["capture_fraction"] = agg["n_measured"] / agg["n_total"].replace(0, np.nan)
    # A window is only 'measured' if every contributing row was
    agg["qc_flag"] = np.where(agg["capture_fraction"] == 1.0, agg["worst_flag"],
                              np.maximum(agg["worst_flag"], QC_INTERPOLATED))
    return agg.drop(columns=["worst_flag"])

The last two lines encode the rule that matters: an aggregate containing any imputed input is itself at best imputed. Taking the modal or minimum flag — both of which appear in real pipelines — produces an hourly mean flagged GOOD that is two thirds invented.

The assertion that protects calibration is three lines and belongs at the top of every fitting function:

def assert_measured_only(df: pd.DataFrame, context: str) -> None:
    """Fail loudly if imputed rows reach a stage that must only see measurements."""
    leaked = df["qc_flag"].isin(IMPUTED).sum()
    if leaked:
        raise ValueError(f"{context}: {leaked} imputed row(s) reached a measurement-only stage")

Parameter Tuning Guide #

Verification and Testing #

# python 3.11 · pandas==2.2.2 · pytest==8.2.0
def test_aggregate_of_partly_imputed_input_is_not_flagged_good():
    df = pd.DataFrame({
        "device_id": ["s1"] * 4,
        "observed_at": pd.to_datetime(["2026-08-01T10:00Z", "2026-08-01T10:15Z",
                                       "2026-08-01T10:30Z", "2026-08-01T10:45Z"]),
        "value": [20.0, 21.0, 22.0, 23.0],
        "qc_flag": [QC_GOOD, QC_GOOD, QC_INTERPOLATED, QC_GOOD],
    })
    out = aggregate_with_provenance(df, every="1h")
    assert out.loc[0, "n_total"] == 4
    assert out.loc[0, "n_measured"] == 3
    assert out.loc[0, "qc_flag"] >= QC_INTERPOLATED       # never GOOD


def test_calibration_refuses_imputed_input():
    df = pd.DataFrame({"qc_flag": [QC_GOOD, QC_GOOD, QC_MODELLED]})
    with pytest.raises(ValueError, match="imputed row"):
        assert_measured_only(df, context="calibration fit")


def test_export_never_drops_the_flag_column(exported_geojson):
    props = exported_geojson["features"][0]["properties"]
    assert "qc_flag" in props and "n_measured" in props

The third test is the boring one that catches the real incident. Export code changes far more often than imputation code, and a SELECT device_id, observed_at, value written in a hurry is how a labelled dataset becomes an unlabelled one.

Gotchas #

Combining flags with min or mode. Both produce an optimistic aggregate flag. Use max on a severity-ordered vocabulary, and make the ordering explicit so nobody renumbers the codes later.

Dropping the flag column in a join. A SELECT r.value, d.latitude in an export query loses it silently. Make the flag part of every published view rather than something each query remembers to include.

Storing provenance only in logs. Logs rotate. If the only record of which neighbour a value came from is a log line from four months ago, the value is unauditable — which for a regulatory dataset means unusable.

Re-imputing already-imputed rows. A second pass that treats a filled value as an input produces a value two models deep with the uncertainty of neither. Filter on the flag before every fill, not just before every fit.

FAQ #

Related #

• Gap Filling and Imputation Strategies for Sensor Gaps — the gap-filling stage whose output this tracks
• Choosing Between Interpolation and Neighbour Imputation — choosing the method whose provenance this records
• Exporting QC-Flagged Sensor Data to GeoJSON for QGIS — carrying the flag all the way into a map a stranger can read