Gap Filling and Imputation Strategies for Sensor Gaps #

Gaps are the normal state of an environmental sensor network. Radios retry, batteries sag in cold weather, a gateway reboots, a sensor is taken away for service. The question is never whether there will be gaps but what the pipeline does with them — and the default behaviour of most analysis tooling is to fill them silently, because interpolate() is one method call and thinking about it is not. The result is a dataset where measured and invented values are indistinguishable, which quietly poisons everything from calibration baselines to regulatory data-capture calculations. This stage of automated calibration, validation and anomaly detection makes the decision explicit: classify the gap, choose a method appropriate to its length and cause, and flag whatever you produce.

The rule underneath all of it: filling is a modelling decision, not a data-cleaning step. Every filled value is a prediction, and a prediction belongs in the dataset only when it is labelled as one.

---

Prerequisites #

• Python 3.11 with # python 3.11 · pandas==2.2.2 · numpy==1.26.4 · scikit-learn==1.4.2.
• A regular time grid. Gaps only exist relative to an expected cadence, so resampling to a fixed grid must run first — it is what turns “no row” into “a row with a missing value”.
• Gap causes already classified. The distinction between a short transmission gap and a hardware failure comes from QC flagging, and it is the input that decides whether filling is permitted at all.
• Neighbour geometry. Neighbour imputation needs to know which sensors are comparable, which comes from the site metadata in the device registry rather than from raw distance — a rooftop and a street-canyon sensor 100 m apart are not substitutes.
• A clean overlapping period. Fitting a neighbour relationship requires a stretch where both sensors were healthy, calibrated and reporting; two to four weeks is typical.

---

Step-by-Step Workflow #

Step 1 — Classify Every Gap by Cause and Length #

# python 3.11 · pandas==2.2.2 · numpy==1.26.4 · scikit-learn==1.4.2
import numpy as np
import pandas as pd

QC_GOOD, QC_INTERPOLATED, QC_MODELLED, QC_MISSING, QC_HARDWARE = 1, 8, 7, 9, 4


def gap_runs(series: pd.Series) -> pd.DataFrame:
    """Contiguous runs of missing values, with their length in intervals.

    Returns one row per gap: start index, end index, length. Filling decisions
    are made per gap, never per row — a row-wise rule cannot see that it sits in
    the middle of a six-hour outage.
    """
    missing = series.isna().to_numpy()
    if not missing.any():
        return pd.DataFrame(columns=["start", "end", "length"])
    edges = np.diff(np.concatenate(([0], missing.view(np.int8), [0])))
    starts = np.flatnonzero(edges == 1)
    ends = np.flatnonzero(edges == -1)
    return pd.DataFrame({"start": starts, "end": ends, "length": ends - starts})

Complexity: O(n) per sensor series. Working per gap rather than per row is what makes the length-based policy expressible at all.

Step 2 — Apply a Length- and Cause-Based Policy #

def fill_policy(length: int, cause_flag: int, *, max_interp: int = 2,
                max_neighbour: int = 60) -> str:
    """Which method, if any, is permitted for this gap.

    Hardware failures are never filled regardless of length: there was no
    instrument producing a value, so any estimate is a model output with no
    measurement behind it at all.
    """
    if cause_flag == QC_HARDWARE:
        return "none"
    if length <= max_interp:
        return "interpolate"
    if length <= max_neighbour:
        return "neighbour"
    return "none"

Complexity: O(1) per gap. The two thresholds are the whole policy, and both should be validated by hold-out rather than chosen by intuition.

Step 3 — Interpolate Short Gaps #

def interpolate_short(df: pd.DataFrame, max_gap: int = 2,
                      value_col: str = "value") -> pd.DataFrame:
    """Linear interpolation for runs of at most `max_gap`, flagged as interpolated.

    limit_area='inside' prevents extrapolation past the ends of the series, where
    there is nothing to interpolate between and the result is invention.
    """
    filled = df[value_col].interpolate(method="linear", limit=max_gap, limit_area="inside")
    invented = filled.notna() & df[value_col].isna()
    out = df.assign(**{value_col: filled})
    out.loc[invented, "qc_flag"] = QC_INTERPOLATED
    out.loc[invented, "uncertainty"] = out.loc[invented, "uncertainty"].fillna(0) + 2.5 * df[value_col].std()
    return out

Complexity: O(n). Note that the uncertainty is inflated, not inherited — the estimate is worse than a measurement and the column must say so.

Step 4 — Fit and Apply Neighbour Regression #

from sklearn.linear_model import HuberRegressor


def fit_neighbour_model(target: pd.Series, neighbours: pd.DataFrame) -> tuple[HuberRegressor, float]:
    """Regress a sensor on its neighbours over a clean overlapping period.

    Huber rather than ordinary least squares: an undetected spike in a neighbour
    should not drag the relationship that will later be used to invent values.
    Returns the model and the residual standard deviation — the imputation's own
    uncertainty, which is the number that matters more than the coefficients.
    """
    mask = target.notna() & neighbours.notna().all(axis=1)
    if mask.sum() < 200:
        raise ValueError(f"only {mask.sum()} overlapping observations — refuse to fit")
    model = HuberRegressor().fit(neighbours[mask], target[mask])
    residual_sd = float(np.std(target[mask] - model.predict(neighbours[mask])))
    return model, residual_sd


def impute_from_neighbours(
    df: pd.DataFrame, neighbours: pd.DataFrame, model: HuberRegressor, residual_sd: float,
    value_col: str = "value",
) -> pd.DataFrame:
    """Fill gaps from a fitted neighbour model, flagged and uncertainty-inflated."""
    fillable = df[value_col].isna() & neighbours.notna().all(axis=1)
    if not fillable.any():
        return df
    out = df.copy()
    out.loc[fillable, value_col] = model.predict(neighbours[fillable])
    out.loc[fillable, "qc_flag"] = QC_MODELLED
    out.loc[fillable, "uncertainty"] = residual_sd
    return out

Complexity: O(n·k) for k neighbours at fit time, O(1) per filled row afterwards. Refusing to fit on fewer than 200 overlapping observations is deliberate — a relationship estimated from a handful of points will be applied to thousands, and its errors are not random.

Step 5 — Record What Was Filled #

Every filled value needs three things attached: the flag, the inflated uncertainty, and the method. The flagging guide covers how those survive to export; the short version is that a boolean is_imputed column is not enough, because it cannot answer “imputed how?” six months later.

---

Configuration and Tuning #

Gap thresholds by measurement type #

Two rows carry the argument. Pressure is smooth and spatially coherent, so long fills are defensible. Rainfall is an event process where a gap contains either zero or a great deal, and any interpolation between two zeros invents a dry period that may have been a storm.

Method characteristics #

Forward fill appears here only to be argued against: it produces a flat line that looks like a real measurement, and it is the default of several time-series libraries.

---

Validation #

• Hold-out on real gaps. Mask known-good stretches matching your real gap-length distribution, impute, and compare. This is the only number that describes how wrong your fills are.
• Fill rate by sensor. A sensor whose values are more than about 10% imputed is not a working sensor; it is a model with a serial number. Report the rate per sensor per month.
• Distribution preservation. Compare the distribution of filled against measured values. Filling systematically compresses variance — that is expected — but a large shift in the mean indicates a biased neighbour relationship.
• Episode behaviour. Verify that no fill occurred during a period flagged as an exceedance event. Filling exactly when the signal is most dynamic is the worst case, and it is where a naive length-only rule fails.
• Downstream exclusion. Confirm that calibration and drift-baseline code excludes imputed rows by flag. Adding an assertion in the calibration path is cheaper than discovering the loop later.

Expect roughly 2–5% of intervals filled on a healthy cellular network, of which nearly all should be short interpolations rather than neighbour models.

---

Failure Modes and Edge Cases #

Filling during the event you care about. Gaps cluster with bad weather, power problems and heavy network load — the same conditions as pollution episodes. A rule based only on gap length will happily interpolate across the peak of a smoke event. Suppress filling when neighbours show elevated variance.

Neighbour models fitted across a change. If either sensor was recalibrated, moved or reflashed during the fitting window, the relationship spans two different regimes. Fit within validity intervals, not across them.

Imputed values feeding calibration. Covered in the FAQ and worth repeating as a failure mode: a drift baseline computed over interpolated data fits the interpolation. Exclude by flag at every fitting step.

Chained imputation. Sensor A is filled from B; B is later filled from A. Neither value has a measurement behind it, and nothing in the data says so. Only ever impute from measured values — check the flag on the neighbour before using it.

Data-capture calculations counting filled values. Regulatory completeness rules count measurements. If the export counts filled rows toward the 75% threshold, an invalid period is published as valid. Keep n_measured separate from n_total.

Silent extrapolation at series ends. interpolate() without limit_area="inside" extends the first and last known values outward, inventing data before deployment and after decommissioning.

---

Integration #

Gap filling sits after flagging and before aggregation, and it changes what the stages after it are allowed to assume. Aggregation must count measured samples separately; uncertainty propagation must use the inflated figure for filled rows; and spatial interpolation should weight them down or exclude them, since interpolating an interpolation compounds the assumption twice over.

The two guides below cover the decision itself and the plumbing that keeps it honest: choosing between interpolation and neighbour imputation and flagging imputed values through the pipeline.

---

FAQ #

When is it acceptable to fill a gap at all?

When the gap is short relative to the process being measured, the cause is transmission rather than instrument failure, and the filled value is flagged. A two-minute gap in a one-minute PM2.5 series meets all three. A four-hour gap during a smoke event meets none of them, because the very period you would be inventing is the one that varies fastest.

Interpolation or neighbour regression?

Interpolation for gaps of one or two intervals, where the assumption that the quantity moved smoothly is defensible. Neighbour regression for longer gaps where a correlated sensor was reporting — it uses real information about what happened during the gap instead of assuming nothing happened.

Should imputed values go into regulatory reports?

Only where the standard explicitly permits it, and always identified. Most air quality standards specify a minimum data capture — typically 75% of an averaging period — and treat the period as invalid below it rather than allowing it to be filled. Carry the measured count separately from the total so the rule can be applied.

How do I stop imputed values leaking into calibration?

Exclude them by flag at every fitting step. A drift baseline computed over interpolated values fits the interpolation, not the sensor, and the resulting coefficients then correct real readings toward an artefact. This is the single most damaging way imputed data propagates.

What uncertainty should a filled value carry?

The residual standard deviation of the method, measured by hold-out validation on your own data — typically two to four times the sensor’s own uncertainty for short interpolation, and larger for neighbour regression across any real distance. Never inherit the sensor’s uncertainty; the value did not come from the sensor.

---

Related #

• Automated Calibration, Validation & Anomaly Detection for Environmental IoT — the validation section this stage belongs to
• Choosing Between Interpolation and Neighbour Imputation — the decision procedure, with hold-out numbers for each method
• Flagging Imputed Values Through the Pipeline — keeping an estimate identifiable from ingest all the way to export
• Automating QC Flags for Missing Environmental Readings — the flagging stage that classifies gaps before this one fills them
• Uncertainty Quantification and Data Quality Scoring — where an imputed value’s inflated uncertainty is recorded