Choosing Between Interpolation and Neighbour Imputation #

Linear interpolation assumes nothing happened during the gap. Neighbour imputation assumes what happened at the target sensor resembles what happened at a nearby one. Both are wrong in different ways, and choosing between them is a question about which assumption your data supports — which means it is answerable, empirically, with a hold-out test on your own network. This guide runs that test and turns it into a decision rule for gap filling and imputation strategies.

The Two Assumptions, Stated Precisely #

Interpolation assumes smoothness. A straight line between the last reading before the gap and the first after it is exactly right if the quantity changed monotonically and slowly, and badly wrong if a plume passed through. Its error therefore grows with gap length and with the quantity’s short-term variability. For a stable variable it stays small for a long time; for urban PM2.5 it degrades within minutes.

Neighbour imputation assumes a stable relationship. If sensor A reliably reads about 1.1 times sensor B plus a small offset, then B’s readings during A’s gap are informative — and crucially, they are observations of the period in question rather than an assumption that the period was uneventful. Its error is set by how well the relationship holds, not by gap length, which is why it dominates for anything beyond a few intervals.

That difference in how the two errors scale is the whole decision. Interpolation error grows with gap length; neighbour error is roughly flat. They cross somewhere, and the crossing point is a property of your sensors and your site, not of the methods.

Production-Ready Implementation #

The comparison harness masks real data and scores both methods over the same gaps:

# python 3.11 · pandas==2.2.2 · numpy==1.26.4 · scikit-learn==1.4.2
from __future__ import annotations

import numpy as np
import pandas as pd
from sklearn.linear_model import HuberRegressor


def evaluate_methods(
    target: pd.Series,
    neighbours: pd.DataFrame,
    gap_lengths: tuple[int, ...] = (1, 2, 5, 15, 30, 60, 120),
    n_trials: int = 200,
    seed: int = 7,
) -> pd.DataFrame:
    """Hold-out comparison of interpolation and neighbour regression by gap length.

    For each gap length, mask that many contiguous known-good intervals at random
    positions, fill with each method, and score against the values that were
    masked. Returns RMSE per method per length — the curve that sets the
    crossover threshold.
    """
    rng = np.random.default_rng(seed)
    clean = target.notna() & neighbours.notna().all(axis=1)
    model = HuberRegressor().fit(neighbours[clean], target[clean])

    rows = []
    idx = np.flatnonzero(clean.to_numpy())
    for length in gap_lengths:
        err_interp, err_neigh = [], []
        for _ in range(n_trials):
            start = int(rng.choice(idx[: max(1, len(idx) - length - 2)]))
            window = slice(start, start + length)
            truth = target.iloc[window].to_numpy()
            if np.isnan(truth).any() or len(truth) < length:
                continue

            masked = target.copy()
            masked.iloc[window] = np.nan
            interp = masked.interpolate(method="linear", limit_area="inside").iloc[window].to_numpy()
            neigh = model.predict(neighbours.iloc[window])

            err_interp.append(np.sqrt(np.mean((interp - truth) ** 2)))
            err_neigh.append(np.sqrt(np.mean((neigh - truth) ** 2)))

        rows.append({
            "gap_intervals": length,
            "rmse_interpolation": float(np.mean(err_interp)) if err_interp else np.nan,
            "rmse_neighbour": float(np.mean(err_neigh)) if err_neigh else np.nan,
        })
    return pd.DataFrame(rows)


def crossover_length(table: pd.DataFrame) -> int | None:
    """The gap length beyond which neighbour imputation wins. None if it never does."""
    worse = table["rmse_interpolation"] > table["rmse_neighbour"]
    return int(table.loc[worse, "gap_intervals"].min()) if worse.any() else None

The decision rule then reads directly off that table:

def choose_method(gap_length: int, crossover: int | None, neighbour_r: float,
                  max_interp: int, max_fill: int) -> str:
    """Method for one gap, from measured crossover rather than intuition."""
    if gap_length > max_fill:
        return "none"
    if neighbour_r < 0.8 or crossover is None:
        return "interpolate" if gap_length <= max_interp else "none"
    return "interpolate" if gap_length < crossover else "neighbour"

The neighbour_r < 0.8 guard is what stops the rule reaching for a neighbour that carries no information. Below that correlation, a regression’s residual is close to the target’s own standard deviation, and a value with that much uncertainty communicates less than an honest gap.

Parameter Tuning Guide #

Measured on a 40-sensor urban network, one-minute cadence, six months of data:

Pressure is the outlier: it is spatially coherent over tens of kilometres and changes slowly, so both methods work well and interpolation stays competitive far longer than for anything else.

Verification and Testing #

# python 3.11 · pytest==8.2.0
def test_interpolation_wins_at_short_gaps_and_loses_at_long_ones(pm25_series, neighbours):
    table = evaluate_methods(pm25_series, neighbours)
    short = table.loc[table["gap_intervals"] <= 2]
    long_ = table.loc[table["gap_intervals"] >= 30]
    assert (short["rmse_interpolation"] < short["rmse_neighbour"]).all()
    assert (long_["rmse_interpolation"] > long_["rmse_neighbour"]).all()


def test_a_weak_neighbour_disables_neighbour_imputation():
    assert choose_method(30, crossover=3, neighbour_r=0.55, max_interp=2, max_fill=60) == "none"


def test_no_method_is_chosen_past_the_fill_limit():
    assert choose_method(200, crossover=3, neighbour_r=0.95, max_interp=2, max_fill=60) == "none"

Re-run evaluate_methods whenever the network changes materially — a new sensor model, a site moved, a season with different meteorology. The crossover is a property of the current network, and a threshold inherited from last year’s hardware is a guess wearing a number.

Gotchas #

Evaluating on gaps that are not like real gaps. Random masking produces gaps uniformly through the day; real gaps cluster at night, in cold weather and during network congestion. Sample your hold-out positions from the observed gap distribution or the numbers will be optimistic.

Fitting the neighbour model on data that includes the gaps you are evaluating. A subtle leak: if the model is fitted on the whole series and then evaluated on masked windows within it, the fit has seen the answers. Fit on a disjoint period.

Comparing methods on different gaps. Both methods must be scored on identical masked windows, or the comparison measures which gaps each method happened to get.

Assuming the crossover is symmetric across sensors. A sensor with no good neighbour has no crossover at all — the correct answer for every gap beyond the interpolation limit is to leave it empty, and the rule needs to express that rather than falling back to a poor regression.

FAQ #

Related #

• Gap Filling and Imputation Strategies for Sensor Gaps — the gap-filling stage this decision belongs to
• Flagging Imputed Values Through the Pipeline — recording whichever method you chose so it stays visible downstream
• Cross-Calibrating PM2.5 Monitors with Linear Regression — the same regression machinery, used for calibration rather than filling