Uncertainty Quantification and Data Quality Scoring #

Every reading a low-cost environmental network produces is wrong by some amount, and the amount is knowable. The uncomfortable part is that most pipelines never state it: readings are published as bare numbers, an analyst treats a rooftop PM2.5 sensor and a reference-grade instrument as interchangeable, and the resulting map implies a precision the hardware cannot deliver. The fix is not to discard the cheap sensors — their spatial density is the entire point of deploying them — but to ship a defensible uncertainty and a quality score alongside every value, so downstream work can weight rather than guess. This stage of automated calibration, validation and anomaly detection builds both.

The two are related and distinct. Uncertainty is physical: a number in the measurement’s own units that says how far the true value plausibly sits from the reported one. Quality is operational: a dimensionless score that folds in coverage, calibration freshness and flag history to say how much you should trust this row at all. A reading can have small uncertainty and poor quality — a precise sensor whose calibration expired eight months ago — and the two numbers say different, useful things.

---

Prerequisites #

• Python 3.11 with # python 3.11 · numpy==1.26.4 · pandas==2.2.2 · scipy==1.12.0.
• Calibration coefficients with residuals. The score needs not just m and b from cross-device normalization but the residual standard deviation of the fit — the number most calibration code computes and discards.
• QC flags already assigned. Flag history is a scoring input, so automated QC flagging must run first.
• A registry with calibration dates. Drift uncertainty grows with time since calibration, which requires device registry metadata to supply the interval.
• Neighbour readings on a common time grid. Cross-sensor agreement needs synchronized timestamps from resampling to a fixed grid.

---

Step-by-Step Workflow #

Step 1 — Enumerate the Uncertainty Budget #

Write down every contribution with a magnitude. The exercise is more valuable than the arithmetic: teams routinely discover that the term they have been optimizing is a tenth the size of one they have never measured.

# python 3.11 · numpy==1.26.4 · pandas==2.2.2 · scipy==1.12.0
from dataclasses import dataclass


@dataclass(frozen=True)
class UncertaintyBudget:
    """Standard uncertainty contributions for one sensor, in measurement units.

    Each term is a standard deviation (k=1), not a tolerance or a spec-sheet
    maximum — mixing the two is the most common error in this whole exercise.
    """

    noise: float                 # short-term repeatability, from co-location
    calibration_residual: float  # residual sd of the transfer-function fit
    drift_rate_per_day: float    # sd growth per day since calibration
    cross_sensitivity: float     # e.g. humidity effect on optical PM2.5

    def combined(self, days_since_calibration: float) -> float:
        """Independent contributions add in quadrature."""
        drift = self.drift_rate_per_day * days_since_calibration
        terms = (self.noise, self.calibration_residual, drift, self.cross_sensitivity)
        return sum(t * t for t in terms) ** 0.5

Complexity: O(1) per reading. The quadrature sum matters: a 3.0 and a 1.0 combine to 3.16, not 4.0, so chasing the smaller term is almost always wasted effort.

Step 2 — Score Each Quality Dimension Separately #

Four dimensions cover nearly every real defect, and each maps to a different remedy.

import numpy as np


def completeness_score(n_samples: int, expected: int) -> float:
    """Fraction of expected readings present in the interval, capped at 1."""
    return float(np.clip(n_samples / max(expected, 1), 0.0, 1.0))


def calibration_freshness(days_since: float, interval_days: float = 180.0) -> float:
    """1.0 at calibration, decaying to 0 at twice the recommended interval."""
    return float(np.clip(1.0 - days_since / (2 * interval_days), 0.0, 1.0))


def agreement_score(value: float, neighbours: list[float], sigma: float) -> float:
    """How consistent this reading is with co-located or nearby sensors.

    Expressed as the probability-like decay of a z-score against the neighbour
    median, so a reading 1 sigma out scores 0.6 and one 3 sigma out scores 0.01.
    """
    if not neighbours or sigma <= 0:
        return 0.5                      # no information is not the same as agreement
    z = abs(value - float(np.median(neighbours))) / sigma
    return float(np.exp(-0.5 * z * z))


def flag_score(qc_flag: int) -> float:
    """Map the CF-aligned flag vocabulary onto a trust weight."""
    return {1: 1.0, 2: 0.6, 3: 0.4, 4: 0.0, 8: 0.3, 9: 0.0}.get(qc_flag, 0.5)

Complexity: O(k) per reading for k neighbours, otherwise O(1). Keeping the dimensions separate is what makes the final score explainable — “0.42 because calibration freshness is 0.15” is actionable, “0.42” is not.

Step 3 — Combine Into One Publishable Score #

DEFAULT_WEIGHTS = {
    "flag": 0.40,          # a bad flag should dominate everything else
    "calibration": 0.25,
    "agreement": 0.20,
    "completeness": 0.15,
}


def quality_score(components: dict[str, float], weights: dict[str, float] | None = None) -> float:
    """Weighted mean of the sub-scores, with a hard veto on disqualifying flags.

    The veto matters: a hardware-failure flag must not be averaged away by three
    healthy dimensions into a score that still passes a 0.7 filter.
    """
    weights = weights or DEFAULT_WEIGHTS
    if components.get("flag", 1.0) == 0.0:
        return 0.0
    total = sum(weights[k] * components[k] for k in weights)
    return float(np.clip(total / sum(weights.values()), 0.0, 1.0))

Complexity: O(1). The veto is a deliberate departure from a pure weighted mean — without it, a reading flagged as a hardware failure scores 0.6 and passes most filters.

Step 4 — Propagate Through Aggregation #

An hourly mean is not sixty independent samples if the sensor is drifting. Split the budget into random and systematic parts and treat them differently.

def aggregate_uncertainty(u_random: float, u_systematic: float, n: int) -> float:
    """Uncertainty of a mean of n readings from one sensor.

    Random error averages down as 1/sqrt(n); systematic error (calibration bias,
    drift) is identical in every reading and passes through untouched.
    """
    if n <= 0:
        return float("nan")
    return ((u_random / n ** 0.5) ** 2 + u_systematic ** 2) ** 0.5

Complexity: O(1) per window. This is the function that stops an hourly mean claiming eight times the accuracy of the readings it came from — see propagating measurement uncertainty through aggregation for the spatial and multi-sensor cases.

Step 5 — Store Both Numbers on the Reading #

ALTER TABLE reading
    ADD COLUMN uncertainty       real,       -- standard uncertainty, k=1, measurement units
    ADD COLUMN quality_score     real,       -- 0..1
    ADD COLUMN quality_components jsonb;     -- the sub-scores, for explanation

Storing the components as JSON rather than four columns is a deliberate trade: they are read for explanation and audit, never filtered on, so query ergonomics matter less than the freedom to add a fifth dimension without a migration.

---

Configuration and Tuning #

Typical uncertainty budgets by sensor class #

The PM2.5 row makes the case for this whole stage: humidity cross-sensitivity is the largest single term, larger than noise and calibration combined, and it is the one most pipelines never quantify.

Score weights by use case #

Alerting weights flags heavily and calibration lightly on purpose: a mis-calibrated sensor still detects a tenfold spike, and delaying an alert for a calibration concern is the wrong trade in an incident.

---

Validation #

• Coverage of the uncertainty interval. Over a co-location period, roughly 68% of readings should fall within one combined uncertainty of the reference, and 95% within two. A coverage of 30% means the budget is understated; 99% means it is padded and the data is being undersold.
• Score-versus-error correlation. Bin readings by quality score and compute the mean absolute error against a reference in each bin. The relationship must be monotonic — if 0.9-scored readings are no more accurate than 0.5-scored ones, the weights are not measuring anything.
• Component independence. If two sub-scores correlate above about 0.8 across the fleet, they are measuring the same defect twice and the weighting is effectively doubled.
• Veto behaviour. Every reading with flag 4 or 9 must have score exactly 0.0. A single non-zero score here means the veto is being bypassed somewhere.
• Stability across recalibration. The score should step up at a recalibration, not jump discontinuously across the whole series. A discontinuity in uncertainty is expected; one in the reported values means the coefficients were applied retroactively, which validity intervals exist to prevent.

For a mature 200-sensor network, expect a median quality score near 0.85, roughly 6% of readings below 0.5, and a combined PM2.5 uncertainty between 4 and 7 µg/m³ at typical calibration ages.

---

Failure Modes and Edge Cases #

Adding uncertainties linearly. Independent contributions combine in quadrature. Summing them overstates the total by up to a factor of two, which sounds conservative and in practice leads to the figure being dismissed as useless.

Confusing a tolerance with a standard uncertainty. A datasheet “±5 µg/m³” is usually a maximum error, not a 1σ figure. Treating it as 1σ overstates uncertainty by roughly a factor of two; treating a 1σ figure as a maximum understates it by the same.

Agreement scoring on sensors that are not comparable. A street-canyon sensor and a rooftop sensor 200 m apart legitimately disagree. Scoring agreement against geographically near but functionally different neighbours penalises the sensor that is right.

A quality score that silently becomes a filter. Once a score exists, someone adds WHERE quality_score > 0.7 to the export and the low-scoring readings vanish from every downstream product without a note. Keep the unfiltered table available and record the threshold in the export’s metadata.

Drift uncertainty that never resets. If the registry’s calibration date is not updated after a service visit, drift uncertainty keeps growing and the score decays toward zero for a sensor that is in perfect condition. The score is only as good as the registry behind it.

Uncertainty on an imputed value. A gap-filled reading has the uncertainty of the imputation, not of the sensor — typically several times larger. Carry it explicitly rather than inheriting the sensor’s figure, as the gap-filling stage describes.

---

Integration #

This stage consumes the outputs of every earlier one: flags from QC, coefficients and residuals from calibration, cadence from resampling, and calibration dates from the registry. It produces two columns that change what downstream work can honestly claim.

1. Spatial interpolation weights observations by inverse uncertainty instead of treating every sensor equally, which is the single largest accuracy improvement available to a kriging or IDW surface built from mixed-grade instruments.
2. Aggregation carries a propagated uncertainty, so an hourly mean states its own error bar.
3. Export ships both columns, so a GeoJSON layer can style by confidence rather than showing every point with equal authority.

---

FAQ #

Is one quality score enough, or do I need the components?

Publish both. The single score is what makes filtering possible — an analyst writes WHERE quality_score > 0.7 and moves on. The components are what make the score defensible when someone asks why a station was excluded, and they are what tell an operator whether the fix is a recalibration, a field visit or nothing at all.

How do I estimate uncertainty for a low-cost sensor with no datasheet figure?

From co-location. Run the unit alongside a reference instrument for two to four weeks and take the standard deviation of the residuals after calibration as the combined uncertainty. It captures noise, cross-sensitivity and the calibration’s own error together, which is exactly what you want, and it is far more honest than a manufacturer figure measured in a chamber.

Does averaging reduce uncertainty?

Only the independent part. Random noise across n readings falls as one over the square root of n, so a 60-sample hourly mean has roughly an eighth of the noise of a single reading. Drift and calibration bias are systematic — every reading in the hour carries the same offset — so they pass through the average unchanged. Separate the two terms before you claim any improvement.

Should low-scoring readings be removed from the dataset?

No. Publish them with their score and let each consumer apply its own threshold. A regulatory submission may need score above 0.9 while a spatial interpolation is better served by including a 0.6 reading with a low weight than by leaving a hole in the coverage.

How often should the scoring weights be revisited?

Whenever the fleet changes materially — a new sensor model, a new firmware, a change of reference instrument — and otherwise annually against a held-out co-location period. Weights that were tuned on one hardware generation quietly stop reflecting reality on the next.

---

Related #

• Automated Calibration, Validation & Anomaly Detection for Environmental IoT — the calibration and validation section this scoring completes
• Computing a Composite Data Quality Score per Reading — the scoring function itself, with weights and worked examples
• Propagating Measurement Uncertainty Through Aggregation — what happens to the uncertainty when readings are averaged or interpolated
• Cross-Device Sensor Normalization Techniques — where the calibration residuals that feed the score come from
• Anomaly Detection Methods for Sensor Networks — the flag history that forms one of the score’s components