LoRaWAN Payload Parsing for Environmental Sensors #

A LoRaWAN uplink from a field sensor is deliberately tiny — often 4 to 20 bytes — because the radio protocol trades payload size for kilometre-scale range and multi-year battery life on a coin cell. That frugality moves all of the interpretation burden onto your ingestion code: the network server hands you a base64 string, and nothing about it tells you that byte 3 is a signed decidegree temperature or that bytes 5 and 6 are a big-endian humidity in half-percent steps. Get the byte offsets, endianness, sign handling, or scaling factor wrong and the error is silent — the pipeline stores a physically plausible but incorrect number that no downstream validation will catch. This is the parsing stage of the IoT Sensor Data Ingestion & Spatial Synchronization pipeline: the point where opaque LoRaWAN bytes become typed, geolocated, unit-correct rows.

The two formats you will meet in practice are Cayenne Low Power Payload (LPP), a self-describing channel/type encoding used by many off-the-shelf devices, and bespoke fixed-layout binary structs emitted by custom firmware where every bit is budgeted. This page covers both, plus the envelope work of pulling payloads out of The Things Network uplink JSON and mapping the result onto a spatial schema.

Prerequisites #

These are the concrete requirements for a production LoRaWAN decoding service. Each maps to a real failure seen in the field, not a boilerplate checklist.

• Python 3.11 with type hints. The decoders below lean on struct.unpack and exact integer widths; silent int/float coercion during scaling is the most common source of off-by-a-factor bugs.
• Pinned libraries: pandas==2.2.2 for assembling decoded frames into typed tables, paho-mqtt==2.1.0 for subscribing to the network server’s MQTT uplink stream, and pydantic==2.6.4 to validate the decoded envelope before it reaches storage. The byte-level decoders themselves use only the standard-library struct and base64 modules, so they carry no third-party dependency; if you prefer not to hand-roll the Cayenne walk, the cayennelpp decoder package is a drop-in alternative that returns the same channel/type dictionaries.
• A payload format contract. You must know, per device profile and per fPort, whether the frame is Cayenne LPP or a fixed struct, and for structs the exact field order, width, endianness, sign, and scale. Treat this as a versioned document owned jointly with firmware; a struct layout change without a decoder version bump corrupts every subsequent reading.
• Envelope access. Your MQTT integration must deliver the uplink JSON with frm_payload/payload_raw (base64), the device EUI, the uplink timestamp, and rx_metadata gateway coordinates. If you are still standing up the transport, complete MQTT broker integration for environmental sensors first — decoding cannot begin until frames arrive reliably.
• UTC normalization downstream. LoRaWAN network servers timestamp on receipt; edge nodes rarely carry a real-time clock. Route decoded rows through timestamp alignment and timezone normalization so every reading lands in UTC before spatial joins.

---

Step-by-Step Workflow #

Step 1 — Recover the Raw Byte String #

Every network server delivers the measurement as base64 text inside a JSON envelope. The first job is to turn that text back into the exact bytes the radio carried, and to reject anything that does not decode cleanly rather than letting a malformed frame poison the batch.

# python 3.11 · pandas==2.2.2 · paho-mqtt==2.1.0 · pydantic==2.6.4
import base64
import struct


def payload_bytes(payload_b64: str) -> bytes:
    """Base64-decode a LoRaWAN frm_payload into raw bytes.

    Raises ValueError on non-base64 input so callers can dead-letter the frame
    instead of decoding garbage.
    """
    try:
        raw = base64.b64decode(payload_b64, validate=True)
    except (ValueError, TypeError) as exc:
        raise ValueError(f"payload is not valid base64: {payload_b64!r}") from exc
    if not raw:
        raise ValueError("empty payload after base64 decode")
    return raw

Parameter notes: pass validate=True so stray characters raise instead of being silently ignored — a truncated uplink should fail loudly. Complexity: O(n) in payload length, and n is at most 222 bytes for the largest LoRaWAN data rate, so this is effectively constant time.

---

Step 2 — Decode Cayenne LPP Channels #

Cayenne LPP is self-describing: the frame is a flat sequence of (channel, type, data...) groups where the type byte determines how many data bytes follow and their scaling. That structure lets one uplink carry a temperature, a humidity, and a GPS fix without any external schema — the decoder just walks the stream. The data-size and scaling rules for the common environmental types are fixed by the specification:

# Cayenne LPP: (data_size_bytes, scale) per type id.
# Signed types are handled explicitly in the decoder below.
LPP_TYPES: dict[int, tuple[str, int, float, bool]] = {
    0x00: ("digital_in",   1, 1.0,   False),
    0x01: ("digital_out",  1, 1.0,   False),
    0x02: ("analog_in",    2, 0.01,  True),
    0x67: ("temperature",  2, 0.1,   True),   # 0.1 °C, signed
    0x68: ("humidity",     1, 0.5,   False),  # 0.5 %RH
    0x71: ("accelerometer", 6, 0.001, True),
    0x88: ("gps",          9, 1.0,   True),   # lat/lon 1e-4, alt 1e-2
}


def decode_cayenne(raw: bytes) -> list[dict]:
    """Walk a Cayenne LPP frame into a list of channel measurements."""
    out: list[dict] = []
    i, n = 0, len(raw)
    while i + 2 <= n:
        channel, type_id = raw[i], raw[i + 1]
        spec = LPP_TYPES.get(type_id)
        if spec is None:
            raise ValueError(f"unknown LPP type 0x{type_id:02x} at offset {i}")
        name, size, scale, signed = spec
        i += 2
        if i + size > n:
            raise ValueError(f"truncated {name} channel at offset {i}")
        chunk = raw[i:i + size]
        i += size
        if type_id == 0x88:  # GPS is three signed 24-bit fields
            lat = int.from_bytes(chunk[0:3], "big", signed=True) * 1e-4
            lon = int.from_bytes(chunk[3:6], "big", signed=True) * 1e-4
            alt = int.from_bytes(chunk[6:9], "big", signed=True) * 1e-2
            out.append({"channel": channel, "type": name,
                        "latitude": lat, "longitude": lon, "altitude_m": alt})
        else:
            value = int.from_bytes(chunk, "big", signed=signed) * scale
            out.append({"channel": channel, "type": name, "value": value})
    return out

Parameter notes: the sign flag matters — temperature and analog inputs swing negative, humidity never does, and reading a signed field as unsigned turns −1.0 °C into 6553.5 °C. Complexity: O(b) in the number of payload bytes; the loop advances by each channel’s fixed width.

---

Step 3 — Decode a Custom Binary Struct #

When firmware engineers need every bit — a solar node budgeting airtime under a 1% duty cycle — they abandon Cayenne LPP’s per-channel overhead for a fixed layout and hand you a format string instead. struct.unpack reads the whole frame in one call, provided you and firmware agree on the byte order and field widths.

# Example fixed layout, big-endian:
#   uint16 battery_mv | int16 temp_centi_c | uint16 humidity_permille | uint8 status
_STRUCT = struct.Struct(">HhHB")


def decode_custom(raw: bytes) -> dict:
    """Decode a fixed-layout environmental node frame via struct.unpack."""
    if len(raw) != _STRUCT.size:
        raise ValueError(f"expected {_STRUCT.size} bytes, got {len(raw)}")
    battery_mv, temp_centi, humidity_permille, status = _STRUCT.unpack(raw)
    return {
        "battery_v": battery_mv / 1000.0,
        "temperature_c": temp_centi / 100.0,   # signed centidegrees
        "humidity_pct": humidity_permille / 10.0,
        "low_battery": bool(status & 0x01),
    }

Parameter notes: the > prefix pins big-endian, which is conventional for LoRaWAN but never guaranteed — confirm it against a known reference frame before trusting the fleet. Complexity: O(1); the layout width is fixed at compile time.

The trade-off between the two formats is worth being explicit about. Cayenne LPP costs two header bytes per channel and constrains you to its predefined types and resolutions, but it is self-describing, version-tolerant, and decodable without any device-specific knowledge — a new sensor added to a node just appears as another channel. A custom struct spends nothing on headers and lets firmware pick exactly the bit widths a measurement needs, which matters when a solar node is fighting a 1% duty cycle for airtime, but every layout change is a coordinated firmware-and-decoder release, and a silent mismatch corrupts the whole fleet at once. As a rule, reach for LPP on mixed off-the-shelf hardware where flexibility wins, and for a pinned struct only on tightly controlled custom nodes where airtime is the binding constraint. Whichever you choose, the decoder registry keyed on (device_profile, fPort) lets both coexist in one ingestion service.

---

Step 4 — Assemble Typed Rows and Map to the Spatial Schema #

Decoded measurements are only useful once they carry identity, time, and place. This step folds the channel dictionaries together with envelope metadata into flat rows whose columns match your spatial store, and builds a pandas frame so downstream stages get consistent dtypes.

import pandas as pd


SPATIAL_COLUMNS = ["device_eui", "channel", "metric", "value",
                   "latitude", "longitude", "received_at"]


def to_spatial_rows(
    measurements: list[dict],
    *,
    device_eui: str,
    received_at: str,
    gateway_lat: float,
    gateway_lon: float,
) -> pd.DataFrame:
    """Fold decoded channels + envelope metadata into typed spatial rows."""
    rows: list[dict] = []
    for m in measurements:
        # GPS channels carry their own coordinates; scalar channels inherit the gateway fix.
        lat = m.get("latitude", gateway_lat)
        lon = m.get("longitude", gateway_lon)
        rows.append({
            "device_eui": device_eui,
            "channel": m["channel"],
            "metric": m["type"],
            "value": m.get("value"),
            "latitude": lat,
            "longitude": lon,
            "received_at": received_at,
        })
    df = pd.DataFrame(rows, columns=SPATIAL_COLUMNS)
    return df.astype({"channel": "int16", "value": "float64",
                      "latitude": "float64", "longitude": "float64"})

Parameter notes: scalar channels inherit the gateway fix as a coarse location; if the node carries a GPS channel, prefer its coordinates. Complexity: O© in channel count per uplink, well under a dozen for typical environmental frames.

---

Configuration and Tuning #

Decoder selection and scaling are device-specific. The table maps common environmental measurements to their Cayenne LPP encoding so you can validate decoded magnitudes at a glance.

Endianness: Cayenne LPP is big-endian for multi-byte scalars; custom structs vary — always pin the byte order in your format string. fPort routing: map (device_profile, fPort) to a decoder function in a registry dict so a device sending a boot frame on fPort 3 never runs the fPort 1 measurement decoder. Unknown types: fail closed. An unrecognised LPP type id means either a firmware change or a corrupted frame, and both deserve a dead-letter entry, not a best-effort guess.

---

Validation #

Before decoded rows enter storage, confirm they are structurally and physically sound.

• Round-trip a known frame. Keep one golden uplink per device profile with a hand-verified expected output and assert equality in CI. Scaling regressions surface immediately.
• Physical bounds. Reject or flag temperatures outside −60 to 85 °C, humidity outside 0–100 %RH, and coordinates outside the deployment bounding box. A GPS field decoded with the wrong sign lands the sensor in the wrong hemisphere — an easy automated catch.
• Channel completeness. For multi-sensor nodes, assert the expected channel set is present. A frame missing its humidity channel usually signals a truncated payload rather than a real dropout.
• Type distribution. Over a batch, the count of decoded channels per uplink should be stable. A sudden shift means a firmware or format change you were not told about.

Expected shape for a single temperature-plus-humidity uplink: two rows from to_spatial_rows, both sharing one device_eui and received_at, distinct channel values, and value columns within physical bounds.

---

Failure Modes and Edge Cases #

Sign errors on temperature and analog inputs. Reading a signed 16-bit field as unsigned turns any sub-zero reading into a value near 6553.5. Winter deployments expose this instantly; summer testing hides it. Always pass signed=True for temperature, analog, and GPS.

Endianness mismatch. A big-endian decoder run on a little-endian struct produces byte-swapped nonsense — 0x0100 read as 0x0001. It is silent when values happen to be small. Verify against a reference frame with a known non-trivial value before fleet rollout.

Truncated frames. Cellular-to-backhaul hops and spreading-factor changes occasionally deliver a short payload. A decoder that indexes past the end raises IndexError deep in the walk; the bounds checks above convert that into a clean dead-letter reason.

Missing GPS fix. Many nodes send 0,0 or all-0xFF coordinates before their first satellite lock. Do not store (0, 0) as a real location in the Gulf of Guinea — detect the sentinel and fall back to the gateway position or mark the location null.

fPort confusion. Configuration and measurement frames share a device but differ in layout. Decoding a config frame with the measurement decoder yields plausible-looking garbage. Route strictly on fPort.

---

Integration #

Decoded, geolocated rows are the input to the rest of the ingestion pipeline. The two focused walkthroughs below take a specific format end to end: decoding Cayenne LPP payloads into a pandas DataFrame builds the struct-based LPP decoder into a typed table, and parsing The Things Network uplinks into PostGIS-ready rows handles the full TTN v3 envelope including gateway metadata.

From here, rows carry a UTC timestamp via timestamp alignment and timezone normalization and then land in spatial storage for indexing and interpolation. Keep the raw base64 payload alongside the decoded columns for a retention window; when a scaling bug is found, you can re-decode history rather than losing it.

---

FAQ #

Why do my decoded LoRaWAN temperatures look 100x too large?

You almost certainly skipped the per-type scaling divisor. Cayenne LPP temperature is a signed 16-bit integer in units of 0.1 °C, so a raw value of 235 means 23.5 °C. Divide by the type’s resolution factor and honour the sign bit before storing the value.

Should I decode payloads on the device, in the network server, or in my ingestion service?

Decode in your ingestion service, not on the device. Keeping the airtime payload compact preserves battery and stays within the duty-cycle limit, while a versioned Python decoder in your pipeline lets you fix scaling bugs and add fields without a firmware over-the-air update to remote nodes.

How do I handle a single uplink that carries several sensors?

Cayenne LPP multiplexes multiple sensors into one uplink using a channel byte before each measurement. Decode every channel/type pair in the frame, then either keep the reading wide (one row per uplink with many columns) or melt it long (one row per channel) depending on whether your spatial store is column-per-metric or metric-value-per-row.

Which fPort should I trust to pick a decoder?

Treat fPort as the payload format selector agreed with the firmware team, not a guess. Many fleets reserve fPort 1 for Cayenne LPP measurement frames and higher ports for configuration or boot frames. Route on the (device_profile, fPort) pair and reject frames on unexpected ports to a dead-letter queue rather than mis-decoding them.

---

Related #

• IoT Sensor Data Ingestion & Spatial Synchronization — parent overview of the full ingestion and spatial-sync pipeline
• Decoding Cayenne LPP Payloads into a Pandas DataFrame — the self-contained struct-based LPP decoder with scaling table and tests
• Parsing The Things Network Uplinks into PostGIS-Ready Rows — turning TTN v3 uplink JSON and gateway metadata into insert-ready rows
• MQTT Broker Integration for Environmental Sensors — the transport that delivers LoRaWAN uplinks to your decoder
• Timestamp Alignment and Timezone Normalization — UTC normalization for decoded readings before spatial joins