Skip to content

Building CRA-ready Mender OTAs

Signed firmware updates delivered over Mender's OTA channel, with the manufacturer's signing key held inside the HSM.

Secure boot stops a device running firmware you did not authorise. This is the other half: making sure the updates that reach it over the air are equally authentic, for the whole of the product's supported life. Together they are what the CRA's essential requirements on update authenticity and tamper resistance are asking for.

The three parties

  • LAAVAT cloud

    Signer

    The HSM holds the manufacturer's RSA signing key. It sees only a 32-byte digest of the artifact manifest — never the firmware bytes.

    Provides — a signature over the manifest

  • Mender server

    Distribution

    Hosts the signed .mender artifact and orchestrates the fleet-wide rollout: scheduling, retries, status reporting.

    Never sees — the signing key, and does not need it

  • mender-client

    Verifier and installer

    On each device: downloads, verifies the signature against the pre-provisioned public key, checks every file's hash, installs to the inactive partition, reboots.

    Trusts — one public key, provisioned once at manufacturing

Core property — HSM-resident signing key

The RSA private key that authenticates every OTA update stays inside the HSM and is never exportable. What crosses the wire is only the signature over the artifact manifest — a few hundred bytes appended to a file inside the .mender tar. Every device in the fleet verifies against a single public key, provisioned once at manufacturing.

What Mender is, and is not

Mender is an open-source OTA firmware update system for embedded Linux and RTOS devices. It solves the delivery problem: getting a signed artifact from a server to a fleet, with rollback, deployment scheduling, retry logic and status reporting. It does not generate signing keys and does not run an HSM.

That is the fit. Mender handles distribution and installation; LAAVAT handles signing. The two meet at a single well-defined file — a base64-encoded signature (manifest.sig) inside the outer tar of every .mender artifact. As long as that signature verifies against the public key the client was provisioned with, the update is trusted.

One external signer among several

Mender's mender-artifact tool natively supports several external key backends, including PKCS#11. The digest-signing approach described here is another external-signer pattern in that family. What differs is how the private key is reached — not what gets signed, and not what the on-device verification looks like.

Inside a .mender artifact

A .mender artifact is a plain, uncompressed tar with a strictly ordered layout. The signature covers the manifest, and the manifest covers every payload file by SHA-256.

firmware-v2.1.mender          # outer tar (uncompressed)
├── version                   # format version (v3). Required first.
├── manifest                  # SHA-256 of every other file, sha256sum(1) format
├── manifest.sig              # base64 signature over the manifest — what LAAVAT produces
├── manifest-augment          # optional, for augmented/delta artifacts
├── header.tar[.gz|.xz|.zst]  # artifact metadata: name, device types, update type
├── header-augment.tar[...]   # optional, for augmented artifacts
└── data/0000/<payload>       # the firmware payload, e.g. update.ext4

Order matters. The format specification is explicit: version first, then manifest, then the optional manifest.sig, then the header files, then data/. Any non-data file appearing after a data file fails verification immediately.

Integrity cascades. The manifest lists the SHA-256 of header.tar.gz and of every file under data/, formatted exactly as sha256sum output. Tamper with a payload and its recomputed hash will not match the manifest; tamper with the manifest and its signature will not verify. Signing one file secures the whole artifact.

The LAAVAT product

Mender signing needs no PKI hierarchy — the device holds a raw public key, not a certificate chain. One product with a single operation is enough.

  • Mender artifact signing

    operationType: DigestSigning

    Signs the SHA-256 hash of each artifact's manifest. You compute the digest locally and submit it; LAAVAT returns an RSA signature over that digest, which you base64-encode into manifest.sig.

    Key — RSA, SHA-256, PKCS#1 v1.5 padding

Mender recommends RSA of at least 3072 bits. ECDSA P-256 is also supported if you prefer EC keys — see supported keys for the full matrix.

The public half is exported once and copied to every device as artifact-verify-key.pem, referenced by ArtifactVerifyKey in the mender-client configuration. Manufacturing programs it in during first boot, or bakes it into the golden image.

Two different keys

This page covers the manufacturer's artifact-signing key. Each Mender client also carries its own device-identity key pair, used to authenticate to the Mender server. That is a separate concern with a separate lifecycle, generated on-device at first boot, and out of scope here.

Example product definition

Group UUIDs are per-tenant; substitute your own.

{
  "name": "Mender OTA signing",
  "description": "Sign Mender .mender artifacts for OTA delivery",
  "productType": "Production",
  "enabled": true,
  "caInfo": [],
  "rndKeys": [],
  "productOperations": [
    {
      "name": "Mender artifact signing",
      "description": "Sign the SHA-256 digest of a .mender manifest",
      "operationType": "DigestSigning",
      "token": {
        "name": "Mender signing key",
        "description": "RSA key for Mender manifest signing",
        "keyType": "RSA3072"
      },
      "approvalRule": {
        "name": "Test rule",
        "description": "Rule used for testing",
        "allowedGroups": ["<allowed-group-uuid>"],
        "approvalGroups": ["<approval-group-uuid>"],
        "blanketGroups": []
      }
    }
  ]
}

Token handling

The examples below pipe the token in on stdin with -t @-, which suits CI secret managers. A config file created with config-init works equally well — see secure token handling.

Signing a release

Six steps. The firmware payload never leaves the build host — only the manifest digest reaches LAAVAT.

1. Build the unsigned artifact

Use Mender's own tool to package the firmware. This produces the tar with version, manifest, header.tar.gz and data/0000, but no manifest.sig.

mender-artifact write rootfs-image \
    --device-type <target-device-type> \
    --artifact-name <fw-name-v2.1.0> \
    --file rootfs.ext4 \
    --output-path firmware.unsigned.mender

For non-rootfs updates the equivalent is mender-artifact write module-image --type <module> --file payload.tar.gz .... Either shape produces the same five-file outer tar.

2. Extract the manifest and hash it

Untar just the manifest, compute its SHA-256, and base64-encode the raw 32 bytes for submission.

mkdir -p work
tar -xf firmware.unsigned.mender -C work/ manifest

DIGEST_B64=$(sha256sum work/manifest | awk '{print $1}' \
             | xxd -r -p | base64)

3. Submit the digest

REQ_ID=$(printf '%s' "$TOKEN" | signing-tool --json -c -t @- \
    -a https://app.laavat.io/<CustomerName>/api/v1 \
    imagesigning add DigestSigning \
    -P <product-id> \
    --operid <mender-oper-id> \
    -p "$DIGEST_B64" \
    -H SHA256 \
    -N mender-artifact \
    -D "Mender OTA signing" | jq -r '.id')

4. Download the signature

The request is asynchronous, moving from Created to Ready. Use --wait to block rather than polling by hand.

printf '%s' "$TOKEN" | signing-tool -c -t @- \
    -a https://app.laavat.io/<CustomerName>/api/v1 \
    imagesigning get -I "$REQ_ID" --wait \
    -O work/sig.bin

RSA-3072 produces 384 bytes; RSA-2048 produces 256, RSA-4096 produces 512.

5. Base64-encode into manifest.sig

The filename must be exactly manifest.sig — that is what mender-client looks for inside the outer tar.

base64 -w0 work/sig.bin > work/manifest.sig

6. Repack the outer tar

Explode the unsigned artifact and rebuild it with manifest.sig in the position mender-artifact sign would have placed it — between manifest and header.tar.gz.

tar -xf firmware.unsigned.mender -C work/

tar -cf firmware.signed.mender -C work/ \
    version manifest manifest.sig header.tar.gz data

The result is a fully signed .mender, equivalent to what mender-artifact sign --key private.pem would produce — except the private key never touched the build host.

All six steps wrap into a shell script for CI to call on every release. Signing needs only signing-tool, tar and standard coreutils (sha256sum, xxd, base64), all present on any Linux build agent.

What the device does at update time

Once the artifact is downloaded, every step is offline: no further contact with LAAVAT, and none with the Mender server for the cryptography.

  1. Download the artifact. Streamed rather than buffered — the client parses the tar on the fly and writes payload chunks straight to the inactive rootfs partition, or to an Update Module for non-rootfs updates.
  2. Verify the manifest signature. Read manifest.sig, base64-decode it, and verify against the manifest bytes using the public key configured in ArtifactVerifyKey. If this fails the update is aborted before anything is installed.
  3. Verify every file's hash. For each entry in the manifest, recompute the SHA-256 of the corresponding file and compare. Any mismatch aborts. This is what makes signing only the manifest sufficient.
  4. Install to the inactive partition. The active partition is untouched throughout.
  5. Commit or roll back. The bootloader is told to try the new partition on next boot. If the new firmware does not call mender commit within a configurable timeout, the bootloader falls back to the previous partition — automatic rollback on a bad update.

Verifying without a device

Every check the client performs is standard file and crypto handling, reproducible on any Linux host. Fetch the public key once:

printf '%s' "$TOKEN" | signing-tool -c -t @- \
    -a https://app.laavat.io/<CustomerName>/api/v1 \
    product getpubkey \
    -P <product-id> \
    --operid <mender-oper-id> \
    -O /tmp/mender_pub.pem

That is also the file to bake into every device's manufacturing image as artifact-verify-key.pem.

Mender's own validator

The authoritative check — the same code path the on-device client uses, run as a CLI:

mender-artifact validate firmware-v2.1.mender \
    --key /tmp/mender_pub.pem

Exits zero if the signature is valid and every file's hash matches. If this passes, a real device with the same public key provisioned would accept the artifact.

Raw OpenSSL, for debugging

tar -xf firmware-v2.1.mender manifest manifest.sig
base64 -d manifest.sig > sig.bin
openssl dgst -sha256 -verify /tmp/mender_pub.pem \
    -signature sig.bin manifest

This bypasses mender-artifact entirely. Useful when a signature fails and you need to isolate whether the problem is the cryptography or the tar layout.

Where off-device verification stops

Claim Off-device What would close the gap
Manifest signature is cryptographically valid Fully proven —
Every payload file's SHA-256 matches the manifest Fully proven —
Tar layout matches what the client parses Fully proven mender-artifact validate enforces the layout
The target device has the correct public key provisioned Not testable off-device Manufacturing workflow or first-boot provisioning
An Update Module's install script behaves correctly on-target Not tested Integration test on real hardware or QEMU
Rollback on a failed boot actually reverts Out of scope Bootloader integration test on real hardware

References