> Markdown version of https://docs.laavat.io/solutions/cra-ready-mender-ota/ from the LAAVAT PKI and Signing Platform documentation. All pages: https://docs.laavat.io/llms.txt # Building CRA-ready Mender OTAs Signed firmware updates delivered over Mender's OTA channel, with the manufacturer's signing key held inside the HSM. Secure boot stops a device running firmware you did not authorise. This is the other half: making sure the updates that reach it over the air are equally authentic, for the whole of the product's supported life. Together they are what the CRA's essential requirements on update authenticity and tamper resistance are asking for. ## The three parties
- __LAAVAT cloud__ --- *Signer* The HSM holds the manufacturer's RSA signing key. It sees only a 32-byte digest of the artifact manifest — never the firmware bytes. **Provides** — a signature over the manifest - __Mender server__ --- *Distribution* Hosts the signed `.mender` artifact and orchestrates the fleet-wide rollout: scheduling, retries, status reporting. **Never sees** — the signing key, and does not need it - __mender-client__ --- *Verifier and installer* On each device: downloads, verifies the signature against the pre-provisioned public key, checks every file's hash, installs to the inactive partition, reboots. **Trusts** — one public key, provisioned once at manufacturing
> **Abstract: Core property — HSM-resident signing key** > The RSA private key that authenticates every OTA update stays inside the > HSM and is never exportable. What crosses the wire is only the signature > over the artifact manifest — a few hundred bytes appended to a file inside > the `.mender` tar. Every device in the fleet verifies against a single > public key, provisioned once at manufacturing. ## What Mender is, and is not Mender is an open-source OTA firmware update system for embedded Linux and RTOS devices. It solves the **delivery** problem: getting a signed artifact from a server to a fleet, with rollback, deployment scheduling, retry logic and status reporting. It does not generate signing keys and does not run an HSM. That is the fit. Mender handles distribution and installation; LAAVAT handles signing. The two meet at a single well-defined file — a base64-encoded signature (`manifest.sig`) inside the outer tar of every `.mender` artifact. As long as that signature verifies against the public key the client was provisioned with, the update is trusted. > **Info: One external signer among several** > Mender's `mender-artifact` tool natively supports several external key > backends, including PKCS#11. The digest-signing approach described here is > another external-signer pattern in that family. What differs is how the > private key is reached — not what gets signed, and not what the on-device > verification looks like. ## Inside a .mender artifact A `.mender` artifact is a plain, uncompressed tar with a strictly ordered layout. The signature covers the manifest, and the manifest covers every payload file by SHA-256. ```text firmware-v2.1.mender # outer tar (uncompressed) ├── version # format version (v3). Required first. ├── manifest # SHA-256 of every other file, sha256sum(1) format ├── manifest.sig # base64 signature over the manifest — what LAAVAT produces ├── manifest-augment # optional, for augmented/delta artifacts ├── header.tar[.gz|.xz|.zst] # artifact metadata: name, device types, update type ├── header-augment.tar[...] # optional, for augmented artifacts └── data/0000/ # the firmware payload, e.g. update.ext4 ``` **Order matters.** The format specification is explicit: `version` first, then `manifest`, then the optional `manifest.sig`, then the header files, then `data/`. Any non-data file appearing after a data file fails verification immediately. **Integrity cascades.** The manifest lists the SHA-256 of `header.tar.gz` and of every file under `data/`, formatted exactly as `sha256sum` output. Tamper with a payload and its recomputed hash will not match the manifest; tamper with the manifest and its signature will not verify. Signing one file secures the whole artifact. ## The LAAVAT product Mender signing needs no PKI hierarchy — the device holds a raw public key, not a certificate chain. One product with a single operation is enough.
- __Mender artifact signing__ --- `operationType: DigestSigning` Signs the SHA-256 hash of each artifact's manifest. You compute the digest locally and submit it; LAAVAT returns an RSA signature over that digest, which you base64-encode into `manifest.sig`. **Key** — RSA, SHA-256, PKCS#1 v1.5 padding
Mender recommends RSA of at least 3072 bits. ECDSA P-256 is also supported if you prefer EC keys — see [supported keys](https://docs.laavat.io/appendixes/supported-keys/) for the full matrix. The public half is exported once and copied to every device as `artifact-verify-key.pem`, referenced by `ArtifactVerifyKey` in the `mender-client` configuration. Manufacturing programs it in during first boot, or bakes it into the golden image. > **Note: Two different keys** > This page covers the **manufacturer's artifact-signing key**. Each Mender > client also carries its own **device-identity key pair**, used to > authenticate to the Mender server. That is a separate concern with a > separate lifecycle, generated on-device at first boot, and out of scope > here. ### Example product definition Group UUIDs are per-tenant; substitute your own. ```json { "name": "Mender OTA signing", "description": "Sign Mender .mender artifacts for OTA delivery", "productType": "Production", "enabled": true, "caInfo": [], "rndKeys": [], "productOperations": [ { "name": "Mender artifact signing", "description": "Sign the SHA-256 digest of a .mender manifest", "operationType": "DigestSigning", "token": { "name": "Mender signing key", "description": "RSA key for Mender manifest signing", "keyType": "RSA3072" }, "approvalRule": { "name": "Test rule", "description": "Rule used for testing", "allowedGroups": [""], "approvalGroups": [""], "blanketGroups": [] } } ] } ``` > **Note: Token handling** > The examples below pipe the token in on stdin with `-t @-`, which suits CI > secret managers. A config file created with `config-init` works equally > well — see > [secure token handling](https://docs.laavat.io/democlient/usage/#providing-the-token). ## Signing a release Six steps. **The firmware payload never leaves the build host** — only the manifest digest reaches LAAVAT. ### 1. Build the unsigned artifact Use Mender's own tool to package the firmware. This produces the tar with `version`, `manifest`, `header.tar.gz` and `data/0000`, but no `manifest.sig`. ```bash mender-artifact write rootfs-image \ --device-type \ --artifact-name \ --file rootfs.ext4 \ --output-path firmware.unsigned.mender ``` For non-rootfs updates the equivalent is `mender-artifact write module-image --type --file payload.tar.gz ...`. Either shape produces the same five-file outer tar. ### 2. Extract the manifest and hash it Untar just the manifest, compute its SHA-256, and base64-encode the raw 32 bytes for submission. ```bash mkdir -p work tar -xf firmware.unsigned.mender -C work/ manifest DIGEST_B64=$(sha256sum work/manifest | awk '{print $1}' \ | xxd -r -p | base64) ``` ### 3. Submit the digest ```bash REQ_ID=$(printf '%s' "$TOKEN" | signing-tool --json -c -t @- \ -a https://app.laavat.io//api/v1 \ imagesigning add DigestSigning \ -P \ --operid \ -p "$DIGEST_B64" \ -H SHA256 \ -N mender-artifact \ -D "Mender OTA signing" | jq -r '.id') ``` ### 4. Download the signature The request is asynchronous, moving from `Created` to `Ready`. Use `--wait` to block rather than polling by hand. ```bash printf '%s' "$TOKEN" | signing-tool -c -t @- \ -a https://app.laavat.io//api/v1 \ imagesigning get -I "$REQ_ID" --wait \ -O work/sig.bin ``` RSA-3072 produces 384 bytes; RSA-2048 produces 256, RSA-4096 produces 512. ### 5. Base64-encode into manifest.sig The filename must be exactly `manifest.sig` — that is what `mender-client` looks for inside the outer tar. ```bash base64 -w0 work/sig.bin > work/manifest.sig ``` ### 6. Repack the outer tar Explode the unsigned artifact and rebuild it with `manifest.sig` in the position `mender-artifact sign` would have placed it — between `manifest` and `header.tar.gz`. ```bash tar -xf firmware.unsigned.mender -C work/ tar -cf firmware.signed.mender -C work/ \ version manifest manifest.sig header.tar.gz data ``` The result is a fully signed `.mender`, equivalent to what `mender-artifact sign --key private.pem` would produce — except the private key never touched the build host. All six steps wrap into a shell script for CI to call on every release. Signing needs only `signing-tool`, `tar` and standard coreutils (`sha256sum`, `xxd`, `base64`), all present on any Linux build agent. ## What the device does at update time Once the artifact is downloaded, every step is offline: no further contact with LAAVAT, and none with the Mender server for the cryptography. 1. **Download the artifact.** Streamed rather than buffered — the client parses the tar on the fly and writes payload chunks straight to the inactive rootfs partition, or to an Update Module for non-rootfs updates. 2. **Verify the manifest signature.** Read `manifest.sig`, base64-decode it, and verify against the manifest bytes using the public key configured in `ArtifactVerifyKey`. **If this fails the update is aborted before anything is installed.** 3. **Verify every file's hash.** For each entry in the manifest, recompute the SHA-256 of the corresponding file and compare. Any mismatch aborts. This is what makes signing only the manifest sufficient. 4. **Install to the inactive partition.** The active partition is untouched throughout. 5. **Commit or roll back.** The bootloader is told to try the new partition on next boot. If the new firmware does not call `mender commit` within a configurable timeout, the bootloader falls back to the previous partition — automatic rollback on a bad update. ## Verifying without a device Every check the client performs is standard file and crypto handling, reproducible on any Linux host. Fetch the public key once: ```bash printf '%s' "$TOKEN" | signing-tool -c -t @- \ -a https://app.laavat.io//api/v1 \ product getpubkey \ -P \ --operid \ -O /tmp/mender_pub.pem ``` That is also the file to bake into every device's manufacturing image as `artifact-verify-key.pem`. ### Mender's own validator The authoritative check — the same code path the on-device client uses, run as a CLI: ```bash mender-artifact validate firmware-v2.1.mender \ --key /tmp/mender_pub.pem ``` Exits zero if the signature is valid and every file's hash matches. If this passes, a real device with the same public key provisioned would accept the artifact. ### Raw OpenSSL, for debugging ```bash tar -xf firmware-v2.1.mender manifest manifest.sig base64 -d manifest.sig > sig.bin openssl dgst -sha256 -verify /tmp/mender_pub.pem \ -signature sig.bin manifest ``` This bypasses `mender-artifact` entirely. Useful when a signature fails and you need to isolate whether the problem is the cryptography or the tar layout. ### Where off-device verification stops | Claim | Off-device | What would close the gap | | --- | --- | --- | | Manifest signature is cryptographically valid | Fully proven | — | | Every payload file's SHA-256 matches the manifest | Fully proven | — | | Tar layout matches what the client parses | Fully proven | `mender-artifact validate` enforces the layout | | The target device has the correct public key provisioned | Not testable off-device | Manufacturing workflow or first-boot provisioning | | An Update Module's install script behaves correctly on-target | Not tested | Integration test on real hardware or QEMU | | Rollback on a failed boot actually reverts | Out of scope | Bootloader integration test on real hardware | ## Related - [Secure boot and CRA](https://docs.laavat.io/cra-compliance/secure-boot-cra/) — the other half of the requirement: only running firmware you authorised - [CRA-ready nRF devices](https://docs.laavat.io/solutions/cra-ready-nrf/) — a secure boot chain using the same digest-signing primitive - [Annex I mapping](https://docs.laavat.io/cra-compliance/annex-i-mapping/) — where "distribute security updates securely" sits among the essential requirements - [Security and architecture](https://docs.laavat.io/security/) — where the keys live and who can reach them ## References - [Mender security overview](https://docs.mender.io/overview/security) - [Sign and verify Mender artifacts](https://docs.mender.io/artifact-creation/sign-and-verify) - [Mender artifact format v3](https://github.com/mendersoftware/mender-artifact/blob/master/Documentation/artifact-format-v3.md)