Release Pipeline

Mero TEE ships two coupled artifact families from one monorepo, both frozen GCP TDX images sealed behind dm-verity: the mero-tee node image and the mero-kms KMS image (merotee-kms-<profile>-<version>), which runs as one cluster per release. Each has its own release workflow, but they share a version number and a measurement contract: the KMS image bakes in the node image’s measurements, and publishes its own for merod to pin. This page traces the real pipeline: bump → build → measure → sign → publish → catalog.

Note

Releases are not tag-triggered. Both release workflows run on push to master (path-filtered) and on manual workflow_dispatch; they create the mero-kms-v<version> / mero-tee-v<version> GitHub release tags themselves as their final step.

Version coupling

The KMS version and the node-image version move together. The release version sync guard (release-version-sync-guard.yaml → scripts/policy/check_release_version_sync.sh) fails a PR unless all three agree:

• mero-kms/Cargo.toml package.version
• mero-tee/versions.json imageVersion
• the mero-kms entry in Cargo.lock

versions.json merodVersion is decoupled — it pins the calimero-network/core merod tag baked into the node image.

KMS release — release-kms.yaml

Workflow name Release mero-kms (the name is part of the Sigstore identity merod checks, so it does not change). Triggers: push to master touching mero-kms/, Cargo.toml, Cargo.lock, playbook-kms.yml, the mero-kms Ansible role, scripts/attestation/**, scripts/release/kms/** or the workflow itself; plus workflow_dispatch. It publishes mero-kms-v<version> only from a push to master, and only when that tag does not exist yet. merod accepts a policy only when a push-triggered run signed it, so a manual dispatch builds nothing; re-run a failed release’s jobs instead, which keeps the original push event.

1. prepare — resolve the version and the release tags (scripts/release/kms/get-version-info.sh).

2. node-policy — wait for the counterpart node release mero-tee-v<version> and fetch its published-mrtds.json (wait-for-node-release.sh). Those are the measurements the KMS will release keys to.

3. build-binary — build the mero-kms binary (x86_64) from the release commit.

4. kms-image (matrix over the three profiles) — build the KMS image. Packer with image_role = "kms" runs mero-tee/playbook-kms.yml: the binary, the profile’s node allowlist from published-mrtds.json baked into /etc/mero-kms/kms.env, the same lockdown as the node image, and the dm-verity seal as the last step. The allowlist is part of the image and so of its measurements. The image is merotee-kms-<profile>-<version> in family merotee-kms-<profile>, and the release keeps it.

   Then measure it (run-cluster-probe.sh): boot it as a two-replica cluster on real TDX VMs (the same check as the KMS TDX image probe). The bootstrap replica generates a root, the second joins, both must report the same transport key, and Intel Trust Authority verifies both quotes. Their MRTD and RTMR0–3 are the KMS measurements the release publishes. An image that is not proven this way is deleted.

   The probe measures each joiner before waiting on its join. Now and then a TD on GCE comes up with a different RTMR0 while MRTD and RTMR1–3 are equal, and the bootstrap rightly refuses it the root. The probe then prints the event log events that differ (tdx_event_log.py diff), deletes that VM and starts another, as the runbooks replace a replica that cannot join. It fails only if no joiner in MAX_JOINER_ATTEMPTS (3) measures like the bootstrap. The measurements themselves are never relaxed.

   Measurements are observed; the TCB allowlist is declared

   MRTD and RTMR0–3 are necessarily taken from the probe’s own quote — they are the thing being published. The accepted TCB statuses are not: they come from --allowed-tcb-status (default uptodate,outofdate), never from what the probe reported. outofdate is in the default as a deliberate, reviewable decision — both images run on GCP Confidential VMs, whose host firmware Calimero cannot patch, and they currently report OutOfDate. Drop it once the probes report uptodate. Deriving them from the probe is how releases 2.3.48–2.3.52 each shipped allowed_tcb_statuses: ["outofdate"] — a policy that accepted only a stale host and would have rejected every patched one, so the fleet would have broken on being updated. When the probe’s own status falls outside the allowed set the run emits a warning naming it; pass --require-observed-allowed to turn that into a hard failure once the probe fleet reports uptodate.

   The post-release E2E enforces the same distinction: MRTD/RTMR must match the release policy exactly, while the probe’s TCB status need only be accepted by it. Asserting equality there was the third place that treated an observation as the spec, and it would have rejected any corrected policy.

5. release-metadata — assemble (assemble-release-assets.sh): the per-profile attestation policy (kms_allowed_* from the measurement, node_allowed_* from the node release), the compatibility map, the release manifest, the checksums and the SBOM (Syft). Sign every asset with cosign keyless (Sigstore / Fulcio / Rekor), producing .sig, .pem and .bundle.json sidecars, verify the signatures, and publish the GitHub release mero-kms-v<version>.

KMS release assets

Uploaded to the mero-kms-v<version> release, each trust asset with .sig, .pem and .bundle.json Sigstore sidecars:

• kms-attestation-policy.<profile>.json (one per profile) and the profile-less kms-attestation-policy.json (the locked-read-only one, which merod fetches by default). It pins the KMS by kms_allowed_mrtd, kms_allowed_rtmr0..3 and kms_allowed_tcb_statuses, and its merod_config_path is tee.kms.attestation.
• kms-compatibility-map.json — the KMS release → node release, the per-profile images, policy URLs and hashes.
• kms-release-manifest.json, kms-checksums.txt (SHA-256 of the binary archives), kms-binaries-sbom.spdx.json, kms-rekor-index.json, and kms-trust-bundle.tar.gz (all of the above, for offline transfer).

The KMS images themselves live in the images project, named merotee-kms-<profile>-<version>; MDMA deploys clusters from them.

Caution

The policy is generated from live measurements of the image at release time; it is never hand-populated. The node allowlist inside it is a copy of what the image already bakes in: the KMS never fetches a policy at runtime, and nothing outside the image can change what it releases keys to.

Node-image release — release-node-image-gcp.yaml

Workflow name Release mero-tee. Triggers: push to master touching mero-tee/versions.json; plus workflow_dispatch.

1. preflight — a release gate that resolves the target merod version and verifies the calimero-network/core release tag (and its required tarball assets) actually exists, so the image build is blocked until core ships. Reads imageVersion from versions.json.

2. profile_pipeline (matrix over the three profiles) — the measurement step, which boots a real TDX VM:

   • Run build-and-release.sh <profile> (Packer + Ansible) to build the GCP image. Its last step seals the root behind dm-verity, and it checks the files the image must carry twice: on the build host before the seal, and inside the sealed EROFS image after it (seal-root.sh --require). Today that is GCE’s disk-naming rule and google_nvme_id, which name the data disk /dev/disk/by-id/google-data; a missing one fails the build.
   • Create an ephemeral firewall rule and a c3-standard-4 Intel TDX VM from the new image, with the same 200 GB pd-balanced data disk (device name data) mdma attaches to every node at creation. The disk is part of the machine RTMR0 measures: an image measured without it publishes an RTMR0 no mdma node has, and the KMS refuses every node’s key request. The staging probe, which the post-release check uses, boots the same shape.
   • Collect TEE info and a TDX quote from the node’s admin API, then verify the quote externally against Intel Trust Authority.
   • Emit mrtd-<profile>.json (MRTD + RTMR0–3), a provenance record, and the attestation artifacts.

3. cleanup_attestation_resources — tear down the ephemeral VM and firewall rule.

4. publish_mrtds — assemble published-mrtds.json (per-profile measurements) with assemble-published-mrtd-payload.sh, generate the SBOM, cosign-sign the assets (.sig + .pem + .bundle.json), and publish the GitHub release mero-tee-v<version>.

Dispatching from a branch builds and measures, and publishes nothing. Publishing runs only on master. Off master, images are named …-<version>-d<run number>-<attempt> in a separate merotee-ubuntu-questing-<profile>-dev family (image_suffix in mero-tee/ubuntu.pkr.hcl). Packer never replaces an existing image for such a build, and the image is deleted at the end of the run. mdma’s dispatcher resolves images by exact name and exact family, so it never boots a branch image. Before this, a branch dispatch at the current imageVersion replaced the released image of the same name.

KMS TDX image probe — kms-tdx-image-probe.yaml

Manual only (workflow_dispatch), and it publishes nothing. It builds the mero-kms TDX cluster image (design; image_role = "kms", playbook-kms.yml) from the dispatched commit, with the node allowlist of a published node release (node_release_tag) baked in. It then runs a two-replica cluster on TDX VMs: one started with kms-bootstrap=true, the other with kms-peers naming the first. It checks that both come to hold the root and report the same transport key (derived from the root, so equal keys mean one root), and that Intel Trust Authority verifies both quotes with identical measurements; a joiner that measures differently is replaced by a new VM, up to three times, as described above. It deletes its image, VMs and firewall rule when done. Run it to prove a change to the image before the Release mero-kms workflow builds and measures it for real. Like branch node builds, its image carries a suffix and its own family, so it can never be mistaken for a release.

Node-image release assets

Uploaded to mero-tee-v<version> (each with .sig, .pem and .bundle.json sidecars; releases before the bundles were added carry only .sig and .pem). The tag points at the commit the run built the image from, the same commit release-provenance.json records as commit_sha. Releases up to 2.3.78 were tagged at the default branch’s tip when the upload ran instead, which is a later commit if master moved during the build (mero-tee-v2.3.78 is on 8612ba4, its image was built from 08f6344); trust commit_sha for those.

• published-mrtds.json — expected MRTD/RTMR0–3 for each profile. A namespace whose TEE policy trusts signed releases has its admitters download this file and its .bundle.json and verify the signature against the Release mero-tee workflow identity before comparing a joining TEE’s quote with it.
• release-provenance.json
• node-image-gcp-release-sbom.spdx.json
• node-image-gcp-checksums.txt

Caution

The two families cross-reference each other: the KMS release waits for the node release’s published-mrtds.json and bakes each profile’s node measurements into that profile’s KMS image, and the KMS measurements plus the node measurements together form each profile’s published attestation policy. A node image built for the debug family cannot obtain keys from a locked-read-only KMS because their RTMR3 profile measurements differ.

Compatibility catalog

compatibility-catalog.json is the source of truth for which KMS and node-image versions pair. It is rebuilt by update-compatibility-catalog.yaml on every release: published event (and on demand), by enumerating published mero-kms-v* tags and pairing each with the identically-versioned node tag:

{
  "schema_version": 1,
  "releases": [
    {
      "version": "2.3.75",
      "kms_tag": "mero-kms-v2.3.75",
      "node_image_tag": "mero-tee-v2.3.75",
      "kms_policy_url": "https://github.com/calimero-network/mero-tee/releases/download/mero-kms-v2.3.75/kms-attestation-policy.json",
      "node_policy_url": "https://github.com/calimero-network/mero-tee/releases/download/mero-tee-v2.3.75/published-mrtds.json"
    }
  ]
}

Caution

The catalog schema is { "schema_version": 1, "releases": [...] } keyed by version + release tags + asset URLs. There are no entries, kms, node_image, profiles, tested, or notes fields — pairing is by identical version number, computed from the published tags, not hand-curated. Entries for releases published before the policy asset was renamed keep the URLs they were published with.

Which token writes releases

Every step that creates or edits a GitHub release uses secrets.RELEASE_TOKEN || secrets.GHCR_PUSH_TOKEN || secrets.GITHUB_TOKEN: the asset uploads, Enforce release notes body update in Release mero-kms, and Upsert umbrella version release links in both release workflows. Steps that only read releases (smoke tests, the auditor, the catalog) keep the Actions github.token.

The umbrella steps used github.token until 2.3.75. From 2.3.74 on that token was refused (HTTP 403: Resource not accessible by integration on POST /releases) while the uploads, on the release token, still worked, so the component releases (mero-tee-v<ver>, mero-kms-v<ver>) were published but the run went red and the umbrella <ver> index release was never created. The component releases are what merod and MDMA read; the umbrella is an index.

A re-run replays the workflow as it was at its commit, so it does not pick up this fix. For a release cut before it (2.3.74, 2.3.75), create the umbrella once by hand, with a token that can write releases:

GH_TOKEN=<release token> GITHUB_REPOSITORY=calimero-network/mero-tee \
  VERSION=2.3.75 TARGET_COMMIT=<release commit> \
  bash scripts/release/node-image-gcp/upsert-umbrella-release-links.sh

Guards and audits

Workflow	Trigger	What it enforces
release-version-sync-guard.yaml	PR / push to master	KMS Cargo.toml, versions.json, and Cargo.lock versions agree.
docs-update-guard.yaml	PR	Changes under .github/workflows/, scripts/, or mero-tee/** ship with a docs/** / README.md / CHANGELOG.md update.
ci-kms.yaml	PR / push (KMS paths)	cargo fmt + clippy -D warnings + tests on mero-kms.
security-audit.yaml	Weekly (Mon 04:00) / PR	cargo audit against the advisory database.
release-auditor.yaml	Weekly (Mon 03:30) / dispatch	Re-verifies recent releases’ assets and cosign signatures.
post-release-mero-tee-node-e2e.yaml	workflow_run after a successful Release mero-tee run on master of this repository (push or dispatch)	Boots fresh TDX VMs and checks their measurement candidates are covered by published-mrtds.json — without waiting on the KMS.

When a post-release probe fails

The post-release workflow runs after publication, triggered by workflow_run. A workflow_run result is attached to nothing a release consumer reads — there is no pull request, it does not appear on the release page, and it sets no commit status on the tag. Left alone, a failure against a published artifact is one red row in the Actions tab.

So it ends with a report_failure job that opens a GitHub issue naming the release, the failing job, and the run (scripts/ci/report_post_release_failure.py). Repeat failures for the same release comment on the existing issue instead of opening another, matched on a marker in the issue body rather than on its title.

Note

The issue is not closed automatically when a later run goes green. The published artifact does not change when CI is re-run, so “it passed the second time” is a finding for a human to weigh, not a resolution.

Two limits worth knowing: a failure before the release is resolved (a bad checkout, say) files nothing, because at that point the run has no release to name and the problem is CI’s rather than the release’s; and if issue creation itself fails the step warns rather than failing, so a permissions problem cannot mask the probe failure underneath it.

Act on the issue quickly. The probe diagnostics are uploaded with a short retention-days, so the artifact naming the failed assertion expires within days, while the release it concerns stays published indefinitely. The issue outlives the evidence.

A replica without its root (policy_not_ready)

A KMS replica that has not yet generated (bootstrap) or joined its cluster’s root answers /get-key with 503 policy_not_ready; /attest and /health still answer, and /health reports clusterRootReady: false.

Danger

policy_not_ready does not relax verification. It blocks key release entirely. There is no runtime policy source to be “not ready” any more: the node allowlist is baked into the image, and the only thing a replica can lack is the root.

Next steps

• Configuration reference — the versions and env vars this pipeline consumes.
• Runbooks — deploying and probing a published release.
• Policy management — how the attestation policy governs key release.