Release Pipeline

Mero TEE ships two coupled artifact families from one monorepo, both frozen GCP TDX images sealed behind dm-verity: the mero-tee node image and the mero-kms KMS image (merotee-kms-<profile>-<version>), which runs as one cluster per release. Each has its own release workflow, but they share a version number and a measurement contract: the KMS image bakes in the node image’s measurements, and publishes its own for merod to pin. This page traces the real pipeline: bump → build → measure → sign → publish → catalog.

Note

Releases are not tag-triggered. Both release workflows run on push to master (path-filtered) and on manual workflow_dispatch; they create the mero-kms-v<version> / mero-tee-v<version> GitHub release tags themselves as their final step.

Version coupling

The KMS version and the node-image version move together. The release version sync guard (release-version-sync-guard.yaml → scripts/policy/check_release_version_sync.sh) fails a PR unless all three agree:

• mero-kms/Cargo.toml package.version
• mero-tee/versions.json imageVersion
• the mero-kms entry in Cargo.lock

versions.json merodVersion is decoupled — it pins the calimero-network/core merod tag baked into the node image.

KMS release — release-kms.yaml

Workflow name Release mero-kms (the name is part of the Sigstore identity merod checks, so it does not change). Triggers: push to master touching mero-kms/, Cargo.toml, Cargo.lock, playbook-kms.yml, the mero-kms Ansible role, scripts/attestation/**, scripts/release/kms/** or the workflow itself; plus workflow_dispatch. It publishes mero-kms-v<version> only from a push to master, and only when that tag does not exist yet. merod accepts a policy only when a push-triggered run signed it, so a manual dispatch builds nothing; re-run a failed release’s jobs instead, which keeps the original push event.

1. prepare — resolve the version and the release tags (scripts/release/kms/get-version-info.sh).

2. node-policy — wait for the counterpart node release mero-tee-v<version> and fetch its published-mrtds.json (wait-for-node-release.sh). Those are the measurements the KMS will release keys to, so the file’s Sigstore bundle must verify against release-node-image-gcp.yaml on master (verify-node-policy.sh): editing the GitHub release does not change them.

3. build-binary — build the mero-kms binary (x86_64) from the release commit.

4. kms-image (matrix over the three profiles) — build the KMS image with mkosi (ROLE=kms, scripts/release/mkosi/build-image.sh), from mero-tee/playbook-kms.yml’s roles: the binary, the profile’s node allowlist from published-mrtds.json baked into /etc/mero-kms/kms.env (extract-node-allowlist.sh takes only 96-hex measurements and known TCB statuses, so no value can add a line or an entry to kms.env), the same lockdown as the node image, and an EROFS root behind dm-verity. The allowlist is part of the image and so of its measurements. The image is merotee-kms-<profile>-<version> in family merotee-kms-<profile>, the names mdma looks up, and the release keeps it.

   Then measure it (run-cluster-probe.sh): boot it as a two-replica cluster on real TDX VMs (the same check as a branch build with role: kms). The bootstrap replica generates a root, the second joins, both must report the same transport key, and Intel Trust Authority verifies both quotes. Their MRTD and RTMR0–3 are the KMS measurements the release publishes. The bootstrap replica’s RTMR1–3 must also be the ones compute_rtmr.py computed from the build before it booted. An image that is not proven this way is deleted.

   On debug and debug-read-only, check the KMS ships its logs: the bootstrap replica boots with logs-endpoint and observability-token pointed at a throwaway receiver VM, and the job fails unless the receiver sees a push carrying that token (the log-auth check, as in the node release). Neither key is measured. locked-read-only skips it: it ships logs only over https, and the receiver is plain http.

   The probe measures each joiner before waiting on its join. Now and then a TD on GCE comes up with a different RTMR0 while MRTD and RTMR1–3 are equal, and the bootstrap rightly refuses it the root. The probe then prints the event log events that differ (tdx_event_log.py diff), deletes that VM and starts another, as the runbooks replace a replica that cannot join. It fails only if no joiner in MAX_JOINER_ATTEMPTS (3) measures like the bootstrap. The measurements themselves are never relaxed.

   Measurements are observed; the TCB allowlist is declared

   MRTD and RTMR0–3 are necessarily taken from the probe’s own quote — they are the thing being published. The accepted TCB statuses are not: they come from --allowed-tcb-status (default uptodate,outofdate), never from what the probe reported. outofdate is in the default as a deliberate, reviewable decision — both images run on GCP Confidential VMs, whose host firmware Calimero cannot patch, and they reported OutOfDate. Drop it once the probes report uptodate. Since 2026-10-15 the probe hosts report UpToDate (Intel TCB date 2026-08-12). The fleet’s hosts are patched in place, though, and some may still report OutOfDate. Until all of them report UpToDate, every probe call site passes --keep-outofdate, which lets a release keep accepting outofdate past the check that otherwise stops it. Deriving them from the probe is how releases 2.3.48–2.3.52 each shipped allowed_tcb_statuses: ["outofdate"] — a policy that accepted only a stale host and would have rejected every patched one, so the fleet would have broken on being updated. When the probe’s own status falls outside the allowed set the run emits a warning naming it; pass --require-observed-allowed to turn that into a hard failure once the probe fleet reports uptodate.

   The post-release E2E enforces the same distinction: MRTD/RTMR must match the release policy exactly, while the probe’s TCB status need only be accepted by it. Asserting equality there was the third place that treated an observation as the spec, and it would have rejected any corrected policy.

5. release-metadata — assemble (assemble-release-assets.sh): the per-profile attestation policy (kms_allowed_* from the measurement, node_allowed_* from the node release), the compatibility map, the release manifest, the checksums and the SBOM (Syft). Sign every asset with cosign keyless (Sigstore / Fulcio / Rekor), producing .sig, .pem and .bundle.json sidecars, verify the signatures, and publish the GitHub release mero-kms-v<version>.

Agents in the KMS release

Private agents ride the KMS release, best effort. Alongside the node-policy wait, agent-image builds the reference agent image for each profile: mero-agent-gate plus the reference agent, named merotee-agent-<profile>-<version>. The reference agent is mero-bot when versions.json pins meroBotVersion: the job downloads that release’s bundle with the MERO_BOT_RELEASE_TOKEN secret (mero-bot is a private repository) and checks it against its sha256 in build-inputs.lock.json. Without a pin, it is the stub agent. It boots that image once with ephemeral-store=true and has Intel Trust Authority measure it. On debug and debug-read-only that VM also gets the two log keys, and the job fails unless it pushed its logs with its token, as the KMS image must (the log-auth check). A measured profile’s KMS image then also bakes that agent allowlist (kms_agent_policy_file, with the node allowlist’s TCB statuses, held to the same shapes by extract-node-allowlist.sh --check and the mero-kms role). Release-metadata publishes it as kms-agent-attestation-policy.<profile>.json, signed like the other assets.

If an agent build or measurement fails, nothing else changes: that profile’s KMS image is built node-only, exactly as before agents, and the release goes on. kms-image and release-metadata check their real dependencies’ results explicitly for this. Every profile gets an agent image. None bakes a provisioner: each running agent is claimed by its first owner (owner claim), so no repository variable or secret decides who sets an agent’s secrets.

After a successful release, Post-release agent e2e (post-release-agent-e2e.yaml) proves the chain on fresh TDX VMs:

1. It boots a one-replica KMS from the release image, and the agent image with a blank data disk pointed at it.
2. merod verifies that KMS against the signed release policy, and the KMS releases the agent’s disk key under the mero-agent/ prefix.
3. It verifies the gate against the signed agent policy, claims the agent with a throwaway owner key made on the runner, provisions a secret, and checks a second key is refused.
4. It resets the agent and checks the disk reopens with the same signing key and owner, the owner provisions again, and the second key is still refused.

It runs for every profile the release published an agent policy for, and skips the others.

KMS release assets

Uploaded to the mero-kms-v<version> release, each trust asset with .sig, .pem and .bundle.json Sigstore sidecars:

• kms-attestation-policy.<profile>.json (one per profile) and the profile-less kms-attestation-policy.json (the locked-read-only one, which merod fetches by default). It pins the KMS by kms_allowed_mrtd, kms_allowed_rtmr0..3 and kms_allowed_tcb_statuses, and its merod_config_path is tee.kms.attestation.
• kms-compatibility-map.json — the KMS release → node release, the per-profile images, policy URLs and hashes.
• kms-release-manifest.json, kms-checksums.txt (SHA-256 of the binary archives), kms-binaries-sbom.spdx.json, kms-rekor-index.json, and kms-trust-bundle.tar.gz (all of the above, for offline transfer).
• kms-agent-attestation-policy.<profile>.json, for each profile whose KMS image serves agents: the agent allowlist that image bakes in, which a provisioner or account owner verifies an agent’s quote against.

The KMS images themselves live in the images project, named merotee-kms-<profile>-<version>; MDMA deploys clusters from them.

Caution

The policy is generated from live measurements of the image at release time; it is never hand-populated. The node allowlist inside it is a copy of what the image already bakes in: the KMS never fetches a policy at runtime, and nothing outside the image can change what it releases keys to.

Node-image release — release-node-image-gcp.yaml

Workflow name Release mero-tee. Triggers: push to master touching mero-tee/versions.json; plus workflow_dispatch.

1. preflight — a release gate that resolves the target merod version and verifies the calimero-network/core release tag (and its required tarball assets) actually exists, so the image build is blocked until core ships. Reads imageVersion from versions.json.

2. profile_pipeline (matrix over the three profiles) — the measurement step, which boots a real TDX VM:

   • Build the image with mkosi (mero-tee/mkosi-image, mdma#480) through scripts/release/mkosi/build-image.sh, as root: an EROFS root behind dm-verity, booted as a UKI, from the pinned archive snapshot. The build checks the UKI’s command line, the image’s files (GCE’s disk-naming rule and google_nvme_id among them) and that files keep their owners, and computes RTMR1–3 from the disk and UKI before anything boots (compute_rtmr.py). Two builds of one commit give the same image (see Build the images with mkosi). create-gce-image.sh creates it under the name, family and description mdma finds images by (release-image-name.sh).
   • Create an ephemeral firewall rule and a c3-standard-4 Intel TDX VM from the new image, with the same 200 GB pd-balanced data disk (device name data) mdma attaches to every node at creation. The disk is part of the machine RTMR0 measures: an image measured without it publishes an RTMR0 no mdma node has, and the KMS refuses every node’s key request. The staging probe, which the post-release check uses, boots the same shape.
   • Collect TEE info and a TDX quote from the node’s admin API, then verify the quote externally against Intel Trust Authority.
   • On debug and debug-read-only, check the node ships its logs: it boots with logs-endpoint and observability-token pointed at a throwaway receiver VM, and the job fails unless the receiver sees a push carrying that token (the log-auth check). Neither key is measured. locked-read-only skips it: it ships logs only over https, and the receiver is plain http. The debug profiles share its vector role, config and token chain.
   • Compare the quote’s RTMR1–3 with the computed ones; a difference fails the release. The computed registers (computed-rtmrs-<profile>.json) and the build evidence go out with the attestation artifacts.
   • Emit mrtd-<profile>.json (MRTD + RTMR0–3), a provenance record, and the attestation artifacts.

3. cleanup_attestation_resources — tear down the ephemeral VMs and firewall rules, the log-auth receivers among them.

4. publish_mrtds — assemble published-mrtds.json (per-profile measurements) with assemble-published-mrtd-payload.sh, generate the SBOM, cosign-sign the assets (.sig + .pem + .bundle.json), and publish the GitHub release mero-tee-v<version>.

Dispatching from a branch builds and measures, and publishes nothing. Publishing runs only on master. Off master, images are named …-<version>-d<run number>-<attempt> in a separate merotee-ubuntu-questing-<profile>-dev family (release-image-name.sh). Such a build never replaces an existing image, and the image is deleted at the end of the run. mdma’s dispatcher resolves images by exact name and exact family, so it never boots a branch image. Before this, a branch dispatch at the current imageVersion replaced the released image of the same name.

Branch builds — node-image-mkosi.yaml

Manual only (workflow_dispatch), and it publishes nothing. It builds one image (role: node, kms or agent) for one profile from the dispatched commit, as the releases do, boots it on TDX and runs that role’s probe:

• node: the staging probe.
• kms: a two-replica cluster, with the node allowlist of a published node release (node_release_tag) baked in, verified and extracted as the release does. It checks that both replicas come to hold the root and report the same transport key, and that Intel Trust Authority verifies both quotes with identical measurements.
• agent: Intel Trust Authority, then a throwaway secret provisioned with a throwaway owner key. The agent is the stub, or with agent: mero-bot the mero-bot release versions.json pins (needs the MERO_BOT_RELEASE_TOKEN secret).

It then compares the quote’s RTMR1–3 with the values computed from the build. With log_auth_check on a debug profile, it also runs the releases’ log-auth check for any role; it is the way to try that check on a KMS or agent image from a branch. Its image carries a suffix and its own family, so it can never be mistaken for a release, and it is deleted afterwards unless keep_image is set. A KMS that serves an agent is built only by the release. See Build the images with mkosi.

Node-image release assets

Uploaded to mero-tee-v<version> (each with .sig, .pem and .bundle.json sidecars; releases before the bundles were added carry only .sig and .pem). The tag points at the commit the run built the image from, the same commit release-provenance.json records as commit_sha. Releases up to 2.3.78 were tagged at the default branch’s tip when the upload ran instead, which is a later commit if master moved during the build (mero-tee-v2.3.78 is on 8612ba4, its image was built from 08f6344); trust commit_sha for those.

• published-mrtds.json — expected MRTD/RTMR0–3 for each profile. A namespace whose TEE policy trusts signed releases has its admitters download this file and its .bundle.json and verify the signature against the Release mero-tee workflow identity before comparing a joining TEE’s quote with it.
• release-provenance.json
• node-image-gcp-release-sbom.spdx.json
• node-image-gcp-checksums.txt

Caution

The two families cross-reference each other: the KMS release waits for the node release’s published-mrtds.json and bakes each profile’s node measurements into that profile’s KMS image, and the KMS measurements plus the node measurements together form each profile’s published attestation policy. A node image built for the debug family cannot obtain keys from a locked-read-only KMS because their RTMR3 profile measurements differ.

Compatibility catalog

compatibility-catalog.json is the source of truth for which KMS and node-image versions pair. It is rebuilt by update-compatibility-catalog.yaml on every release: published event (and on demand), by enumerating published mero-kms-v* tags and pairing each with the identically-versioned node tag:

{
  "schema_version": 1,
  "releases": [
    {
      "version": "2.3.75",
      "kms_tag": "mero-kms-v2.3.75",
      "node_image_tag": "mero-tee-v2.3.75",
      "kms_policy_url": "https://github.com/calimero-network/mero-tee/releases/download/mero-kms-v2.3.75/kms-attestation-policy.json",
      "node_policy_url": "https://github.com/calimero-network/mero-tee/releases/download/mero-tee-v2.3.75/published-mrtds.json"
    }
  ]
}

Caution

The catalog schema is { "schema_version": 1, "releases": [...] } keyed by version + release tags + asset URLs. There are no entries, kms, node_image, profiles, tested, or notes fields — pairing is by identical version number, computed from the published tags, not hand-curated. Entries for releases published before the policy asset was renamed keep the URLs they were published with.

Which token writes releases

Every step that creates or edits a GitHub release uses secrets.RELEASE_TOKEN || secrets.GHCR_PUSH_TOKEN || secrets.GITHUB_TOKEN: the asset uploads, Enforce release notes body update in Release mero-kms, and Upsert umbrella version release links in both release workflows. Steps that only read releases (smoke tests, the auditor, the catalog) keep the Actions github.token.

The umbrella steps used github.token until 2.3.75. From 2.3.74 on that token was refused (HTTP 403: Resource not accessible by integration on POST /releases) while the uploads, on the release token, still worked, so the component releases (mero-tee-v<ver>, mero-kms-v<ver>) were published but the run went red and the umbrella <ver> index release was never created. The component releases are what merod and MDMA read; the umbrella is an index.

A re-run replays the workflow as it was at its commit, so it does not pick up this fix. For a release cut before it (2.3.74, 2.3.75), create the umbrella once by hand, with a token that can write releases:

GH_TOKEN=<release token> GITHUB_REPOSITORY=calimero-network/mero-tee \
  VERSION=2.3.75 TARGET_COMMIT=<release commit> \
  bash scripts/release/node-image-gcp/upsert-umbrella-release-links.sh

Guards and audits

Workflow	Trigger	What it enforces
release-version-sync-guard.yaml	PR / push to master	KMS Cargo.toml, versions.json, and Cargo.lock versions agree.
docs-update-guard.yaml	PR	Changes under .github/workflows/, scripts/, or mero-tee/** ship with a docs/** / README.md / CHANGELOG.md update.
ci-kms.yaml	PR / push (KMS paths)	cargo fmt + clippy -D warnings + tests on mero-kms.
security-audit.yaml	Weekly (Mon 04:00) / PR	cargo audit against the advisory database.
release-auditor.yaml	Weekly (Mon 03:30) / dispatch	Re-verifies recent releases’ assets and cosign signatures.
post-release-mero-tee-node-e2e.yaml	workflow_run after a successful Release mero-tee run on master of this repository (push or dispatch)	Boots fresh TDX VMs and checks their measurement candidates are covered by published-mrtds.json — without waiting on the KMS.

When a post-release probe fails

The post-release workflow runs after publication, triggered by workflow_run. A workflow_run result is attached to nothing a release consumer reads — there is no pull request, it does not appear on the release page, and it sets no commit status on the tag. Left alone, a failure against a published artifact is one red row in the Actions tab.

So it ends with a report_failure job that opens a GitHub issue naming the release, the failing job, and the run (scripts/ci/report_post_release_failure.py). Repeat failures for the same release comment on the existing issue instead of opening another, matched on a marker in the issue body rather than on its title.

Note

The issue is not closed automatically when a later run goes green. The published artifact does not change when CI is re-run, so “it passed the second time” is a finding for a human to weigh, not a resolution.

Two limits worth knowing: a failure before the release is resolved (a bad checkout, say) files nothing, because at that point the run has no release to name and the problem is CI’s rather than the release’s; and if issue creation itself fails the step warns rather than failing, so a permissions problem cannot mask the probe failure underneath it.

Act on the issue quickly. The probe diagnostics are uploaded with a short retention-days, so the artifact naming the failed assertion expires within days, while the release it concerns stays published indefinitely. The issue outlives the evidence.

A replica without its root (policy_not_ready)

A KMS replica that has not yet generated (bootstrap) or joined its cluster’s root answers /get-key with 503 policy_not_ready; /attest and /health still answer, and /health reports clusterRootReady: false.

Danger

policy_not_ready does not relax verification. It blocks key release entirely. There is no runtime policy source to be “not ready” any more: the node allowlist is baked into the image, and the only thing a replica can lack is the root.

Next steps

• Configuration reference — the versions and env vars this pipeline consumes.
• Runbooks — deploying and probing a published release.
• Policy management — how the attestation policy governs key release.