Skip to content
Calimero — Mero TEECalimero — Mero TEEMero TEE

Runbooks

Concrete, ordered procedures for running Mero TEE. They reference the real scripts and workflows in this repository; confirm exact flags against --help for your checked-out version before running against production.

The model in one paragraph

Section titled “The model in one paragraph”

Each release ships a node image and a KMS image, both frozen GCP TDX images sealed behind dm-verity. The KMS image (merotee-kms-<profile>-<version>, family merotee-kms-<profile>) runs as one cluster per release: five replicas across three zones in at least two regions, behind one VPC-internal URL. The cluster’s root key is generated in RAM by the bootstrap replica and reaches the others only by attested join. It is never written anywhere and never backed up. Nothing in a running cluster is ever reconfigured, patched or restarted in place: an upgrade is a new node image, a new KMS image, a new cluster and new keys; a dead replica is replaced by a new VM that joins. MDMA runs all of these procedures; the steps below say what it does, so you can check it or do it by hand in staging. The design has the rationale.

Deploy a release’s KMS cluster

Section titled “Deploy a release’s KMS cluster”

  1. Verify the release assets first. Never deploy an unverified artifact:

    Terminal window
    scripts/release/verify-kms-release-assets.sh mero-kms-v<version>

    This checks the checksums and the cosign Sigstore bundles for the KMS release assets, including every kms-attestation-policy[.<profile>].json.

  2. Start the bootstrap replica. Create one TDX VM (c3-standard-4, --confidential-compute-type=TDX, --maintenance-policy=TERMINATE) from the release’s image merotee-kms-<profile>-<version>, on the VPC only (no external address), with instance metadata kms-bootstrap=true. It generates a random 32-byte root inside the TD. Wait until its /health reports clusterRootReady: true:

    Terminal window
    curl -s http://<replica-ip>:8080/health
    # {"status":"alive","service":"mero-kms","clusterRootReady":true}

  3. Start the other four replicas in the remaining zones and regions (five in total, three zones, at least two regions), each with kms-peers set to the base URLs of the replicas already ready, e.g. kms-peers=http://10.0.0.5:8080. Each one asks a peer for a nonce, sends its own quote, and receives the root sealed to a one-time key once the peer has checked that the quote carries exactly its own MRTD and RTMR0–3 and is not a debug TD. The joiner checks the giver the same way. Wait for clusterRootReady: true on every replica.

  4. Clear kms-bootstrap on the first replica (set it to false). It already holds the root and reads the key only at boot, but no VM that could ever boot again should carry kms-bootstrap=true: a second bootstrap is a second root.

  5. Put one VPC-internal URL in front of all five: a name in a Cloud DNS private zone whose A record lists every ready replica’s internal IP (TTL 30s), on port 8080. The replicas span regions, so no regional internal load balancer can front them. No session affinity is needed: challenges are stateless, so any replica can serve the /get-key that follows another replica’s /challenge. Keep the record in step with the live replicas; merod tries the next address when one does not answer. This URL is the cluster’s KMS URL.

Configure a merod node to use the KMS

Section titled “Configure a merod node to use the KMS”

Node images do this at first boot. calimero-init reads kms-url and tee-release-version from instance metadata. MDMA sets kms-url to the cluster’s VPC-internal URL. calimero-init creates the node with merod init --kms-url, which fetches the storage key before writing anything, so the node’s signing identity and account root never reach the data disk in plaintext. TDX protects the VM’s memory, not its disks: a plaintext store is readable by anyone who can snapshot the data disk.

merod verifies the KMS before it asks for anything: the KMS’s quote must carry a TCB status and MRTD/RTMR0–3 in the kms_allowed_* lists of the signed kms-attestation-policy[.<profile>].json in release mero-kms-v<version>, where <version> is tee-release-version. So a node only ever trusts the KMS image of the release it was told to use.

  • locked-read-only: refuses to create a node without kms-url, or when the baked merod lacks init --kms-url. A KMS URL without tee-release-version is always refused, because the KMS could not be verified.
  • Debug profiles: still create an unencrypted node when no KMS is given, and log that they did.
  • ephemeral-store=true (any profile): the node lives on a tmpfs mounted over /mnt/data, in memory TDX encrypts, and is gone at power-off. The store may then be created without a KMS, but only if the mount table confirms the home is on tmpfs and no swap is active. The flag alone relaxes nothing, so setting it on a normal node yields a node that forgets itself on reboot, never a plaintext disk. The release pipeline’s measurement VM uses this: it cannot have a KMS, because a release’s KMS image bakes in the node image’s measurements and is built after them.
  • Nodes created before this: they keep running and log that their store is unencrypted. Encrypting afterwards cannot work (every read is decrypted) and would not un-expose keys that have already sat on the disk. Recreate them.

The data disk is encrypted as a whole, too. The store is not everything on it: config.toml, the TLS key, mero-auth’s database and the fleet token sit beside it. So a new node’s data disk is LUKS2 with dm-integrity, keyed by merod kms disk-key from the same KMS (a dedicated disk-unlock identity, kept in the LUKS2 header). locked-read-only refuses to create a plaintext data disk under the same conditions it refuses a plaintext store. Existing plain-ext4 disks keep working, with a warning; recreate those nodes to encrypt them. See the boot sequence. Requires a core release with merod kms disk-key; until merodVersion carries it, the build-time conformance assert fails locked-read-only image builds.

tee-release-version cannot name a release older than the image: the image bakes its own version as the floor, and both calimero-init and merod (MERO_TEE_MIN_VERSION) refuse anything lower.

For a node you set up by hand, do the same: MERO_TEE_VERSION=<release> merod init --kms-url http://<kms-address>:8080/ .... merod writes the URL to [tee.kms] and fetches and verifies the release’s signed KMS policy itself, so there is no allowlist to copy into config.toml by hand. --kms-url must be given at init: a store already written in plaintext cannot be encrypted afterwards.

Replace a dead replica

Section titled “Replace a dead replica”

Replicas never restart in place: the service is Restart=no, and a replica that stops, crashes or loses its host is gone along with the root in its memory.

  1. Delete the dead VM and its disk.
  2. Start a new VM from the same release’s image, in the same zone or another one that keeps the spread (three zones, at least two regions), with kms-peers naming live replicas and kms-bootstrap unset or false.
  3. Wait for clusterRootReady: true, then add it behind the internal URL.

For GCP host maintenance (--maintenance-policy TERMINATE; TDX has no live migration), MDMA reads the maintenance-event notice and starts the replacement first, so it joins before the old VM goes. Maintenance windows differ by zone, which is one reason for the spread.

Recover a lost cluster

Section titled “Recover a lost cluster”

If every replica of a release’s cluster is gone, its root is gone, and there is no backup to restore: by design, nobody holds a copy. Running nodes keep their keys in memory and keep serving, but a node of that release that restarts cannot reopen its data disk.

Recovery needs no build, because the image holds no secret. It is the upgrade rollover, triggered early:

  1. Start a new cluster from the same release’s image (deploy a release’s KMS cluster). It generates a fresh root, so it has a new URL and derives keys no old disk opens with.
  2. Start new nodes against it (kms-url set to the new URL). They join their groups and sync from peers: TEE nodes are replicas (ReadOnlyTee, or RelayTee, which also relays members’ writes), and even TEE-authored writes replicate to every peer.
  3. Delete the old nodes and destroy their data disks once the new ones have synced. Their disks are sealed to a root that no longer exists.

Drill this in staging, including killing every KMS VM at once.

Probe images before release (KMS or node)

Section titled “Probe images before release (KMS or node)”

Both probes are manual (workflow_dispatch) GitHub Actions workflows. They boot real TDX VMs, exercise the flow end to end, verify quotes with Intel Trust Authority, and publish nothing.

node-image-mkosi.yaml with role: kms (see Build the images with mkosi). Inputs: profile, node_release_tag (the node release whose allowlist the KMS image bakes in), and wait_timeout_minutes. It builds the KMS image from the dispatched commit (playbook-kms.yml, ROLE=kms), starts a two-replica cluster (one with kms-bootstrap=true, the other with kms-peers naming the first), and checks that both hold the root and report the same transport key — derived from the root, so equal keys mean one root. Intel Trust Authority verifies both quotes, and their measurements must be identical. It deletes its VMs and firewall rule at the end, and its image unless keep_image is set.

Probe a UKI image’s measurements (prototype)

Section titled “Probe a UKI image’s measurements (prototype)”

uki-tdx-probe.yaml is Phase 0 of moving the images to reproducible builds with measurements computed from the build output (mdma#480). It builds mero-tee/mkosi-probe with mkosi v26 as a single unified kernel image, boots it once on a c3-standard-4 TDX VM with the node’s 200 GB data disk, and reads the evidence the image prints on its serial console. It needs no merod and no network on the VM, and it publishes nothing.

Run it from Actions → UKI TDX probe → Run workflow. Defaults build from Ubuntu resolute (26.04) in the staging probe’s project and zone. With no upload_bucket (input or the GCP_IMAGE_UPLOAD_BUCKET variable) it creates a temporary bucket and deletes it afterwards, which needs storage.buckets.create on the service account.

The image pins mkosi’s Seed=, SourceDateEpoch= and Snapshot= (every package, the tools tree’s included, from one snapshot.ubuntu.com instant). RTMR1 measures the disk’s GPT (GUIDs included) and the UKI’s Authenticode hash, and RTMR2 the UKI’s sections, so without them two builds of the same content measure differently. Two runs of the same commit should report the same RTMR1 and RTMR2.

The run passes only when all three hold:

  1. replaying the CCEL event log reproduces the quote’s RTMR0–3,
  2. the UKI’s Authenticode SHA-384 is a boot-application event in RTMR1, and
  3. RTMR1 and RTMR2 computed from the disk image and UKI before the VM exists (scripts/attestation/shared/compute_rtmr.py compute) equal the quote’s, event by event (compare).

RTMR1 is three fixed UEFI action strings, a separator, the disk’s GPT and the UKI’s Authenticode hash; RTMR2 is systemd-stub’s measurements of the UKI’s sections, each as its name and then its contents. Neither depends on anything but the build output. RTMR0 is the firmware and machine shape, RTMR3 what the booted system extends.

The job summary shows both checks and every event; the uki-tdx-probe-<run> artifact keeps the UKI, the mkosi manifest, the serial log, ccel.bin, quote.bin and evidence.json (readable by scripts/attestation/shared/tdx_event_log.py show). A run that never prints MEROEV done failed to boot or to reach the evidence service: the last serial lines are in the log of the collect step and in the artifact. An interactive prompt among them (such as systemd-firstboot’s “Please configure the system!”) means a unit is waiting for console input and holding the boot short of multi-user; the image masks the first-boot prompts for this. Payload lines that stop short of their -sha256 line, followed by a login prompt, mean a getty hung up the console mid-output; the image masks serial-getty@ttyS0 and orders the evidence service ahead of any getty.

Every package comes from the snapshot, but apt also downloads the live archive’s index files when a source has a Snapshot: field (for archive.ubuntu.com and security.ubuntu.com). It does not install from them; apt-cache policy lists the snapshot as the only package source. A build therefore still needs those two hosts reachable over plain HTTP.

Build the images with mkosi

Section titled “Build the images with mkosi”

node-image-mkosi.yaml (named for the image it began with) builds a node, KMS, agent or inference image with mkosi (mdma#480) from a branch, as the releases do. Its role input picks the image and its profile input the profile: locked-read-only, debug-read-only or debug. The workflow publishes nothing; it is how an image change is proven on TDX before a release builds it. 2.3.120 was the first release built this way; the Packer build, its GRUB and seal steps and build-and-release.sh were removed after it ran in production, so earlier releases are rebuildable only from their tags.

The roles take their inputs from environment variables:

  • node: nothing beyond versions.json.
  • kms: MERO_KMS_BINARY, KMS_NODE_POLICY_FILE and optionally KMS_AGENT_POLICY_FILE. The workflow builds mero-kms from the commit and takes the node allowlist from the node_release_tag release’s published-mrtds.json, which must verify against its .bundle.json (verify-node-policy.sh) and pass extract-node-allowlist.sh. Locally, pass KMS_NODE_POLICY_FILE a file the extract script wrote: the role refuses any value that is not 96 hex characters or a TCB status name.
  • agent: MERO_AGENT_GATE_BINARY, and MERO_AGENT_BINARY or MERO_BOT_BUNDLE. The workflow builds mero-agent-gate from the commit and bakes the stub agent (scripts/agent/stub-mero-agent.py), or with agent: mero-bot the mero-bot release versions.json pins, downloaded and checked as Release mero-kms does (needs the MERO_BOT_RELEASE_TOKEN secret).
  • inference: MERO_INFERENCE_GATE_BINARY; vLLM comes from mero-tee/inference/requirements.lock, installed with uv. The workflow boots it on a GPU TDX VM (H100 quota needed) set by inference_options, JSON: zone, machine_type and provisioning_model (default the INFERENCE_VM_* repository variables, then us-central1-a, a3-highgpu-1g, STANDARD), and model. Without a model it boots measure-only, as the release does. With one ({"model": {"repo", "revision", "sha256", "served_name", "engine_args", "disk_gb"}}, sha256 from scripts/inference/model-digest.sh) it serves it, checks /attest names it and carries GPU evidence, runs one chat completion, and the compare job replays the model’s RTMR3 event onto the computed image event (private inference).

The profiles differ:

  • Roles: the renderer evaluates the roles’ own when: conditions. Lockdown runs on locked-read-only only. On the node, conformance runs there too, and the fleet sidecar on the read-only profiles.
  • Access: on the debug profiles, build.sh also installs openssh-server and google-compute-engine (the guest agent and OS Login, so SSH keys come from instance metadata) and adds console=ttyS0,115200. Only locked-read-only boots quiet; the debug profiles show the kernel and systemd on the serial console, where the KMS probe looks for roothash= and the verity setup.
  • Checks: mkosi.finalize makes the lockdown checks, and build.sh refuses the cloud agents, on locked-read-only only. The unit, library and per-role checks run on every profile.
  • Measurements: calimero.role= and calimero.profile= are on each image’s command line, so RTMR2 and RTMR3 differ per role and profile.

Locally: ROLE=kms PROFILE=debug MERO_KMS_BINARY=... KMS_NODE_POLICY_FILE=... mero-tee/mkosi-image/build.sh OUT. ROLE defaults to node and PROFILE to locked-read-only.

The build (mero-tee/mkosi-image/build.sh) takes its files from the Ansible roles. scripts/image/render_role_tree.py reads the role’s playbook (playbook.yml, playbook-kms.yml or playbook-agent.yml) and does what its copy, template, file, get_url, unarchive and set_fact tasks do into a directory. The release binaries come from a download cache and are checked against build-inputs.lock.json. It also evaluates the roles’ asserts, so a KMS with an empty allowlist or an agent bundle that is not the pinned one fails the build. It skips an assert only when it checks a probe of the VM or the VM’s facts. The renderer records the roles’ packages, users and the units they enable and mask in a manifest. It lists every task it does not run, such as probes of a running VM, so none disappears silently.

build.sh adds the packages to the image. mkosi.postinst applies the enable list, creates the roles’ users (the agent’s slot users, baked in because the root is read-only) and locks root. mkosi.finalize applies the mask list and makes the lockdown checks of merotee-conformance (whose tasks probe a running VM, so the renderer skips them), plus each role’s own:

  • node: merod, meroctl, mero-auth and calimero-init are present, and the journal is volatile.
  • kms: mero-kms, kms-init and kms.env are present.
  • agent: the gate, the agent, merod and cryptsetup are present. finalize also runs the agent role’s probes, which the renderer skips: merod has kms disk-key, the kernel has dm_crypt and dm_integrity, and a baked mero-bot starts on the baked Node.

On the node and the agent, the GCE disk naming files their data disk needs are checked too. On every role, the services the init script starts are checked not to start at boot. After mkosi, build.sh checks that none of the removed packages (ssh, crash reporters, cloud agents) is installed.

How the image boots:

  • Boot: it boots as a UKI from the ESP, with no GRUB and no shim.
  • Root: the root is EROFS behind dm-verity, laid out by systemd-repart (mkosi.repart), with a tmpfs overlay (systemd.volatile=overlay).
  • Command line: mkosi puts the verity root hash on the UKI’s command line as roothash=, so RTMR2 covers every byte of the root. calimero.role and calimero.profile are there too.
  • Network: systemd-networkd’s DHCP (80-gce.network) replaces netplan and the guest agent. networkd does not set the hostname (it would need polkit); merotee-hostname.service sets it as root from the metadata server’s instance/hostname, before the role’s init, so vmagent’s instance_name and Vector’s hostname name the instance.
  • Initrd: it carries mkosi-initrd’s default module set plus dm-verity, erofs, overlay and autofs4 (KernelInitrdModules= in mkosi.conf), not every module of the kernel. The rest stay in the root. A boot that stops in the initrd for want of a driver needs it added there.
  • TPM: the seven systemd-pcrlock-* units are disabled. Nothing seals to a TPM policy: a node is trusted by its TDX quote, and its disk key comes from the KMS.

The run:

  1. builds the image, checks the UKI’s command line (roothash=, role, profile, no console= on locked-read-only), and computes RTMR1, RTMR2 and RTMR3 from the disk and UKI;
  2. creates a GCE image through the GCP_IMAGE_UPLOAD_BUCKET bucket and runs that role’s probe:
    • node: the node staging probe (node-image-gcp-staging-probe.yaml, also callable as a reusable workflow);
    • kms: a two-replica cluster (scripts/release/kms/run-cluster-probe.sh);
    • agent: ITA, then a throwaway secret provisioned to the agent (scripts/release/agent/run-agent-probe.sh);
  3. compares the quote’s RTMR1-3 with the computed values. It compares event by event when the image returns its event log, and the registers only otherwise.

The image is deleted at the end unless keep_image is set.

The build, its checks and the GCE upload are scripts under scripts/release/mkosi/, for this workflow and the releases alike: install-tools.sh installs the runner’s tools; build-image.sh builds as root and checks the command line and ownership, packages the disk, computes RTMR1-3 and writes the root and initrd digests; create-gce-image.sh uploads and creates the image; release-image-name.sh gives a release image the name, family and description mdma finds images by.

Two builds of one commit give the same root and so the same RTMR1-3: checked for the node image, a CI build against a local one. Two things stood in the way. The build leaves out what differed on every build: nvme-cli’s random host NQN and host id, alternatives.log and ldconfig’s aux cache. It also runs mkosi as root, because as an unprivileged user mkosi’s user namespace gave every file root as its owner, and the run fails when no file in the root is owned by anyone else. To find what differs between two builds, the run logs per-directory digests of the root and the initrd (scripts/image/image_digest.py); run the same on the other build and diff the two. Build locally as root, as CI does.

A locked-read-only image writes nothing to the serial console, because it has no console=. When it never comes up, run the workflow with serial_console set. This appends console=ttyS0,115200 systemd.journald.forward_to_console=1, so the kernel, systemd and the init script’s journal reach the serial port. The probes print the last lines of the serial port whenever their wait times out. A diagnostic build measures differently (RTMR2 covers the command line), so its registers are not the image’s.

When GCP fails to create a probe VM

Section titled “When GCP fails to create a probe VM”

The release probes create their VMs through gcp_create_instance (scripts/ci/gcp_create_instance.sh): the node release’s measurement VM, the KMS release’s agent probe and two-replica cluster, the mkosi run’s KMS and agent probes, the post-release agent e2e, and the log-auth receiver the node release’s debug profiles start. The manual probes create theirs the same way: node-image-gcp-staging-probe.yaml (its TDX VM) and uki-tdx-probe.yaml. No workflow or script calls gcloud compute instances create directly any more, so every VM mero-tee’s CI starts is retried. It makes up to three attempts, 30 and then 60 seconds apart, and only for errors GCP says to try again on: Internal error, backendError, HTTP 500/502/503/504, rate limiting and a zone out of capacity (ZONE_RESOURCE_POOL_EXHAUSTED). Each retry is a ::warning:: in the step’s log naming the error. A bad flag, a permission denied, an exceeded quota, an image that does not exist, a name already taken and any error it does not recognise fail on the first attempt.

A create that failed this way can leave the instance half-made. Before a retry the helper deletes any instance of that name, disks included, and creates it again under the same name, so the job’s own cleanup and the attestation sweep still find it. If that delete fails it stops instead of retrying. When all three attempts fail, re-run the failed jobs: the zone is having a bad hour, not the release. Release run 39287102050 (2.3.121) was lost to a single Internal error before this.

Verify release assets

Section titled “Verify release assets”

Before trusting any release, run the matching verifier (all under scripts/release/):

Terminal window
scripts/release/verify-kms-release-assets.sh           mero-kms-v<version>
scripts/release/verify-node-image-gcp-release-assets.sh mero-tee-v<version>
# or the combined entrypoint:
scripts/release/verify-release-assets.sh <tag>

Each downloads the release assets, checks SHA-256 checksums, and verifies the cosign signatures. A non-zero exit means the release is incomplete or tampered — do not deploy it. On a schedule, release-auditor.yaml runs the same class of checks across recent releases automatically.

Upgrade to a new release

Section titled “Upgrade to a new release”

Nothing is upgraded in place. Release N+1 brings a new node image, a new KMS image and so a new cluster with a new root; old nodes are replaced, not re-keyed.

  1. Verify both releases’ assets (above) and confirm the version appears in compatibility-catalog.json.
  2. Deploy the new release’s KMS cluster (above). It runs beside the old one; the two share nothing.
  3. Start new nodes from the new node image with kms-url set to the new cluster and tee-release-version set to the new release. They join their groups and sync from peers.
  4. Delete the old nodes and destroy their data disks once the new ones have synced.
  5. Retire the old release (below).

Retire a release

Section titled “Retire a release”

Once no node of release N is left, delete its cluster: all five replica VMs and their disks, and its DNS record. The root is then gone for good, so no copy of an old node disk can ever be opened again. The KMS image itself stays published with the release (its measurements are what anyone auditing the release checks against); retiring a cluster deletes only the running VMs.

Post-release e2e

Section titled “Post-release e2e”

After a Release mero-tee run completes, post-release-mero-tee-node-e2e.yaml (workflow_run-triggered) boots fresh TDX VMs per profile and checks their measurement candidates are covered by the release’s published-mrtds.json, independent of the KMS. The KMS side is proven inside Release mero-kms itself: each image is booted as a two-replica cluster, with the node allowlist of that very node release baked in, before anything is published.

Incident response

Section titled “Incident response”
Symptom Likely cause Action
/get-key → 503 policy_not_ready The replica has not joined its cluster yet (no root) Check /health clusterRootReady; check kms-peers names live replicas of the same release. A replica that cannot join is replaced, not restarted.
A replica never reaches clusterRootReady kms-peers names no live replica, names a different release’s cluster (different measurements, so the join is refused), or the joiner is a debug TD Read lastJoinError on the joiner’s /health and lastJoinRefusal on the peer’s. Point kms-peers at live replicas of the same release image; start a new VM.
Every replica is gone Cluster lost Recover a lost cluster.
Node key release → 403 measurement_policy_rejected / tcb_status_rejected Node image or profile is not the one this release’s KMS image bakes in, or the node’s TCB status is outside the baked list Run the node image of the same release and profile as the KMS; never widen anything to fit.
merod refuses the KMS (measurements not in kms_allowed_*) kms-url points at a different release’s cluster than tee-release-version names, or a non-release image Point the node at its own release’s cluster.
A release fails at “Create ephemeral TDX VM” or a probe’s instances create GCP returned the same transient error on three attempts, or an error that is not transient Read the step’s ::warning:: and ::error:: lines. Transient on every attempt: re-run the failed jobs. Otherwise fix what the error names (flag, permission, quota, image). See When GCP fails to create a probe VM.
cargo audit advisory flagged Vulnerable dependency Bump the dependency (see security-audit.yaml / CHANGELOG.md for the pattern); it ships in the next release.

See Error handling for the full error catalog.

Next steps

Section titled “Next steps”