Runbooks

Concrete, ordered procedures for running Mero TEE. They reference the real scripts and workflows in this repository; confirm exact flags against --help for your checked-out version before running against production.

Caution

Mero TEE is pre-1.0. Commands, asset names, and defaults can change between releases. Test each procedure against a non-production cluster and node first.

The model in one paragraph

Each release ships a node image and a KMS image, both frozen GCP TDX images sealed behind dm-verity. The KMS image (merotee-kms-<profile>-<version>, family merotee-kms-<profile>) runs as one cluster per release: five replicas across three zones in at least two regions, behind one VPC-internal URL. The cluster’s root key is generated in RAM by the bootstrap replica and reaches the others only by attested join. It is never written anywhere and never backed up. Nothing in a running cluster is ever reconfigured, patched or restarted in place: an upgrade is a new node image, a new KMS image, a new cluster and new keys; a dead replica is replaced by a new VM that joins. MDMA runs all of these procedures; the steps below say what it does, so you can check it or do it by hand in staging. The design has the rationale.

Deploy a release’s KMS cluster

1. Verify the release assets first. Never deploy an unverified artifact:

   scripts/release/verify-kms-release-assets.sh mero-kms-v<version>

   This checks the checksums and the cosign Sigstore bundles for the KMS release assets, including every kms-attestation-policy[.<profile>].json.

2. Start the bootstrap replica. Create one TDX VM (c3-standard-4, --confidential-compute-type=TDX, --maintenance-policy=TERMINATE) from the release’s image merotee-kms-<profile>-<version>, on the VPC only (no external address), with instance metadata kms-bootstrap=true. It generates a random 32-byte root inside the TD. Wait until its /health reports clusterRootReady: true:

   curl -s http://<replica-ip>:8080/health
   # {"status":"alive","service":"mero-kms","clusterRootReady":true}

3. Start the other four replicas in the remaining zones and regions (five in total, three zones, at least two regions), each with kms-peers set to the base URLs of the replicas already ready, e.g. kms-peers=http://10.0.0.5:8080. Each one asks a peer for a nonce, sends its own quote, and receives the root sealed to a one-time key once the peer has checked that the quote carries exactly its own MRTD and RTMR0–3 and is not a debug TD. The joiner checks the giver the same way. Wait for clusterRootReady: true on every replica.

4. Clear kms-bootstrap on the first replica (set it to false). It already holds the root and reads the key only at boot, but no VM that could ever boot again should carry kms-bootstrap=true: a second bootstrap is a second root.

5. Put one VPC-internal URL in front of all five: a name in a Cloud DNS private zone whose A record lists every ready replica’s internal IP (TTL 30s), on port 8080. The replicas span regions, so no regional internal load balancer can front them. No session affinity is needed: challenges are stateless, so any replica can serve the /get-key that follows another replica’s /challenge. Keep the record in step with the live replicas; merod tries the next address when one does not answer. This URL is the cluster’s KMS URL.

Note

/health is a liveness check with one extra field: clusterRootReady. A replica without the root answers /get-key with 503 policy_not_ready. To confirm a cluster actually issues keys, boot a node against it (below) or run the KMS TDX image probe. ACCEPT_MOCK_ATTESTATION is inert in the production build.

Caution

Metadata is untrusted and every key read is validated by kms-init: kms-bootstrap, kms-peers, and the optional observability keys (logs-endpoint, metrics-endpoint, observability-token). A wrong peer address only fails a join; a wrong endpoint only sends the replica’s logs, which hold no key material, somewhere else. Anything that decides what the KMS releases keys to — the node allowlist, the profile, the listen port — is baked into the image and measured, so it cannot be changed without producing a different KMS, which the running cluster refuses to join.

Configure a merod node to use the KMS

Node images do this at first boot. calimero-init reads kms-url and tee-release-version from instance metadata. MDMA sets kms-url to the cluster’s VPC-internal URL. calimero-init creates the node with merod init --kms-url, which fetches the storage key before writing anything, so the node’s signing identity and account root never reach the data disk in plaintext. TDX protects the VM’s memory, not its disks: a plaintext store is readable by anyone who can snapshot the data disk.

merod verifies the KMS before it asks for anything: the KMS’s quote must carry a TCB status and MRTD/RTMR0–3 in the kms_allowed_* lists of the signed kms-attestation-policy[.<profile>].json in release mero-kms-v<version>, where <version> is tee-release-version. So a node only ever trusts the KMS image of the release it was told to use.

• locked-read-only: refuses to create a node without kms-url, or when the baked merod lacks init --kms-url. A KMS URL without tee-release-version is always refused, because the KMS could not be verified.
• Debug profiles: still create an unencrypted node when no KMS is given, and log that they did.
• ephemeral-store=true (any profile): the node lives on a tmpfs mounted over /mnt/data, in memory TDX encrypts, and is gone at power-off. The store may then be created without a KMS, but only if the mount table confirms the home is on tmpfs and no swap is active. The flag alone relaxes nothing, so setting it on a normal node yields a node that forgets itself on reboot, never a plaintext disk. The release pipeline’s measurement VM uses this: it cannot have a KMS, because a release’s KMS image bakes in the node image’s measurements and is built after them.
• Nodes created before this: they keep running and log that their store is unencrypted. Encrypting afterwards cannot work (every read is decrypted) and would not un-expose keys that have already sat on the disk. Recreate them.

The data disk is encrypted as a whole, too. The store is not everything on it: config.toml, the TLS key, mero-auth’s database and the fleet token sit beside it. So a new node’s data disk is LUKS2 with dm-integrity, keyed by merod kms disk-key from the same KMS (a dedicated disk-unlock identity, kept in the LUKS2 header). locked-read-only refuses to create a plaintext data disk under the same conditions it refuses a plaintext store. Existing plain-ext4 disks keep working, with a warning; recreate those nodes to encrypt them. See the boot sequence. Requires a core release with merod kms disk-key; until merodVersion carries it, the build-time conformance assert fails locked-read-only image builds.

tee-release-version cannot name a release older than the image: the image bakes its own version as the floor, and both calimero-init and merod (MERO_TEE_MIN_VERSION) refuse anything lower.

For a node you set up by hand, do the same: MERO_TEE_VERSION=<release> merod init --kms-url http://<kms-address>:8080/ .... merod writes the URL to [tee.kms] and fetches and verifies the release’s signed KMS policy itself, so there is no allowlist to copy into config.toml by hand. --kms-url must be given at init: a store already written in plaintext cannot be encrypted afterwards.

Caution

Profile must match end to end. A locked-read-only KMS image bakes in only the locked-read-only node measurements, so a debug node is rejected on measurement policy, and a node pinned to the locked-read-only KMS policy refuses a debug KMS.

Replace a dead replica

Replicas never restart in place: the service is Restart=no, and a replica that stops, crashes or loses its host is gone along with the root in its memory.

1. Delete the dead VM and its disk.
2. Start a new VM from the same release’s image, in the same zone or another one that keeps the spread (three zones, at least two regions), with kms-peers naming live replicas and kms-bootstrap unset or false.
3. Wait for clusterRootReady: true, then add it behind the internal URL.

For GCP host maintenance (--maintenance-policy TERMINATE; TDX has no live migration), MDMA reads the maintenance-event notice and starts the replacement first, so it joins before the old VM goes. Maintenance windows differ by zone, which is one reason for the spread.

Recover a lost cluster

If every replica of a release’s cluster is gone, its root is gone, and there is no backup to restore: by design, nobody holds a copy. Running nodes keep their keys in memory and keep serving, but a node of that release that restarts cannot reopen its data disk.

Recovery needs no build, because the image holds no secret. It is the upgrade rollover, triggered early:

1. Start a new cluster from the same release’s image (deploy a release’s KMS cluster). It generates a fresh root, so it has a new URL and derives keys no old disk opens with.
2. Start new nodes against it (kms-url set to the new URL). They join their groups and sync from peers: TEE nodes are replicas (ReadOnlyTee, or RelayTee, which also relays members’ writes), and even TEE-authored writes replicate to every peer.
3. Delete the old nodes and destroy their data disks once the new ones have synced. Their disks are sealed to a root that no longer exists.

Drill this in staging, including killing every KMS VM at once.

Probe images before release (KMS or node)

Both probes are manual (workflow_dispatch) GitHub Actions workflows. They boot real TDX VMs, exercise the flow end to end, verify quotes with Intel Trust Authority, and publish nothing.

KMS image (GCP TDX)

kms-tdx-image-probe.yaml. Inputs: profile, node_release_tag (the node release whose allowlist the KMS image bakes in), and wait_timeout_minutes. It builds the KMS image from the dispatched commit (playbook-kms.yml, image_role = "kms"), starts a two-replica cluster (one with kms-bootstrap=true, the other with kms-peers naming the first), and checks that both hold the root and report the same transport key — derived from the root, so equal keys mean one root. Intel Trust Authority verifies both quotes, and their measurements must be identical. It deletes its image, VMs and firewall rule at the end.

Node image (GCP)

node-image-gcp-staging-probe.yaml. Key inputs: profile, image_name, image_project, admin_api_port (default 80), wait_timeout_minutes, and ita_appraisal_url. It boots a TDX VM from the image, waits for readiness, collects a quote, and verifies it against Intel Trust Authority.

The verifying happens in CI, over the admin API — the VM only supplies the quote, from /admin-api/tee/attest. CI picks a fresh nonce and then asserts four things (scripts/ci/probes/node_anti_fake_http.py, artifact node-client-verification.json):

check	proves
nonce_binding	the quote’s report_data carries the nonce CI just chose, so it is not a replay
wrong_nonce	report_data does not match a different nonce
genuine_hardware	Intel Trust Authority validates the quote
tampered_quote	ITA rejects the same quote with one byte flipped

Note

Nothing here takes the node’s word for anything, and no step needs a shell on the VM — which is what lets it run on locked-read-only, where merod-lockdown has removed openssh. An ITA call that fails for a transport or authorization reason is reported as inconclusive and fails the run; only an explicit rejection from ITA counts as a verdict about the quote.

Log-auth check (log_auth_check)

Off by default, and it needs no setup. It checks the chain that makes a node’s logs arrive at all. MDMA stamps the fleet’s observability token into instance metadata as observability-token. calimero-init writes it to /etc/vector/provided_token, and vector presents it on every push through the provided secret provider. If any link breaks, the sink rejects every write, and the node never hears about it.

With log_auth_check set, the probe (scripts/ci/probes/node_log_auth_probe.sh):

1. generates a random per-run token;
2. starts a receiver VM (log_auth_receiver.py, plain Debian, no service account, no external address) that accepts vector’s pushes and records only whether each one carried that token;
3. boots the node with observability-token set to the token and logs-endpoint pointed at the receiver, the same two keys MDMA sets;
4. passes once the receiver’s serial console shows an authenticated push, and deletes the receiver and its firewall rule either way.

The node has no shell, so the result is observed from outside, as with the attestation checks. On failure the reason code names what the receiver saw: LOG_PUSH_UNAUTHENTICATED (pushes with no credential), LOG_PUSH_WRONG_TOKEN, NO_LOG_PUSHES, or RECEIVER_NOT_READY. The artifact is node-log-auth-result.json.

Verify release assets

Before trusting any release, run the matching verifier (all under scripts/release/):

scripts/release/verify-kms-release-assets.sh           mero-kms-v<version>
scripts/release/verify-node-image-gcp-release-assets.sh mero-tee-v<version>
# or the combined entrypoint:
scripts/release/verify-release-assets.sh <tag>

Each downloads the release assets, checks SHA-256 checksums, and verifies the cosign signatures. A non-zero exit means the release is incomplete or tampered — do not deploy it. On a schedule, release-auditor.yaml runs the same class of checks across recent releases automatically.

Upgrade to a new release

Nothing is upgraded in place. Release N+1 brings a new node image, a new KMS image and so a new cluster with a new root; old nodes are replaced, not re-keyed.

1. Verify both releases’ assets (above) and confirm the version appears in compatibility-catalog.json.
2. Deploy the new release’s KMS cluster (above). It runs beside the old one; the two share nothing.
3. Start new nodes from the new node image with kms-url set to the new cluster and tee-release-version set to the new release. They join their groups and sync from peers.
4. Delete the old nodes and destroy their data disks once the new ones have synced.
5. Retire the old release (below).

Retire a release

Once no node of release N is left, delete its cluster: all five replica VMs and their disks, and its DNS record. The root is then gone for good, so no copy of an old node disk can ever be opened again. The KMS image itself stays published with the release (its measurements are what anyone auditing the release checks against); retiring a cluster deletes only the running VMs.

Post-release e2e

After a Release mero-tee run completes, post-release-mero-tee-node-e2e.yaml (workflow_run-triggered) boots fresh TDX VMs per profile and checks their measurement candidates are covered by the release’s published-mrtds.json, independent of the KMS. The KMS side is proven inside Release mero-kms itself: each image is booted as a two-replica cluster, with the node allowlist of that very node release baked in, before anything is published.

Incident response

Symptom	Likely cause	Action
/get-key → 503 policy_not_ready	The replica has not joined its cluster yet (no root)	Check /health clusterRootReady; check kms-peers names live replicas of the same release. A replica that cannot join is replaced, not restarted.
A replica never reaches clusterRootReady	kms-peers names no live replica, names a different release’s cluster (different measurements, so the join is refused), or the joiner is a debug TD	Read lastJoinError on the joiner’s /health and lastJoinRefusal on the peer’s. Point kms-peers at live replicas of the same release image; start a new VM.
Every replica is gone	Cluster lost	Recover a lost cluster.
Node key release → 403 measurement_policy_rejected / tcb_status_rejected	Node image or profile is not the one this release’s KMS image bakes in, or the node’s TCB status is outside the baked list	Run the node image of the same release and profile as the KMS; never widen anything to fit.
merod refuses the KMS (measurements not in kms_allowed_*)	kms-url points at a different release’s cluster than tee-release-version names, or a non-release image	Point the node at its own release’s cluster.
cargo audit advisory flagged	Vulnerable dependency	Bump the dependency (see security-audit.yaml / CHANGELOG.md for the pattern); it ships in the next release.

See Error handling for the full error catalog.

Next steps

• Error handling — every runtime error and its operator action.
• Release pipeline — how the assets these runbooks verify are produced.
• Attestation flow and Key release — the flows these procedures exercise.