Deploy the node image

How to build a GCP TDX merod node image with this repository and boot it as an Intel TDX Confidential VM. This is the layer underneath the runbooks: the runbooks point an already-running merod at the KMS; this page gets a node image built, launched, and healthy in the first place. Everything below is grounded in the mkosi build and Ansible roles under mero-tee/ and the GCP workflows — confirm exact flags against --help and your checked-out version before running against production.

Caution

Mero TEE is pre-1.0. Image names, machine types, pinned versions, and defaults can change between releases. Boot a debug or debug-read-only image first; never make a fresh locked-read-only build your only path to a fleet.

What gets baked into an image

A node image is an Ubuntu 26.04 LTS (Resolute Raccoon) image built by mkosi (mero-tee/mkosi-image) from mero-tee/playbook.yml’s roles. Its packages come from a pinned snapshot of the Ubuntu archive (Snapshot= in mkosi.conf), so two builds of one commit give the same image. RTMR3 sysfs support needs kernel 6.17+. The image boots as a UKI with an EROFS root behind dm-verity. Component versions are pinned in mero-tee/versions.json:

Component	Version key	Value at time of writing
Node image	imageVersion	2.3.145
merod / meroctl (core tag)	merodVersion	1.0.0-rc.15
Traefik	traefikVersion	3.5.0
node-exporter	nodeExporterVersion	1.9.1
vmagent	vmagentVersion	1.132.0
Vector	vectorVersion	0.50.0

The node image runs merod 1.0 (core’s 1.0.0 line). Core 1.0 has no mero-auth and no peer-to-peer swarm: merod authenticates its own callers (passkeys, session tokens, API keys) inside a sealed channel that clients open after verifying the node’s TDX attestation, and each app is replicated by its own Raft log across the cluster’s nodes. See core’s merod reference.

The playbook.yml applies these roles in order: mero-traefik (the TLS front), calimero-core (bakes merod/meroctl at the core tag so calimero-init skips the runtime download), merotee (the init script and systemd units), node-exporter, vmagent, vector, cleanup, and — only for locked-read-only — merod-lockdown followed by merotee-conformance.

Profiles

The PROFILE the build is given selects one of three profiles (mero-tee/mkosi-image/build.sh accepts exactly this set):

Profile	Intended use	Lockdown role
debug	Local / dev	not applied
debug-read-only	Integration / pre-production	not applied
locked-read-only	Production	applied + conformance

Every profile runs merod in TDX mode (never --dev); the profile is what MERO_TEE_PROFILE names, which merod’s attestation reports and its peers and clients check. v1’s read-only merod mode is gone: in 1.0 every node of a cluster serves the same API.

The merod-lockdown role (ansible/roles/merod-lockdown/tasks/main.yml) hardens the production image so its measurements differ from the debug profiles. It:

• R1 removes openssh-server/openssh-client and deletes /etc/ssh;
• R2 masks all getty@/serial-getty@ttyS0/console-getty/getty.target units; the image’s kernel cmdline carries no console= (the build refuses one);
• R3 masks emergency/rescue/debug-shell (and overrides emergency/rescue to reboot -f);
• R4 locks the root account and empties /etc/securetty;
• R5 masks google-startup-scripts and the cloud-init* units so no GCP metadata script is ever executed — metadata is only read as data by the init script’s whitelist.

Note

Each profile deliberately produces distinct on-disk content (/etc/calimero/image-profile, calimero.role/calimero.profile in the kernel cmdline) so its MRTD / RTMR measurements differ. That is what lets the KMS release keys only to a matching profile — see Verification.

Prerequisites

• mkosi (v27, as CI pins it) and Python with Jinja2 and PyYAML. CI installs the rest with scripts/release/mkosi/install-tools.sh.
• Root: build as root, as CI does; unprivileged, mkosi’s user namespace gives every file root as its owner.
• GCP credentials only to upload and boot the image.

Build an image

1. Build one role and profile, with every version read from versions.json:

   sudo ROLE=node PROFILE=locked-read-only mero-tee/mkosi-image/build.sh OUT

   It writes OUT/merotee-node.raw (the disk), merotee-node.efi (the UKI) and mkosi’s package manifest. ROLE is node, kms or inference.

2. Compute the measurements and upload, as the releases do: scripts/release/mkosi/build-image.sh builds, checks the command line, computes RTMR1–3 and packages the disk, and scripts/release/mkosi/create-gce-image.sh creates the GCE image.

To build, boot and probe an image on GCP TDX from a branch, run node-image-mkosi.yaml (see Build the images with mkosi).

Where images are published

Each release image is published into the GCP project with a deterministic name and family (scripts/release/mkosi/release-image-name.sh):

• Image name: merotee-ubuntu-questing-25-10-<profile>-<version-with-dashes> (e.g. merotee-ubuntu-questing-25-10-locked-read-only-2-3-51).
• Image family: merotee-ubuntu-questing-<profile>.
• Disk: the disk mkosi writes, about 2 GB.

The questing-25-10 in those names is a frozen identifier, not the base release — the base is 26.04 LTS. mdma’s dispatcher matches on these prefixes, so they do not move when the base does.

The release-node-image-gcp.yaml workflow builds the images in CI and records the published project / family in its release provenance. Its VM-facing defaults (used for the post-build attestation probe) match Calimero Cloud MDMA: project cloud-486420, zone europe-west4-a, machine type c3-standard-4.

Launch a TDX Confidential VM

Boot the image as an Intel TDX Confidential VM. The flags below are exactly those the release-node-image-gcp.yaml and node-image-gcp-staging-probe.yaml workflows use to create their ephemeral VMs:

gcloud compute instances create <instance-name> \
  --project <vm-project> \
  --zone europe-west4-a \
  --machine-type c3-standard-4 \
  --confidential-compute-type TDX \
  --maintenance-policy TERMINATE \
  --min-cpu-platform "Intel Sapphire Rapids" \
  --no-service-account \
  --no-scopes \
  --image <image-name> \
  --image-project <image-project> \
  --subnet <subnetwork> \
  --metadata merod-version=<merod-version> \
  --tags merotee

Note

TDX requires --confidential-compute-type TDX with --maintenance-policy TERMINATE and --min-cpu-platform "Intel Sapphire Rapids" on a c3 machine type (c3-standard-4 is the tested size). The CI probes run with --no-service-account --no-scopes — the node reads metadata as data but never needs GCP API credentials.

GCP metadata the boot script reads

calimero-init reads a fixed whitelist of instance metadata attributes (ansible/roles/merotee/templates/calimero-init.sh.j2) — never arbitrary scripts. All are optional except where noted:

Metadata key	Effect	Default
tee-release-version	written as MERO_TEE_VERSION into /etc/calimero/merod.env (sanitized/escaped), with MERO_TEE_MIN_VERSION (the image’s own version) and MERO_TEE_PROFILE (the image’s profile), so merod verifies the KMS against that profile’s signed policy and names the release in its attestation. Refused if older than the image (downgrade protection). merod 1.0 does not run in TDX mode without it	unset → env file removed, merod not started
kms-url	the VPC-internal URL of the release’s KMS cluster, which the data-disk key and merod’s storage key come from; verified against tee-release-version’s signed policy	unset → locked-read-only refuses to boot (unless its store is memory-only); other profiles boot without merod
node-id	merod --node-id: this node’s id in its cluster (a u64)	required for a new node
public-url	merod --public-url: the URL clients and peers reach this node at; it must reach this node, not a balancer	required for a new node
cluster	merod --cluster, mixed into the storage key: every node of a cluster needs the same one	required on locked-read-only; merod’s dev elsewhere
kms-identity	the cluster’s shared KMS identity: base64 of a protobuf-encoded libp2p Ed25519 keypair (68 bytes), written 0600 to merod’s --kms-identity file, /mnt/data/merod/kms-identity, before merod init. A malformed value stops the boot. Never logged. See merod’s KMS client	required on locked-read-only when cluster is set (unless the store is memory-only); elsewhere merod generates a per-node identity and the boot logs it
kms-identity-wait-sec	how long a first boot that needs kms-identity and finds none waits for it while serving its release attestation; past it the boot fails before merod init, and the next boot waits again. A value that is not a whole number of seconds stops the boot	1800
job-node	exactly true makes a job node: no KMS, no store (a tmpfs scratch), no merod init. The boot writes MEROD_JOB_NODE, MEROD_CLUSTER_IDENTITY, MEROD_NODE_ID and MEROD_CLUSTER to /etc/calimero/merod.env and a per-boot drop-in unsets MEROD_METRICS_LISTEN, which merod refuses with --job-node. Needs tee-release-version, node-id, cluster (the name of the cluster whose identity is given) and cluster-identity. kms-url, kms-identity, peers, bootstrap and the governance keys (release-approvals, app-upgrades, outbound-approvals) are refused; other replica-only keys are ignored with a warning. Any other value stops the boot	unset → a normal node
cluster-identity	the cluster’s Ed25519 identity public key, 64 lowercase hex digits; only with job-node=true, and its presence without it stops the boot	unset
peers	merod --peers: the founding members, id=url,...	none
bootstrap	merod --bootstrap: true on exactly one founding node’s first boot	false
rp-id, origins	merod --rp-id / --origins: the WebAuthn relying party and the origins allowed to run passkey ceremonies and call the API (CORS)	merod’s (localhost, http://localhost:3000)
ops-token-sha256	merod --ops-token-sha256: the Ops API token’s hash; the token itself is never in metadata	unset → the Ops API answers 403
registry-url, apps-domain, peer-profiles, peer-min-release, replicas	the merod flags of the same names	merod’s defaults
release-approvals, app-upgrades, outbound-approvals	merod’s governance flags of the same names, owner or operator: who approves new node images, app upgrades and outbound hosts for the cluster. Read on a first boot only and fixed at the cluster’s genesis; set the same values on every node of a cluster. Any other value stops the boot. See Deployment governance	unset → merod’s default, owner
deployments, deployers	merod’s governance flags of the same names: deployments is open or granted (whether only granted accounts and organizations may install apps), deployers is operator or owner (who grants them). First boot only and fixed at genesis, like the keys above; a value set for a merod that does not take the flag stops the boot. See Who may deploy	unset → merod’s defaults, open and operator
sign-up	merod’s --sign-up: open, invite or closed (who may create an account). First boot only, like deployments; set for a merod that does not take the flag (1.0.0-rc.8 and earlier), it stops the boot. See Who may sign up	unset → nothing passed; merod’s default, open
metrics-endpoint	if set, configures and starts vmagent and node_exporter. Must be https:// except on the debug and debug-read-only profiles	unset → vmagent stays off
logs-endpoint	if set, configures and starts vector. Must be https:// except on the debug and debug-read-only profiles	unset → vector stays off
observability-token	the bearer token both shippers present; written to /etc/vector/provided_token and used via the provided secret provider	unset → the shippers push unauthenticated
tls-cert	the PEM chain (leaf first) mdma issued for the node’s TLS key; watched by calimero-tls-cert.service, which installs it as /mnt/data/tls/cert.pem and restarts Traefik only when every certificate parses, each is signed by the next, the leaf has not expired and its public key is the node’s own. Anything else is refused and the current certificate kept. See The node’s TLS certificate	unset → :443 serves the self-signed placeholder
tls-custom-hosts	custom domains: a JSON array of the hosts mdma has verified as this cluster’s served origins (["kv.example.com"]). calimero-tls-cert-custom.service publishes one CSR per host over the node’s TLS key as guest attribute calimero/tls-csr-custom. See Custom-domain certificates	unset → no custom certificates; any installed are removed
tls-cert-custom	custom domains: a JSON object {"<host>": "<PEM chain, leaf first>"}. A chain is installed for Traefik only for a listed host, the node’s own key, and a leaf naming exactly that host	unset → nothing new installed
ephemeral-store	true mounts a tmpfs over /mnt/data, for a VM meant to live minutes	unset
merod-version	only used to download binaries at boot if they were not baked into the image — debug profiles only; locked-read-only refuses to boot rather than run an unmeasured binary	required only when unbaked

The merod keys are merod’s own flag names, read once, when the node is first initialised: merod init writes them to config.toml on the encrypted data disk, and later metadata changes are not applied. Each value is checked against the shape merod takes, and a bad one stops the boot. The governance keys are checked against merod’s values (owner or operator; open or granted for deployments; operator or owner for deployers; open, invite or closed for sign-up), logged on the first boot as the line GOVERNANCE (fixed at the cluster's genesis, ...), and on every later boot ignored, with a line saying so. Not read: --dev, storage keys, --allow-unsigned-bundles and the plain --ops-token. server-port and swarm-port are no longer read: merod 1.0 has one listener, on loopback behind Traefik.

merod 1.0 derives its storage key from the KMS key released for the --kms-identity it presents, and every node of a cluster must present the same identity to derive the same key. mdma mints one identity per cluster and stamps it on every node as kms-identity metadata; the image checks it and writes it to /mnt/data/merod/kms-identity on the encrypted data disk before merod init. locked-read-only refuses a node with a cluster and no kms-identity; the debug profiles let merod create a per-node one, which only suits a one-node cluster.

The release-attestation gate

Stamping kms-identity on an instance releases the cluster’s storage key to it, so mdma should do it only for a TD it has seen run this release’s image and merod. The node’s own /attestation comes too late for that: merod run fetches its key before it serves anything. So a locked-read-only first boot with a cluster, a disk store and no kms-identity yet (and none installed from an earlier, interrupted boot), whose merod has attest --serve:

1. starts merod --home /mnt/data/merod --node default attest --serve 127.0.0.1:2528 in the background: merod’s own listen address, so Traefik, started early for it, routes https://<public-url>/attestation/release there with no route of its own. Each GET /attestation/release?nonce=<64 hex> answers with a fresh quote over the caller’s nonce and what this merod is (merodVersion, merodSha256, teeRelease = MERO_TEE_VERSION, teeProfile = MERO_TEE_PROFILE); see core’s merod attest reference;
2. waits for kms-identity metadata: it reads the key, and while it is unset (404) long-polls the attributes directory (?recursive=true& wait_for_change=true, by ETag; the body is discarded), backing off up to 30 s on metadata errors, for at most kms-identity-wait-sec (default 1800 s). Past that the boot fails before merod init, with the attestation server stopped;
3. sends the server SIGTERM (SIGKILL after 30 s) and reaps it, so the port is free for merod run, then installs the identity and continues as below.

mdma’s side: fetch the release attestation over its own fresh nonce, verify it (meroctl attest-release, or verify_release) against the release named by tee-release-version, and only then stamp kms-identity.

A kms-identity already set when the boot starts is installed at once, with no server. An older merod without attest, the debug profiles and memory-only stores read kms-identity once, as before, and a node already initialised never waits. The data disk’s own per-node identity (step 5 below) is not gated: it is unlocked before, and shredded from /run once the disk is open, so the release attestation binds no identity (identity: null).

Observability is off by default (R7): vmagent and vector are stopped and disabled at boot and started only when the matching *-endpoint metadata is present. node_exporter is started alongside vmagent, because the default scrape config targets it — its unit file is installed by the image but is not enabled, so without that it would be a dead scrape target.

The scrape config’s other target is merod itself (job merod): merod.service sets MEROD_METRICS_LISTEN=127.0.0.1:9464, so merod serves its metrics there, in plain HTTP on loopback only, with no credential. Most are request, latency, error, Raft, disk and process metrics, labelled with counts, route templates and group numbers. Since core 1.0.0-rc.9 the text also holds per-app usage for billing, the calimero_app_* series (calls, WASM CPU and memory, compute runs, outbound requests, state and blob bytes). Each is labelled app (the app’s id), org (its payer: an organization’s id, or account:{id} for a personal app) and developer (who is billed for it on a shared cluster, in the same form); calimero_app_calls_total adds kind. They hold ids, never names or state; the series and their meaning are on core’s operate/observability page (Usage metering). vmagent ships them with the rest, adding the same instance_name (the node’s FQDN), instance_type and instance_profile labels as to node_exporter’s, and presents the fleet’s observability-token.

Shipped usage series are not a billing record

The token is the same on every node and its vmauth user is write-only: it cannot read the store, but whoever holds it, including anyone who reads a node’s instance metadata, can write any series under any label, a calimero_app_*{developer="…"} included. Nothing in the store proves a sample came from merod on an attested node. Use these series for dashboards, not to charge anyone: mdma shows them in its consoles but does not post them to Stripe (MDMA_STRIPE_POST_VM_USAGE, off by default) until it bills from core’s usage book, which the app’s leader records inside the TEE and its Raft log commits.

Shipped logs are readable by whoever sets logs-endpoint

Instance metadata picks the log sink, so treat every shipped line as readable by the VM’s operator. Node and KMS images send their journal through the guard_egress transform before the sink. It runs mero-tee/ansible/roles/vector/files/egress-guard.vrl, which is baked into the image and measured.

The guard does three things:

• Allowlists fields. Only timestamp, message, unit, hostname, instance_name, instance_type, instance_profile and priority leave. journald’s _CMDLINE, _PID, _BOOT_ID, _MACHINE_ID and the rest are dropped.
• Masks credentials in the message: PEM private keys, JWTs, Bearer and Basic credentials, and the value of any secret-named key (password, token, secret, api_key, private_key, seed, mnemonic, cookie, authorization and similar).
• Masks application data merod logs at its default level. That is a guest’s execution log lines, a method’s error payload, guest panic text and the panic hook. It also masks admin login names, client values that serde errors echo, a refresh token’s jti, the JSON-RPC client id, KMS error bodies, URL userinfo, token query parameters and 24-word recovery phrases. Public keys, context IDs and peer IDs are kept, because diagnosis needs them.
• Caps each message at 8 KiB.

A line the program fails on is dropped, never shipped unguarded. configure_vector.sh refuses a config without the guard, and the locked-read-only conformance role refuses to build one. scripts/ci/tests/log-egress-guard-test.sh runs the guard with the pinned Vector release.

This is a backstop. Services still must not log secrets.

The observability credential comes only from MDMA

observability-token is the only way a node gets the token its shippers present, and the provided provider simply reads it from disk. There is no Secret Manager path: MDMA creates node instances with no serviceAccounts field at all, so a node has no cloud identity to read a secret with.

Instance metadata is set at creation only

It is never backfilled. A node created before any of these keys existed will never pick them up and must be recreated. That includes the observability credential: the v1 fleet sidecar delivered rotations over mdma’s should-join poll, and the 1.0 image has no sidecar.

The calimero-init boot sequence

calimero-init.service is a oneshot unit that runs /usr/local/lib/calimero/init.sh after network-online.target; merod and traefik order themselves After=calimero-init.service, and merod also Requires it, so it never writes to /mnt/data if the data disk could not be unlocked. Its log is /run/calimero/calimero-init.log (RAM) and the journal; the journal is volatile, so nothing it logs persists on the boot disk. The script (ansible/roles/merotee/templates/calimero-init.sh.j2) runs, in order:

1. Extend RTMR3 for attestation. It reads the dm-verity root hash (roothash=) from /proc/cmdline, computes an SHA-384 digest of calimero-rtmr3-v3:<role>:<profile>:<roothash>, and writes the 48-byte digest into the TDX guest sysfs (/sys/class/misc/tdx_guest/measurements/rtmr3:sha384, falling back to .../mr/rtmr3:sha384). Every block of the root filesystem is checked against that hash, so RTMR3 names the image’s whole root, its role and its profile. On locked-read-only a missing roothash= or RTMR3 interface stops the boot; the debug profiles log a warning and go on. (Releases up to 2.3.118 extended calimero-rtmr3-v2:... with calimero.root_hash, a build-time hash of a few directories.)

2. Disable observability by default (stops/disables vmagent and vector), then reads the metadata whitelist above.

3. Resolve binaries. If merod/meroctl are baked in /usr/local/bin (they are, for images built here), it uses them; otherwise, on the debug profiles only, it downloads them from the core release matching merod-version metadata. locked-read-only stops instead: a downloaded binary is in no measurement.

4. Apply tee-release-version. If set, it refuses a release older than the image’s own version (/etc/calimero/min-tee-release-version, baked at build), then writes MERO_TEE_VERSION, MERO_TEE_MIN_VERSION and MERO_TEE_PROFILE (the image’s profile, which selects the release’s per-profile KMS policy) to /etc/calimero/merod.env, which the merod unit loads via EnvironmentFile, and exports them for the steps below.

5. Unlock the data disk. Swap must be off (it is refused on locked-read-only). Then, for /dev/disk/by-id/google-data, the link GCE’s disk-naming udev rule makes for the disk attached as data (the image ships the rule and its google_nvme_id helper, vendored from GoogleCloudPlatform/guest-configs; if the link is still missing after 10 s, the one whole disk with nothing on it is used, and the journal says so):

   • blank (blkid -p finds no signature): fetch a key with merod kms disk-key --create-identity into a dedicated tmpfs, cryptsetup luksFormat LUKS2 (aes-xts-plain64, --integrity hmac-sha256), store the disk-unlock identity in a calimero-kms-identity LUKS2 token, open it as /dev/mapper/calimero-data, create ext4 on the mapper and mount it at /mnt/data. The integrity wipe covers the whole disk, so the first boot is slow (about 17 minutes for 200 GB pd-balanced); the journal logs Integrity wipe: N% (… of … GB, …m…s left) every 30 s while it runs, and mdma’s MDMA_TEE_POST_BOOT_VERIFY_TIMEOUT_SECONDS must outlast it. No fstab entry is written.
   • LUKS: read the identity from the token, fetch its key, open, mount.
   • plain ext4 (a node from before disk encryption, or a debug node with no KMS): mounted as before, with a loud warning. Never reformatted.
   • anything else, a failed probe, a LUKS header with no token, or an opened volume without ext4: the boot stops. Nothing is ever formatted.

   When the baked merod’s kms disk-key has --evidence-out (core 1.0, #4768), each fetch also writes a record of the KMS it verified (its quote’s registers, TCB status, signed release and profile, the quote’s SHA-256 and the time) to /run/calimero/kms-disk-evidence.json, on tmpfs, and MEROD_KMS_DISK_EVIDENCE pointing at it is appended to /etc/calimero/merod.env. merod run then reports that KMS, bound into the node’s quote, beside the storage key’s at GET /attestation?purpose=kms. An older merod, a fetch that wrote no record, or a boot that fetched no disk key (a plain or memory-only disk) names none, and the boot goes on.

   The key is shredded as soon as the disk is open. locked-read-only refuses a blank disk without kms-url + tee-release-version, or on a merod without kms disk-key. With ephemeral-store=true, a tmpfs is mounted over /mnt/data instead and the disk is untouched.

6. Prepare TLS for Traefik: the in-enclave key and a self-signed placeholder under /mnt/data/tls, then publish the CSR and its quote (see The node’s TLS certificate). This comes before merod’s configuration so Traefik can be started early by the next step.

7. Initialise merod 1.0 under /mnt/data/merod (node default), on the first boot only: when the cluster identity is still missing, serve the release attestation and wait for kms-identity (the gate); then write kms-identity metadata, checked, to /mnt/data/merod/kms-identity (0600), then merod init --kms-url <kms-url> --kms-identity /mnt/data/merod/kms-identity plus the merod metadata keys above. The data disk’s own identity (step 5) is separate and per node, so the cluster identity is read only once the disk is open. A v1 node’s state under /mnt/data/calimero is reported and left alone; merod 1.0 cannot read it. Without kms-url there is no merod (merod 1.0 takes its storage key only from mero-kms in TDX mode): locked-read-only with its store on a disk stops the boot, and any other node boots without starting merod and says so. A baked merod that is not 1.0 stops the boot on every profile.

8. Start services. It daemon-reloads, then enables and non-blocking-starts traefik (already running if the gate started it), and merod when it is configured. merod.service runs merod --home /mnt/data/merod --node default run with MEROD_LISTEN=127.0.0.1:2528; merod fetches its storage key from mero-kms before it opens its store.

What Traefik does on a 1.0 node

Traefik is a TLS front and nothing else. One router sends every path, on :80 and :443, to merod on 127.0.0.1:2528, with the Host header kept (merod’s --apps-domain routes on it) and no middleware:

• no forwardAuth: there is no mero-auth, and merod authenticates its callers inside the sealed channel;
• no path rules or rewrites: everything but /health and /attestation is a sealed envelope only the attested node can open, so Traefik passes the body through byte for byte;
• no rate limit: peers reach the node at the same public-url for Raft traffic, and a per-source limit would throttle a peer;
• no CORS: merod’s --origins decides.

:443 terminates TLS with a key generated in the TD under /mnt/data/tls. That TLS carries only sealed bytes, so it exists for browsers (passkeys need a secure context) rather than for confidentiality. Until a certificate is installed there, :443 serves a self-signed placeholder.

The node’s TLS certificate

mdma orders a publicly trusted certificate for the node’s key; the key never leaves the TD. The exchange rides GCE metadata rather than the node’s port, so Traefik stays a pass-through with no route of its own for it (ansible/roles/merotee/files/calimero-tls-cert.sh):

1. CSR out. On every boot calimero-init reads public-url and apps-domain and keeps a CSR over /mnt/data/tls/key.pem at /mnt/data/tls/csr.der, naming exactly the public-url host (lowercased) and, when apps-domain is set, *.<apps-domain>: DNS names only, CN the host. It is rebuilt only when that name set or the key changes, so its bytes are stable across boots. It takes a TDX quote from configfs-tsm whose report data is SHA-256(csr.der) || SHA-256("calimero.tee-attest.tls-csr.v1") and writes both, base64, as the guest attributes calimero/tls-csr and calimero/tls-quote (the instance needs enable-guest-attributes=TRUE, which mdma sets). A failure is logged and never stops the boot; the service below retries every five minutes until a publish succeeds.
2. mdma verifies the quote and the node’s measurements, checks the CSR’s names against its own record of the node, orders the certificate over ACME DNS-01, and writes the chain back as tls-cert instance metadata.
3. Certificate in. calimero-tls-cert.service (root, started with Traefik) long-polls tls-cert (wait_for_change, with backoff while it is unset). It installs a chain only for the node’s own key – the host can write metadata too – writes cert.pem atomically (0644) and restarts Traefik, which does not watch its certificate files. A renewal arrives the same way.

scripts/ci/tests/tls-csr-test.sh runs the tool against a mock metadata server and configfs-tsm.

Custom-domain certificates

A package’s served origin (a custom domain such as kv.example.com, core’s docs/specs/custom-domains.md) gets its own certificate for the same key. calimero-tls-cert-custom.service (the same tool, watch-custom) long-polls the instance attributes and, on each change:

1. CSRs out. For each host in tls-custom-hosts (lowercased; an entry that is not a DNS name is logged and left out; a value that is not a JSON array, or one naming more than 64 hosts, changes nothing) it keeps /mnt/data/tls/custom-csr/<host>.der, a CSR over /mnt/data/tls/key.pem naming exactly CN=<host> and DNS:<host>, rebuilt only when the key changes, and writes them all as guest attribute calimero/tls-csr-custom, {"<host>": "<base64 DER>"} ({} for none). No quote: mdma orders only when the CSR’s key is the one it attested through calimero/tls-csr. One certificate per host keeps one customer’s name out of another’s certificate in CT logs.
2. Certificates in. A chain in tls-cert-custom is installed at /mnt/data/tls/custom/<host>.pem (0644) only when the host is listed, every certificate parses and is signed by the next, the leaf has not expired, its key is the node’s own, and it names exactly that host (one subjectAltName, DNS:<host>). A refused renewal keeps the installed chain while it still passes.
3. Traefik. The installed chains are listed, with keyFile: /mnt/data/tls/key.pem, in /etc/traefik/dynamic/custom-certs.yml, beside routing.yml in the directory Traefik’s file provider watches. Traefik picks a certificate by SNI and falls back to the node certificate; a change reloads it without a restart. The chain is written into the file rather than named by path, because the file provider reloads only when the configuration it reads differs, and a renewal at the same path would not.
4. Reported. Guest attribute calimero/tls-custom-installed is the JSON array of hosts with a chain installed; mdma’s resolver routes a host only to nodes that report it.

A host leaving tls-custom-hosts loses its chain, CSR and Traefik entry. The node certificate (cert.pem, calimero/tls-csr, calimero/tls-quote) is never touched by this service. scripts/ci/tests/tls-custom-test.sh runs it against a mock metadata server.

The boot stage mdma shows

Until the data disk is ready nothing on the node listens (Traefik serves from it), and a first boot spends minutes on the integrity wipe, so mdma’s probes see only refused connections. calimero-init therefore writes where the boot is as the guest attribute calimero/boot-stage, as KEY: text, at each step: init, disk-wait, disk-key (fetching the disk key from mero-kms), disk-wipe (with its percentage every 30 seconds), disk-mkfs, disk-ready, tls, attest (serving the release attestation, then the KMS identity received), merod-init, services, done, and failed: <reason> from any fatal error. mdma’s dispatcher shows it as the node’s status line while the node does not answer yet.

It is the same channel as the TLS CSR above, and as untrusted: the host can read and write guest attributes, so the stage is progress for people only. Nothing secret goes in it, and nothing decides on it. Writing it is best effort (a 3-second timeout, errors ignored) and never holds the boot.

Verify the booted node

1. Confirm it reached RUNNING and answers. After the VM is RUNNING, merod answers GET /health and GET /attestation in the clear through Traefik on the external IP (port 80). Everything else needs a client that verifies the attestation and opens the sealed channel, such as meroctl --api URL --tee-profiles <profile>. A VM booted without kms-url does not run merod; it answers GET /attestation/release?nonce=<64 hex> instead (merod attest --serve, unit merod-attest), which is how the release and staging probes measure an image; see the release pipeline.

2. Attest the measurements. Confirm the node’s MRTD and RTMR values (the RTMR3 extend from step 1 of the boot sequence, plus calimero.role=node / calimero.profile measured into RTMR[2]) match the published, signed set. See Verification for the operator verify procedure and how to verify a release for the release-asset side.

Next steps

• Runbooks — point the booted node at the KMS and roll versions.
• Configuration reference — every runtime knob.
• Verification — MRTD/RTMR, root hash, operator verify.
• Verify a release — trust the release assets before you deploy them.