Hardening & Security

A Calimero cluster is built so that whoever runs the machines, Calimero included, cannot read apps’ data. This page says what holds that up, what an operator controls, and which settings break it.

What the operator can and cannot see

The operator can	The operator cannot
Start, stop and place nodes; move app replicas between nodes	Read app state, blobs, account names or credentials: the Ops API has no read path into them
See counts, Raft status, placement and per-app usage by app and payer id (/ops/status, /ops/metrics)	Open the disks: they are sealed under a key only an attested TD receives
Set quotas and trigger backups	Open a backup: snapshots are sealed under the same key
Make the governance decisions the deployment gave the operator	Make the ones it gave the owners

This holds only for TDX-mode nodes running a mero-tee image the users’ clients trust. Everything below is about keeping it so.

What an organization’s admins can see

An organization’s data is the organization’s, as with any workplace tool. Private storage keeps a member’s data from other members, the operator and Calimero, not from the organization: its admins choose which code its apps run, so they can reach what those apps hold, members’ private data included. Tell members so where they join, for example: “Admins of this organization can read your data in its apps.”

An app a person deploys outside any organization has no admin: only its owner reaches it.

About the members themselves, admins and owners see what tells them someone may be locked out, and nothing more: a passkey count capped at “3+”, whether a recovery code is set, the UTC day of their last activity and how many live sessions they hold. Device labels, credential ids, sessions, origins and the activity log stay the account holder’s alone, and a deployment’s managers see only counts of their customers’ members (Members’ security).

Recovering an organization

An owner who loses every passkey gets their organization back only with its recovery seed: a 32-byte Ed25519 seed the owner downloads when creating the organization (or sets later, with a passkey) and keeps offline. The cluster stores only its public key. Nobody else can recover an organization: not its admins, not the operator (no Ops API route touches it), not a node (the seed never reaches one, and every step is verified where it applies in the system log, on every node).

Step	Who	Proof
Set, rotate or remove the key	The owner	Their passkey
Start a recovery	Any signed-in account	The seed’s signature naming that account
Wait 72 hours	Every member sees the pending recovery on GET /orgs/{org}
Cancel	The owner	Their passkey
Finish	The account the seed named	Its passkey

Finishing makes the recovering account the owner; the previous owner leaves the organization, and the apps they owned in it move to the new owner. Their account and its other organizations are not touched: one organization’s seed reaches only that organization. One recovery runs at a time, and each signature is used once.

If the seed leaks, the owner rotates the key: a recovery under way is cancelled and the old seed is refused from then on. An owner who has lost both every passkey and the seed cannot recover the organization; neither can anyone else. Design: docs/specs/org-recovery.md.

Recovering an account

A person who loses every passkey gets their account back only with its recovery code, made with a passkey while signed in (POST /auth/recovery) and shown once: 32 characters to write down. The cluster stores only the public key the code stands for. The recovery works as an organization’s does, with the same 72-hour delay (RECOVERY_DELAY_SECS):

Step	Who	Proof
Make or remove the code	The account	Its passkey
Start (/auth/recovery/begin, /start)	Anyone with the code, signed out, with a new passkey	The code’s key signs the account and that passkey; the code is spent
Wait 72 hours	Every signed-in device sees recoveryPending on /auth/me
Cancel (/auth/recovery/cancel)	The account	A passkey it already has
Finish (/auth/recovery/finish)	The new passkey	Its sign-in assertion over the session the finish opens

Finishing adds the new passkey, ends every session the account had, revokes every API key it made (whoever took the lost passkey could have made keys with it), and signs the new passkey in; machines that used a key need a new one. Nobody else can recover an account: the operator and the nodes never hold the code, the start names the one passkey it may add, and the finish needs that passkey. One recovery runs at a time, and a code starts only one. Someone who steals the code is stopped by any of the account’s passkeys within the 72 hours. Design: docs/specs/account-recovery.md.

TDX mode and development mode

	TDX mode (default)	--dev
Quotes	Real TDX quotes (configfs-tsm)	Mock quotes, which no client trusts unless told to
Peers	Must present real quotes matching --peer-profiles	Mock peers accepted
Storage key	From mero-kms only	--storage-key, --storage-key-file, or a key derived from the cluster name
Plain requests	Refused, except /health and /attestation	Accepted
Raw WASM uploads	Refused unless --allow-unsigned-bundles	Accepted

A --dev node’s development key is SHA-256("calimero-dev-storage-key|" + cluster): anyone can compute it. Never put real data on a --dev node.

The storage key

• K_r is derived from the KMS key and --cluster, and changes with each mero-tee release (each has its own mero-kms). It wraps the cluster key K_c, under which every value on disk and every snapshot is AES-256-GCM sealed, bound to its column, group and key. A new node gets K_c from a peer, which delivers it only to the attested TD at the address the operator registered for that node (The cluster key).
• A TDX node takes the KMS key from mero-kms only, after mero-kms checks the node’s quote, and only after the node checks mero-kms’s quote against the signed release policy. --storage-key and --storage-key-file are refused without --dev.
• The disk identity (--kms-identity) is shared by the cluster’s nodes. It is the handle mero-kms derives the key for; it is not the key, and a host holding it still needs an attested TD to get the key.
• mero-kms releases the key to any TD that matches the release policy, and a node fetches it before it serves anything. A deployment that wants to check a fresh TD first (its image, its merod) asks it for a release attestation (merod attest, or merod attest --serve while it boots) and hands over the cluster’s identity only once that verifies (release attestations).

From K_c the cluster also derives the cluster secret peers present, each app’s TEE key (what TeeSecret state opens with), and a stand-in key per member and app.

Peer and client trust

Clients and peers judge a node the same way (calimero_client::attest::verify):

1. The quote is genuine (Intel DCAP), or a mock quote the caller explicitly allows.
2. Its report data binds the nonce and the node’s transport key.
3. Its measurements are listed, under an allowed profile, in the signed published-mrtds.json of the mero-tee release the node names, fetched and Sigstore-verified; that release is at least the minimum.
4. TCB status is up to date, it is not a debug TD unless debug profiles are allowed, and the app hash binding holds.

Side	Profiles	Minimum release	Debug profiles
A node judging peers	--peer-profiles (default MERO_TEE_PROFILE)	--peer-min-release (default 2.3.109)	Only if the node itself runs a debug profile
meroctl	--tee-profiles (required for a real node)	--min-release (default 2.3.109)	--allow-debug-profiles

A profile whose name contains debug is a debuggable image: the host can read its memory. Never list one for a production cluster.

Governance

Three decisions are each held by exactly one party, chosen when the cluster is formed:

Decision	Flag	Owner side	Operator side
Which mero-tee releases may hold the data	--release-approvals	POST /orgs/{org}/releases, /auth/releases with a passkey	POST /ops/releases
Which bundle an app runs	--app-upgrades	POST /apps/{app}/upgrade, /orgs/{org}/upgrade with a passkey	POST /ops/apps/{app}/upgrade, /ops/orgs/{org}/upgrade
Which URL prefixes apps may call	--outbound-approvals	POST /orgs/{org}/outbound, /auth/outbound with a passkey	POST /ops/outbound

• The default is owner for all three.
• The party that does not hold a decision is refused (403), and the system log refuses the command too.
• The first leader writes the choice to the system state at genesis; it cannot change afterwards, whatever later starts are given.
• Every receipt-key quote binds it, so meroctl attest --context and verify_app_cluster in the clients prove it to users.

Release approvals are enforced by the replicas holding the data: a node does not send an app’s Raft entries or snapshots to a peer whose attested release is not approved for that app, does not place the app there, and refuses a replica move onto it. A new account or organization starts with the release it was created on approved.

Caution

Giving the operator --app-upgrades lets the operator choose the code that reads every app’s data; giving it --outbound-approvals lets it choose where that code may send it. Choose operator only where users accept that, and tell them: they can see it in the attestation.

The ops token

• Prefer --ops-token-sha256: the node then holds only the hash, and the machine’s configuration and metadata hold nothing usable.
• merod writes config.toml with mode 0600 because it may hold ops-token.
• Without either flag the Ops API is off (403 on every /ops/* route).
• The token reaches only counts, placement, quotas, backups and the decisions governance gave the operator.

Outbound HTTPS and secrets

Apps can make HTTPS requests only through the five gates described in Configuration. Secrets come from environment variables or a loopback sidecar, are bound to packages, signers and hosts, are filled in by the node and are redacted from what the app sees. Keep hosts as narrow as the apps need; "*" means any host an installed manifest declares.

Files on the host

File	Holds	Protect
{home}/{node}/config.toml	possibly ops-token; storage-key on dev nodes	mode 0600 (written so)
--kms-identity	the disk identity	readable by the node only
{home}/meroctl/sessions.json	saved sessions per node	mode 0600 (written so)
--dev-passkey files	a software passkey’s private key	development only
--backup-dir	sealed snapshots	sealed under K_c; keep them anyway

Rollback

A host can stop a node, copy its disk and start it later on the copy; sealing hides the content, not its age. A node whose key comes from a mero-kms that keeps rollback counters (docs/specs/rollback-protection.md) keeps a counter in its sealed store and in mero-kms, raises both every second, and refuses to start on a store behind mero-kms’s: a restored older copy, an empty store under a node id that ran before, or another node’s store.

• One node behind: wipe its data directory, remove it (DELETE /ops/nodes/{id}) and register it under a new node id; it joins empty and catches up.
• Every node behind, or the newest disks lost: start each node once with merod run --accept-rollback. mero-kms raises the cluster’s rollback epoch, which /health (rollbackEpoch) and /ops/status report: a forced rollback is visible, not silent.
• Limits: a rollback shorter than one second looks like a crash (Raft keeps those writes on the other replicas); counters live in mero-kms replicas’ memory, so a host that replaces a majority of them at once resets them, and every node then refuses to start until --accept-rollback. A node on a mero-kms without counters starts unprotected and logs it.

Known gaps

The node’s own notes (crates/node/AGENTS.md) list what is not done: no keys per app from mero-kms, no restore of a snapshot into a running cluster, no event audiences, no pruning of revoked session ids, and search indexes rebuilt from state at each start, and no recovery when every mero-kms replica is lost: backups are sealed only under the cluster’s key (planned: an organization’s own key opens its backups; the recovery seed above restores access, not data). Sign-up is open by default; bound it with the maxAccounts quota, or deploy with --sign-up invite|closed (Sign-up policy).

Checklist

• No node runs with --dev, --storage-key, --storage-key-file or --allow-unsigned-bundles.
• --peer-profiles and clients’ --tee-profiles list only production profiles; no debug profile anywhere.
• --peer-min-release and clients’ --min-release are at least the release you run, once all nodes run it.
• The governance flags are what you told users, on every founding node.
• The ops token is configured by hash and stored outside the machines.
• [outbound] is absent, or hosts lists only what apps need.
• Quotas, maxAccounts included, fit the cluster.