Runbooks

Commands below assume meroctl can reach the cluster and trusts it:

export CALIMERO_API=https://n1.eu1.example.com/
export CALIMERO_TEE_PROFILES=<profile>
export CALIMERO_OPS_TOKEN=<ops token>

Add a node

A node that is not one of the founding --peers joins a running cluster as a learner of the system group once it is registered.

1. Start it with the cluster’s settings (--cluster, KMS identity, governance flags, --rp-id, --origins), its own --node-id and --public-url, and without --bootstrap. It waits.

2. Register it:

   meroctl --operator-key-file operator.key ops add-node 4 https://n4.eu1.example.com/ [--zone eu1-b]

   Registering a node is an operator decision: the node answers 428 OPERATOR_SIGNATURE_REQUIRED and meroctl signs it with the operator key, as it does for ops remove-node, ops move and ops set-quotas. Only the nodes in the first leader’s --peers are registered without it, at genesis.

3. The system leader adds it as a learner; the log shows a registered node joined the system group. meroctl ops status lists it under the system group’s learners and in nodes.

4. New apps may now be placed on it (spread across zones, least-loaded first within that; see placement), and you can move existing ones onto it.

A learner holds the directory and can serve and host apps, but does not vote in the system group. While the system group has fewer than 3 voters, the system leader promotes caught-up learners to make exactly 3 (never 2), those in zones no voter is in first; a voter removed (below) is replaced the same way. It does not grow past 3 by itself.

Grow from one node

A cluster can start on one node (--bootstrap, --peers naming only itself) and grow:

1. Until it has three voters the first node is a single point of failure: every system write and access check needs it. Apps created now get one replica.

2. Add two nodes as above, registering each with --zone when they are in different zones. With only the second registered, the system group stays at 1 voter (2 voters would tolerate no failure); once both have caught up, the leader makes them voters in one change. meroctl ops status shows system.voters [1, 2, 3].

3. Spread the apps placed while there was one node:

   meroctl ops placement          # each app's replicas, each node's app count
   meroctl ops move <app> --replicas 1,2,3

   mdma does this as its rebalancer: it reads GET /ops/placement and sends POST /ops/apps/{app}/replicas, which answers unchanged: true when the app already has that set and 409 MOVE_IN_PROGRESS while another move of the app runs.

4. A fourth node and later ones stay learners of the system group; ops placement lists them with apps: 0 until a move puts apps on them.

Remove a node

1. Move every app off it: for each app whose replicas name the node, meroctl ops move APP --replicas ... with a set that leaves it out (run against the app’s leader). The node refuses to be removed while an app keeps a replica on it.

2. Remove it, against the system leader (another node answers 421 NOT_LEADER with details.leaderUrl): meroctl --ops-token T --operator-key-file F ops remove-node ID (DELETE /ops/nodes/{id}). It leaves the directory and the system group; a registered learner takes its vote while fewer than 3 voters are left.

3. Stop the process and retire its data disk. Only a signed ops add-node brings it back.

Move an app’s replicas

1. Find the app’s group and its leader: meroctl ops status (the groups entry for the app on a node that hosts it).

2. Move it:

   meroctl ops move <app-or-context-id> --replicas 2,3,4

   meroctl sends it to the node you named; a follower answers 421 NOT_LEADER and meroctl retries at the leader’s URL.

3. The leader hands the new nodes the app’s bundle and blobs, adds them as learners, waits, switches the voters to the new set, and records it. The answer is the group’s new status.

Refusals: TOO_FEW (fewer than min(3, nodes in the cluster) replicas), DUPLICATE_REPLICA, UNKNOWN_NODE (an id not registered), 403 naming a node whose release the app’s owner has not approved, NOT_HOSTED when the node asked does not hold the app, or 409 MOVE_IN_PROGRESS while another move of the app runs (retry once it answers). Repeating a move is safe: a move to the set the app already has answers unchanged: true.

Back up app state

With --backup-dir set, every node writes, every 60 seconds, a snapshot of each app group it leads that has moved since its last one: {backup-dir}/{app-id}/{index:020}.{group}.{snapshot-id}.snap, with its manifest beside it (….snap.manifest). The system group’s leader writes the system group’s the same way, under {backup-dir}/system/ (accounts and passkeys, organizations and teams, packages, the app directory, governance, the operator key), with the code the directory names (code/{hash}, sealed). Every node also keeps the cluster key K_c wrapped under its own release’s storage key, once per release (cluster-key/{release}). Force a backup now on a node with meroctl ops backup (it answers { written }); meroctl ops backup --all backs up every group the node holds, leader or not.

• Snapshots are the Raft snapshot files as written, sealed frame by frame under the cluster key K_c. The manifest names the app, group, log index, the file’s SHA-256, the backup format and the merod version and mero-tee release that wrote it, sealed under a key derived from K_c and bound to the app and the file name. Only a node holding this cluster’s K_c, on any release, opens either.
• Only leaders write, so collect --backup-dir from every node into one place; the newest file per app (and under system/) is the newest backup. Copy the manifest with its file: a backup without one is taken only as a pre-manifest backup (below). Keep cluster-key/ and code/ with them: without a kept cluster key no new node can open any of it.
• Copy them off the machine; the directory stands in for object storage.

Schedule, RPO and RTO

• RPO (what a restore loses) is the time since the backup restored, at most the 60-second interval plus however long your copy job takes to take it off the machine. The same holds for the system group: accounts, passkeys and memberships made after the system backup are lost when a cluster is founded again. Copy --backup-dir off every node at least as often as you can afford to lose; keep several generations per app (the files are named by index, so a plain sync keeps them all) and prune by age.
• RTO (how long a restore takes) is about the time to read the backup once on the seed and stream it once to each other replica: seconds for megabytes, minutes for gigabytes. The app answers 409 APP_RESTORING meanwhile; every other app keeps serving.
• Before an app upgrade you may need to undo (an app’s code cannot be downgraded, and a bad release is otherwise only fixed forward), take a backup by hand: meroctl ops backup on the app’s leader.
• Test a restore regularly, on a development cluster or on a scratch app.
• Before a release upgrade, check that cluster-key/ holds a copy for the new release (a node of it writes one once it joined): a cluster can be founded again only on a release that kept one and whose mero-kms still serves.

Restore an app

Restoring puts an app back to the state of a backup, on every replica: what the app wrote after the backup is gone, and it runs the bundle the backup’s state ran (an upgrade made since is undone with it). Use it for an app whose state a bad release or a bad write corrupted, or one whose replicas were lost.

1. Pick the backup: {backup-dir}/{app}/{name} and its .manifest, the newest one before the damage. Put both into the --backup-dir of a node that holds app replicas, under {backup-dir}/{app}/ (the node that wrote it already has them). That node will seed the restore and become the app’s first replica.

2. Restore, against that node:

   meroctl --operator-key-file operator.key ops restore <app> [--backup <name>]

   Without --backup the node takes its newest backup of the app. A restore is an operator decision: the node answers 428 OPERATOR_SIGNATURE_REQUIRED over <app> / __restore_app / { backup, sha256 } and meroctl signs it.

3. The node checks everything before it asks for the signature: the backup opens under this cluster’s key for this app (400 BAD_BACKUP otherwise: another cluster’s, another app’s, a swapped or truncated file), this release reads its format (409 UNKNOWN_BACKUP_FORMAT), a merod no newer than this node’s wrote it (409 BACKUP_FROM_NEWER_RELEASE), the bundle its state runs is still installed (409 BUNDLE_GONE: install that release again), and every node runs this release (409 UPGRADE_PENDING).

4. The app moves to a new group: the node seeds it from the backup alone, then adds the app’s other replicas, which copy the restored state, and makes them voters. The answer is { app, group, rows, backup: { name, index, format, merod, release }, status }. The old group is dropped on every node. Calls and views answer 409 APP_RESTORING until then.

If the restore stops half way (the seed restarts, a replica never catches up: 503), the app stays in APP_RESTORING; send the same restore again, which starts over in a newer group. One move or restore of an app runs at a time (409 MOVE_IN_PROGRESS).

Versions. A backup restores on the release that wrote it or any later one: a later release reads every earlier backup format and state, and refuses, by name, a format it has no reader for. A backup is never restored on an older release than the one that wrote it. A backup from before manifests (format 0, a lone .snap file) restores only into the group it was taken from, since nothing seals its file name: take a new backup once the cluster runs a release that writes manifests.

Trust. A backup is sealed under the cluster key K_c, so only a node of the same cluster opens it, whatever its release; the operator holding the files reads nothing in them. What the operator can do is choose which backup an app goes back to: a restore is a rollback of the app’s data, which is why it needs the operator’s signature, and why the log records it.

Found a cluster again from its backups

When a cluster must be recreated (its storage format changed one way, its nodes or region are gone), new nodes found it again from the backups, under the cluster’s own key K_c: to clients, apps and job nodes it is the same cluster (the cluster identity and every app’s keys derive from K_c).

1. Collect the backups (every node’s --backup-dir) into one directory: system/, cluster-key/, code/ and one directory per app.

2. Stop every node of the cluster backed up. A founder refuses to start while one still answers, as a node of this cluster, at an address the new --peers do not reuse: two live clusters must never share K_c.

3. Start the founder on an empty data directory with the cluster’s name, governance flags and --operator-key, on a release that kept a copy of the cluster key and whose mero-kms still serves:

   merod run --bootstrap --restore-from /backups --backup-dir /backups --peers 1=...,2=...,3=... ...

   It opens cluster-key/{release} with its storage key (no cluster key in ... opens with this node's storage key names the releases that kept one), forms the system group alone, seeds it with the newest system backup and loads the code. The fleet becomes --peers; replica nodes of the old fleet that are not in it are removed, job nodes stay. Start the other --peers as in Add a node, without --bootstrap: they get K_c from the founder and join as learners, and the first two become voters.

4. People sign in again with their passkeys, or refresh their sessions: refresh tokens come back with the system state; sessions the old nodes signed answer 401 once.

5. Every app waits for its own backup (409 APP_RESTORING, hosted nowhere). Restore each one as in Restore an app, on the founder or any node holding its backup:

   for app in $(ls /backups | grep -v -e system -e cluster-key -e code); do
     meroctl --operator-key-file operator.key ops restore "$app"
   done

Trust. Founding again grants nothing new: K_c reaches the founder only through a copy wrapped under the K_r of a release, which only a TD of that release and cluster name, attested by that release’s mero-kms, derives, the same bar a node’s own disk sets. A backup copied anywhere else opens nowhere. Once a release’s mero-kms is retired, its copy is dead: carry the key forward by letting a node of each new release join before the old one is retired (it writes its own copy).

Disaster recovery: the system group lost its majority

While a majority of the system group’s voters answers, nothing below is needed: a lost node is replaced and the cluster heals. When a majority is lost for good, no sign-in, install or organization change commits, and there is no route that changes the system group’s voters without a majority: the surviving nodes found the cluster again.

1. Make sure the lost nodes are gone for good (their disks destroyed, or their machines stopped and never restarted): a lost node that comes back after step 4 is a second cluster under the same key.

2. On every surviving node, back up everything it holds, leader or not:

   meroctl --api https://n1.eu1.example.com/ ops backup --all

   Each survivor writes its system group and every app group it hosts, as far as it applied them, and its copy of the cluster key.

3. Collect the survivors’ backups with the regular ones. Per app and for system/, the restore takes the newest file by index (the file names sort by it); a survivor that lagged wrote an older one.

4. Stop the survivors, then found the cluster again from that directory and restore every app.

What was committed but applied by none of the survivors is lost, as is anything since the newest backup of an app none of them hosted. An app whose own group kept its majority loses nothing it had applied on the survivors.

Set quotas

meroctl ops quotas
meroctl ops set-quotas '{"maxFileBytesPerOrg":50000000000}'

set-quotas changes the limits it names (a field left out keeps its value). They live in the system state, so every node enforces the same limits when commands apply; refusals are 429 QUOTA_EXCEEDED. On a cluster whose sign-up is open, maxAccounts is what bounds the cluster’s growth; see below for one that is not.

maxFuelPerDeveloperPerDay (a decimal string; "0", the default, is no limit) caps the fuel a developer’s apps burn per UTC day. The developer is the identity usage is billed to: the owner of the deployment an app was created from, or for an app made from a release directly, its organization or owner. It is checked where a call runs, not at apply (Limits).

A node of an earlier release does not know maxFuelPerDeveloperPerDay: it would drop the field, and its snapshots would bring the cap back to 0. So a non-zero cap is written only once every node runs a release that knows it (see UPGRADE_PENDING below).

After setting the fuel cap, don’t add nodes older than release 1.0.0-rc.12. A node that old, added later and then sent a quota change itself, writes the quotas without the cap, and every node then applies 0, no limit (#4809).

409 UPGRADE_PENDING

What refuses with it: a write that only the receiving node’s release, or a newer one, understands, while a registered node runs an older release or could not be attested. Today that is set-quotas with a non-zero maxFuelPerDeveloperPerDay; a quota change that leaves the cap at "0" is written as before. The gate comes before the operator is asked to sign.

Why: a node on an older release would drop what it does not understand from the system state it applies, and its snapshots would carry that loss to the other nodes.

What to do: read details.release (the receiving node’s release) and details.nodes (each node behind it, with the release its attestation named, or null if it could not be attested). Upgrade those nodes to that release or newer, or retire them (Upgrade the cluster’s release, Remove a node), then send the same request again.

409 FEATURE_NOT_ACTIVE

What refuses with it: a command, or a field of one, that a node of the cluster does not support yet, during a rollout from a release that lacks it (The cluster version). The message names the feature and the release it needs; details is { feature, needs, active } (active: null when a registered node did not say what it supports).

Why: a node that cannot decode an entry skips it, and its state would no longer match the others’.

What to do: finish the rollout: once no app keeps a replica on the old nodes, remove them (DELETE /ops/nodes/{id}; a node that is stopped but still registered holds the version back as well). The system leader raises the cluster version within about 10 seconds (clusterVersion in GET /ops/placement), then send the request again.

Close sign-up

A cluster’s sign-up policy is open, invite or closed. Start a new cluster with it (--sign-up invite, MEROD_SIGN_UP, or sign-up = "invite" in config.toml; mero-tee and mdma stamp it with the other genesis flags), or change an existing one with the operator’s signature:

# 428 OPERATOR_SIGNATURE_REQUIRED answers the bytes to sign; send them back signed
curl ... -X POST /ops/sign-up -d '{"policy":"invite"}'

Let people in on an invite or closed cluster with a sign-up invite: pick a random token, issue it signed, and hand the token to the person (their frontend sends it as invite at sign-up):

curl ... -X POST /ops/sign-up/invites \
  -d '{"token":"sup_<random>","uses":1,"expiresAt":"2026-12-01T00:00:00Z"}'

GET /ops/sign-up/invites lists the live ones (keys and terms, no tokens); DELETE /ops/sign-up/invites/{inviteKey} withdraws one. On an invite cluster an organization’s admins also bring people in with their organization invites. Where the operator has named the cluster’s owner (--deployers owner), the owner runs all of this instead (/sign-up, passkey).

Upgrade the cluster’s release

A new merod reaches production as a new mero-tee release (a new image). An app’s data moves only to nodes on a release approved for it.

1. Get the release approved. Where the operator holds release approvals:

   meroctl ops release <version>

   Where owners hold them (the default), each organization’s owner approves it with a passkey (POST /orgs/{org}/releases; meroctl namespace approve-release on development clusters), and each account for its own apps. Until then, nodes on the new release get none of that owner’s data.

2. Raise the peers’ floor only after every node runs the new release: --peer-min-release on nodes, --min-release on clients.

3. Start nodes on the new image and add them. Their K_r comes from the new release’s mero-kms and differs from the old nodes’; each new node waits, answering /health with a waiting field, until it is registered (/ops/nodes) and a peer delivers the cluster key K_c to its address, by attestation (up to 10 minutes; then it exits and is restarted) (The cluster key).

4. Move each app onto nodes on the new release with ops move. A move onto a node whose release the app’s owner has not approved is refused.

5. Retire the old nodes once they host no app: remove them (Remove a node), then stop them. Until every old node is removed, commands the new release brings are refused 409 FEATURE_NOT_ACTIVE; then the system leader raises the cluster version (clusterVersion in GET /ops/placement), and no node of an older release joins again.

New nodes, not new images under old nodes

A node does not move to a new release in place: the new release’s mero-kms gives it another K_r, which does not unwrap its cluster-key file, and it stops with “this node’s storage key does not open its cluster key”. Retire the old release’s mero-kms only once no node of that release remains: a node of it that restarts afterwards cannot open its disk. A data directory from an earlier merod storage format (log.redb) is migrated at start.

Upgrade apps

Who upgrades depends on --app-upgrades.

• Owner (default). The app’s owner, or for an org app anyone holding MANAGE_APPLICATION there (the org owner, an admin, a team lead, or a member granted it), approves each upgrade with their own passkey: POST /apps/{app}/upgrade { version }, or an organization-wide run POST /orgs/{org}/upgrade { package, version, team?, cascade? } approved once. Development clusters: meroctl context update, meroctl group upgrade trigger.

• Operator. Install the release the operator upgrades to, then upgrade:

  meroctl ops install com.example.kv@1.3.0
  meroctl ops upgrade <app-id> --version 1.3.0
  meroctl ops upgrade-namespace <org> --package com.example.kv --version 1.3.0 [--group <team> --cascade]
  meroctl group upgrade status <org> <run>

An upgrade plans its migration from the two bundles’ embedded ABIs: one state version up runs the target’s #[app::migrate] edge (or the export named with --migrate), the app’s migration check must pass, and a downgrade, a jump or a missing edge is refused. A run upgrades its apps one at a time and records each failure; POST /orgs/{org}/upgrades/{run}/retry runs the failed ones again as a new run.

Rotate the ops token

The token is per node configuration, not cluster state.

1. Make a new token and its hash (openssl rand -hex 32, then sha256sum).
2. On each node: merod --node nX config --ops-token-sha256 <new hash> (or the variable), then restart it.
3. Switch your tools to the new token; the old one fails with 403 on each node as it restarts.

Session-signing keys are each node’s receipt key, kept in its data directory (signing-key, wrapped under K_r) across restarts. A node replaced on a new data directory gets a new one; the sessions it signed before answer 401 and clients refresh. Removing a node ends the sessions it signed the same way.

A node is lost

• App groups on it keep serving while a majority of each group’s replicas answers; leaders move within seconds and forwarded calls retry at the new leader. Move the lost node’s apps to healthy nodes with ops move to restore their margin.
• The system group keeps working while a majority of its voters answers. If a majority of its voters is lost, sign-ins, installs and organization changes stop; there is no route to change the system group’s voters without a majority: follow Disaster recovery.
• Restarting a node with the same data directory and configuration resumes it: it catches up from the leaders.

Inspect a stopped node

merod --node n1 inspect                              # the raw layout, sealed sizes
merod --node n1 inspect --storage-key-file k_r.hex   # plus each group's vote, applied index, membership

Stop the node first: inspect opens the store for writing. The key file holds K_r as 64 hex characters, which only a development node gives you directly; inspect unwraps the cluster key from <data-dir>/cluster-key with it.

Clean up a development cluster

merod local --fresh        # start over
rm -rf .merod-local        # or delete its state