Runbooks

Concrete, ordered procedures for the operations you run on a live deployment. They assume a node started as:

merod --home data/ --node node1 run

so its home directory is data/node1/, holding config.toml, the RocksDB datastore (data/node1/data/), and the blob store (data/node1/blobs/).

Caution

Calimero is pre-1.0. These runbooks use commands and on-disk layout that exist today; flags and behaviour can change between releases. Test each procedure on a non-production node first, and confirm command names against the merod and meroctl references for your build.

Upgrade a node binary

Replacing the merod binary in place. Because a node that runs an older binary safely buffers newer-schema deltas in its absorb buffer (see stale-schema deltas), a rolling upgrade across a fleet is tolerated — but each node still needs a clean stop/replace/restart.

1. Confirm the running build. Scrape merod_build_info (the constant-1 beacon carries version and peer_id) so you know what you are replacing:

   curl -s http://127.0.0.1:2528/metrics | grep merod_build_info

2. Quiesce client traffic. Stop new JSON-RPC / admin calls against this node at your load balancer or proxy. There is no in-process “drain” command; draining means cutting client traffic upstream and letting in-flight syncs settle.

   Caution

   merod has no documented graceful-drain subcommand. “Drain” here is an operator action (stop sending it work), not a CLI call. Watch sync_sessions_active and governance_pending_queue_depth settle before stopping.

3. Stop the node. Send SIGINT/SIGTERM to the merod process (e.g. Ctrl-C, systemctl stop, or docker stop) and wait for it to exit. Do not kill -9 unless it hangs — an abrupt kill risks leaving RocksDB to recover on next start.

4. Replace the binary. Install the new merod (and meroctl) over the old one. The home directory is untouched.

5. Restart. Start the node with the same home and node name:

   merod --home data/ --node node1 run

6. Verify. Confirm the new version is live and the node re-converges:

   curl -s http://127.0.0.1:2528/metrics | grep merod_build_info
   meroctl --node node1 peers

   Watch governance_pending_queue_depth and any absorbed-delta counters drain toward zero — an upgraded reader can now decode what the old binary buffered.

Recover a stuck upgrade

Use this when an application upgrade (a new bytecode version) does not finish on a node: a context stays on its old code, or a peer never reports as migrated. The recovery walks the node-local migration & recovery markers — inspect what they say, request a resync, then verify they clear.

The endpoints below are on the loopback HTTP server (default 127.0.0.1:2528); reach them the same way you reach the rest of the admin API for your auth setup.

1. Inspect — find the stuck members. The migration-status rollup is the operator-facing “have all peers migrated?” view. It snapshots each member’s migration heartbeat, including any failure:

   curl -s http://127.0.0.1:2528/admin-api/groups/<namespace_id>/migration-status

   Read rollup.allMigrated (the go/no-go) and rollup.failed. Any member whose migrationFailed is set names why it stalled — these map directly onto the ContextMigrationFailed marker:

   migrationFailed value	What happened
   check_aborted	The app’s migration check rejected the upgrade; the migrate was rolled back (zero residue).
   apply_failed	The migrate apply itself errored (e.g. the target wasm would not run).
   no_migration_path	The context cannot replay its next upgrade rung — the intermediate bytecode is unobtainable. Recovery is a resync (below) or reinstalling the missing version.

   Under the hood, a context that aborted its migrate is pinned to its pre-upgrade code by ContextExecutingBlob, while ContextActivatedBlob lags the group’s current app key — that gap is the “running old code” symptom. See the marker reference for what each column means.

2. Decide the recovery. A check_aborted / apply_failed member usually recovers by re-driving the upgrade once the underlying cause is fixed (a bad migration function, a missing dependency). A no_migration_path (stranded) context cannot replay its ladder locally — its recovery is a full-state resync from a healthy peer that already holds the upgraded state.

3. Request the resync. This sets the ContextResyncRequested marker and pulls a fresh snapshot, overwriting local state for that context:

   curl -s -X POST http://127.0.0.1:2528/admin-api/contexts/<context_id>/resync \
     -H 'content-type: application/json' \
     -d '{"force": false}'

   A successful call returns {"contextId": "...", "resyncStarted": true}.

   Caution

   A resync is destructive: it discards this node’s local state for the context and adopts a peer’s. The request is group-only, and it refuses when the context still holds local DAG heads (un-synced local work) unless you pass {"force": true} — forcing throws that local work away. Only resync when at least one healthy peer holds the authoritative upgraded state, and never on the last/only holder.

4. Verify the markers clear. Re-run the migration-status rollup. As the snapshot lands, ContextResyncRequested is cleared on completion, ContextActivatedBlob advances to the group’s current app key (so the context now runs the new code), and the ContextMigrationFailed marker self-clears — the member should move out of failed toward migrated:

   curl -s http://127.0.0.1:2528/admin-api/groups/<namespace_id>/migration-status

   rollup.allMigrated == true means the cohort converged. If a member is still failed after the resync settles, the cause is upstream (the upgraded state isn’t actually available on the peers you resynced from) — fix that before retrying.

Note

These markers are node-local columns that are not synchronized — they describe one node’s upgrade state. The rollup reads each peer’s self-reported heartbeat, so a member that is simply offline shows as unknown, not failed.

Back up & restore the datastore

The datastore (RocksDB) and blob store are plain directories under the node home. A consistent backup is a cold copy taken while the node is stopped.

Caution

There is no dedicated merod backup/restore subcommand. The procedure below is a filesystem-level copy. Copying a running RocksDB directory can capture a torn, unrecoverable state — always stop the node first (or use a filesystem/volume snapshot that is atomic).

Back up

1. Stop the node (SIGINT/SIGTERM, wait for clean exit) so RocksDB flushes and releases its lock.

2. Copy the whole node home. This captures config, identity keypair, datastore, and blobs together — they must be restored as a set:

   tar czf node1-backup-$(date +%Y%m%d).tar.gz -C data node1

3. Restart the node and confirm it is healthy (meroctl --node node1 peers).

Restore

1. Stop the node if it is running.

2. Replace the home directory with the backed-up copy (keep the original aside until you have verified the restore):

   mv data/node1 data/node1.broken
   tar xzf node1-backup-YYYYMMDD.tar.gz -C data

3. Restart with the same --home / --node:

   merod --home data/ --node node1 run

4. Let it re-converge. A restored node may be behind the live group; the reconciler and sync will catch it up from peers once it reconnects.

Note

The node’s identity keypair lives in config.toml ([identity]). Restoring the whole home preserves the peer_id, so the node rejoins as itself. Restoring only the datastore while regenerating identity would make it a different peer.

Recover a context from divergence

Use this when a context will not converge on its own — a scope_root / root-hash mismatch the reconciler keeps reporting, sync_verification_failures_total incrementing, or snapshot selection climbing without resolution. Escalate from gentlest to most forceful.

1. Confirm divergence, don’t guess. Check the sync counters (Observability) and logs for the context before acting. Transient lag after a partition is normal and self-heals.

2. Force a targeted sync. This is the first, non-destructive lever:

   meroctl --node node1 context sync --context <context_id>

3. Drive governance convergence if the deltas are buffered awaiting authorisation (growing governance_pending_queue_depth). Sync the owning group’s state so the governance DAG catches up:

   meroctl --node node1 group sync <group_id>

4. Sync everything if multiple contexts drifted (e.g. after a long partition):

   meroctl --node node1 context sync --all

5. Last resort — force a clean resync from peers. If the local copy is genuinely corrupt and will not reconcile, opt the node out of the context and re-join so it pulls a fresh snapshot from a healthy peer:

   meroctl --node node1 context leave-context <context_id>
   # then rejoin via the owning group / namespace
   meroctl --node node1 group join-context <context_id>

   Caution

   leave-context is a local opt-out (no DAG operation) — it drops this node’s local copy of the context so a rejoin re-materialises it from peers. Only do this when at least one other member holds the authoritative state, and never on the last/only holder of a context. This does not remove you from the group’s membership; it only resets local context state.

Rotate & leave safely

Removing this node (or identity) from a group, namespace, or context in a way that the rest of the fleet records correctly.

Leave a group

Publishes a MemberLeft operation to the group DAG, so peers record the departure:

meroctl --node node1 group leave <group_id>

Leave a namespace

A namespace is a root group; leaving cascades through its descendant groups:

meroctl --node node1 namespace leave <namespace_id>

Opt out of a context

Local-only opt-out (no DAG op) — drop this node’s copy of a context without leaving the group:

meroctl --node node1 context leave-context <context_id>

Caution

Leave is durable governance. group leave / namespace leave publish a signed departure that other members apply — you cannot silently un-leave; you rejoin via a fresh invite. Confirm you are leaving the right group/namespace ID first.

Signing-key rotation

Group signing keys are managed via:

meroctl --node node1 group signing-key register <group_id> <hex_signing_key>

Caution

The full key-rotation lifecycle (generating the new key, registering it, retiring the old one, and the per-group encryption-key handling that runs on eviction / self-purge for TEE fleet nodes) is broader than a single command and deployment-specific. Treat signing-key register as the registration step only, and consult Governance and Identities for the rotation model before rotating production keys.

Membership & governance How joins, leaves, roles, and key registration are recorded.

Next steps

Troubleshooting Diagnose the symptoms these procedures are meant to resolve.

Observability The metrics and logs to watch while running these runbooks.

Production deployment The deployment shape these operational tasks assume.