Replication

Every group — the system group and each app instance’s group — is a Raft log on openraft 0.9, run by calimero-log. One Engine per node hosts all of that node’s groups over one RocksDB database; a Group<M> is one log with its state machine M.

What a log entry is

The engine orders opaque command bytes. The node gives them meaning:

• A system entry is one SystemCmd (The system group).
• An app entry is one AppCmd (crates/node/src/app.rs):

AppCmd	Written when	What applying it does
Init { bundle, base_index, outcome }	The instance’s first leader runs init	Applies the outcome; records the bundle; marks the app initialized or records why init failed
Call(CallEntry)	A call, a follow-up or a timer ran	Applies the outcome; consumes the follow-up or records the timer tick; remembers the idempotency key
Upgrade { bundle, base_index, outcome }	An upgrade ran its migration	Applies the outcome and, if it succeeded, switches the bundle
Blob { id, size }	A blob was uploaded for the app	Marks the blob as the app’s
BlobForget { id }	The app’s owner deleted a blob	Removes the mark
Done { seqs }	The leader delivered an xcall	Drops those follow-ups

A CallEntry records the method, arguments, caller, idempotency key, the passkey consent for a confirm method, the leader’s timestamp, the trigger (for a follow-up or timer), the base index it ran against, and the outcome: status, result or error, the storage diff (each write as key → value or a delete, in the order they last happened), the events, the follow-ups it asked for, and the storage entities it touched.

Entries are capped at MAX_ENTRY_BYTES, 1 MiB. openraft gives each append RPC one heartbeat interval, so a much larger entry would never reach a follower and would stall everything behind it. Large content goes to the blob store and the entry carries its hash. Group::write refuses a larger command (WriteError::TooLarge) before anything is appended.

Proposing: the leader runs, everyone applies

▶ Replay

Group::propose serializes proposals on a group behind one lock and hands the builder the applied state, on a node that believes it leads (its Raft metrics say so; otherwise openraft’s core answers, refusing a follower at once). It does not first confirm leadership with a majority: the entry’s own replication does that, and base_index makes a stale build harmless.

The builder records as base_index the index of the app’s last entry (AppState::last_index), and apply compares it with the index of the last entry the machine applied. Blank entries (a new leader’s first) and membership changes do not count: they change no app state, and counting them on one side only made the first call after every election conflict once. Regression: the_first_call_after_a_leader_change_commits_without_a_rerun in crates/node/tests/cluster.rs; calimero_build_conflicts_total in /ops/metrics counts the conflicts a node’s entries met. If they differ, other entries came in between (a new leader’s, or a write that was not yet applied when the call ran) and the diff was computed against replaced state: the entry applies as a no-op with conflict: true, records no idempotency key, and the node that proposed it runs the call again (Node::call loops). A client that gets 503 for a conflict that could not be resolved retries with the same idempotency key.

So a leader that has lost its majority without knowing it may still run a call on old state and append it, and nothing wrong follows. Either client_write answers NotLeader, or the entry never commits (the new leader’s log replaces it), or the next leader had received it and commits it: then, if anything the build did not see was applied ahead of it, it applies as the no-op above. Regression: a_stale_build_by_a_deposed_leader_never_applies in crates/log/tests/cluster.rs.

The base check covers the state, not the code. A call’s module (and its ABI facts and outbound client) is compiled from the bundle the leader reads before it takes the propose lock, so an Upgrade that applies in between would let the old code run against the upgraded state on a base that matches. The build therefore checks that the state still names the bundle its module came from (AppMachine::stale_module); if not, it answers a conflict without appending, and Node::call runs the call again with the new module. An upgrade checks the bundle it planned its migration from the same way. Regression: a_call_built_with_a_module_an_upgrade_replaced_never_commits in crates/node/src/app.rs.

A proposal that answers without appending (Proposal::Reply: an idempotent replay, a follow-up already run, a read-only member’s refused write) is a read, so it is confirmed: the leader runs a read-index round (openraft’s ensure_linearizable) before answering, and builds again on the confirmed state if entries committed in between. Only a machine that checks the base it was built on may use propose; the system group’s commands are built on no state (Group::write) and apply judges them.

Followers never run WASM for an entry. They write the diff’s keys into the app’s state, append the events to the app’s event ring, queue the follow-ups, and record the idempotency answer, all in one RocksDB batch together with the new applied index. The leader’s run is the only run.

Why not re-execute?

Re-executing on every replica (most blockchains’ model) needs every replica to reach the same answer, so the runtime must be deterministic and the inputs identical. Applying a diff in log order needs neither. The leader may read the clock (time_now), draw randomness (random_bytes), call out over HTTPS or open a blob, and the followers still end in the same state, because they copy the result rather than recompute it.

Why replicas cannot diverge

In a Raft group:

• There is one order. An entry is committed only once a majority has it, and every replica applies committed entries in index order.
• A replica that is behind is only behind. It holds a prefix of the same log. The leader sends it the entries it lacks, or a snapshot if they were compacted.
• A minority cannot write. A node cut off from the majority cannot commit, so it refuses writes (503) instead of accepting ones the rest of the group would later have to reconcile. Strong reads go through the leader for the same reason (Reads).

There is therefore no divergence report, no sync handshake, no verdict and no merge on the replication path.

Timing

Setting (calimero_log::Config)	Default
Heartbeat	200 ms
Election timeout	1.2–2.4 s
Snapshot after	1000 applied entries
Log entries kept behind a snapshot	100
Snapshot chunk on the wire	1 MiB (SNAPSHOT_CHUNK_BYTES)
Snapshot install timeout	10 s per chunk RPC

A frozen or partitioned leader is replaced after an election timeout. A call forwarded to an app’s leader waits at most FORWARD_TIMEOUT (5 s) before the forwarding node retries at whoever leads now; calls without a client idempotency key get a node key first (node:…, remembered 10 minutes), so the retry replays a call that did commit instead of running it twice. A multi-process scenario that freezes a leader is the regression test (Testing).

State, restarts and snapshots

An app group’s state lives on disk, in its group’s rows of the state column (Storage schema); the machine keeps only a small Head in memory (bundle, pending follow-ups, timer ticks, ring bounds). Each applied batch commits its writes and the applied index together, so a restarted node resumes from where it stopped instead of replaying the log.

The system group’s machine keeps its state in memory and is checkpointed with every snapshot; a restart restores the checkpoint and replays the log after it.

The commit index is written without a sync: a crash can lose it or leave an older one. On start openraft takes the applied index when the record is behind it, replays the log up to the record when it is ahead, and learns the rest from the leader, so no entry applies twice or is skipped (a_lost_or_older_commit_index_is_learned_again and an_in_memory_machine_applies_each_entry_once_after_losing_its_commit_index in crates/log/tests/cluster.rs).

A snapshot is a file of sealed frames under <data-dir>/snapshots, written from a consistent RocksDB view of the group’s rows, never the whole state in memory. A follower that needs one receives the file in chunks. Every node of a cluster holds the same cluster key K_c, whatever release it runs, so a follower opens the leader’s frames as they are. Installing writes the group’s rows under the next generation (group ‖ generation ‖ key) and switches generations in one batch, so a crash mid-install leaves the previous state intact. calimero-log’s tests stream a ~6 MiB state to a new node this way.

Groups come and go with the directory

The system group’s app directory decides which nodes host which app groups. The reconcile loop on every node starts the groups listed for it and stops the rest, deleting their data from that node.

• Creation. CreateApp assigns the next group number. The first replica in the instance’s replica list initializes the group with all replicas as voters; whichever replica becomes leader runs init and appends the Init entry.
• Late start. A Raft RPC for a group a node does not run yet goes through the engine’s group factory: the node starts it if the directory lists it as a replica, and answers “not hosted here” otherwise.
• Release gate. Before any Raft RPC (entries or snapshot) leaves for a peer, Node::may_send checks that the peer’s attested mero-tee release is one approved to hold that app’s data (Deployment governance). The system group goes to every attested peer.

Replica moves

POST /ops/apps/{app}/replicas { replicas } moves an app to a new replica set (at least three when the cluster has three nodes). It runs on the app group’s leader; any other replica answers 421 NOT_LEADER with details.leaderUrl before writing anything. The leader:

1. refuses any target node whose attested release the app’s owner has not approved;
2. records the union of old and new replicas in the directory, so every node involved starts the group;
3. hands each new node the app’s bundle and every blob the app’s state marks (bytes that live outside the log, and that a cluster grown from one node holds only on that node), so the new replica can run and serve the app without the old ones;
4. adds each new node as a learner and waits for it to catch up;
5. changes the voter set to the new replicas (joint consensus);
6. records the new replica list, after which reconcile stops the group on nodes that left.

A rebalancer may repeat or race a move. A move to the replica set the app already has (as a set, in any order) writes nothing and answers unchanged: true. While a move of an app runs on its leader, a second move of that app is refused 409 MOVE_IN_PROGRESS before it writes anything, so two moves never mix their replica sets in the directory; the move runs to the end even if its caller disconnects. The guard is the leader’s: a move sent to a new leader while the old one is still moving is refused by the group’s own membership change instead.

Joining nodes

A node started without --bootstrap waits. Once the operator registers it (POST /ops/nodes), the system leader adds it to the system group as a learner; it catches up from the log or a snapshot, learns the directory, and from then on placement and replica moves can put apps on it.

Learners become voters up to 3. On every reconcile pass, a system leader with fewer than 3 voters promotes registered learners that are caught up (at most PROMOTE_MAX_LAG, 16 entries, behind its log) so that the system group has exactly 3 voters, in one membership change (joint consensus). It never stops at 2: two voters tolerate no failure and make every system write wait on both, so a cluster started on one node keeps that one voter until two learners are ready, then goes from 1 to 3. Learners in a zone no voter is in come first (zone from POST /ops/nodes), then learners with no zone, then the rest. The leader never grows the system group past 3 by itself; a fourth node stays a learner (5 voters would be an operator’s choice, not implemented yet). Removing a voter still promotes a registered learner while fewer than 3 are left (DELETE /ops/nodes/{id}).

So a cluster can start on one node and grow: until the second and third nodes are voters, the first is a single point of failure for every system write and every access check (sys_fresh asks the system leader); from then on any one node, the first included, can fail. Apps placed while the cluster was one node have one replica; a replica move (ops placement, then ops move) spreads them. Growing a cluster draws the whole sequence.

Upgrading the on-disk format

A data directory that still holds the previous single-file store (log.redb) is migrated when the engine opens it: entries, vote, commit and purge marks, blobs, and each machine’s old whole-state snapshot (imported through Machine::import). The old file is kept beside, renamed log.redb.migrated. Scenario tests run one build on fresh state, so this path has its own tests (crates/log/src/migration_tests.rs).