Skip to content

Replication

Every group — the system group and each app instance’s group — is a Raft log on openraft 0.9, run by calimero-log. One Engine per node hosts all of that node’s groups over one RocksDB database; a Group<M> is one log with its state machine M.

The engine orders opaque command bytes. The node gives them meaning:

  • A system entry is one SystemCmd (The system group).
  • An app entry is one ComponentCmd (crates/node/src/component.rs), JSON tagged by op:
op Written when What applying it does
componentInit The instance’s first leader runs init Applies the outcome; records the bundle, its compute component and the executor; marks the app initialized or records why init failed
componentCall A client’s call, an xcall delivery, or an action’s ctx.call ran Applies the outcome; remembers the idempotency answer
componentTick A timer fired Applies the outcome if the slot is later than the timer’s last applied one
componentUpgrade An upgrade (code only) Switches the bundle and its compute component
componentBlob, componentFile A blob or a sealed file was uploaded for the app (its bytes are in the blob store already) The app holds it, idle until a value refers to it
componentBlobForget, componentBlobRevoke, componentBlobsCollected The owner deleted a blob or revoked its grants; the leader collected forgotten bytes Drops the hold, the grants, or the queue entry
componentActionEnd, componentXcallDone An action ended or timed out; the leader delivered or dead-lettered the outbox’s head Records it; a failed action runs onActionFailed
componentBarrier The leader settles which earlier entries stand Nothing
componentRestored A restored group’s first entry after the backup’s rows Ids continue past every term the backup carries
computeAttempt, computeFinal The leader assigned a compute run to a job node; the run ended Records the attempt; registers the outputs and applies the result method’s outcome (Compute actions)
pipelineStart, pipelineStep, pipelineCancel A pipeline run starts, a step is decided, a run is cancelled Moves the pipeline run’s record

A componentCall entry records the method, arguments, caller, idempotency key, the passkey consent for a confirm method, the leader’s timestamp, the xcall it delivers if any, the ids the run drew (idTerm, idSeq), the base index it ran against, and the outcome: status, result or error, the storage diff (each write as key → value or a delete, in the order they last happened), the events with their audiences, the xcalls and actions it started, the outbound HTTP exchange it recorded, the files it wrote and the compute runs it started or cancelled.

An entry is format 1 (component::ENTRY_FORMAT): it carries no marker. A release that changes what an entry means writes {"v":N, first, and only once every replica of the group reads format N; a replica that finds a newer format applies the entry as a no-op that names it, so it never applies part of one (Format versions). An SDK 0.12 app’s entries (AppCmd, wasmer) apply as no-ops: that runtime is gone.

Entries are capped at MAX_ENTRY_BYTES, 1 MiB. openraft gives each append RPC one heartbeat interval, so a much larger entry would never reach a follower and would stall everything behind it. Large content goes to the blob store and the entry carries its hash. Group::write refuses a larger command (WriteError::TooLarge) before anything is appended.

a log entry, from the client to applied state1 clientsealed POST/apps/{app}/intents{ method, args, key }2 entry nodeprincipal + gate ata read index n;internal/call to leader3 app leaderpropose lock; replay aknown key, else run thecomponent at last_index4 Outcomestatus, result, diff,events, xcalls, files,http, compute starts5 entry { "op": "componentCall", … }method, args, caller, key, consent,tsMs, baseIndex, idTerm/idSeq, outcomeJSON · at most 1 MiB · ENTRY_FORMAT 16 Raft log (calimero-log)appended to the replicas;committed once a majority has it`log` column, sealed under K_c7 apply, every replicabaseIndex = last_index?yes: apply the diffno: no-op, conflict;the leader runs again8 one RocksDB batch, with the new applied indexapp keysrows `c`+key, hiddenthe diff's writesevent ringrows `e`‖n, last 10 000with their audiencesidempotencyrows `j`, hidden; `q`the answer, for a retryhead `h`last_index, lastWriteMs,timers, active runs9 snapshot, every 1000 entriesCALIMERO-SNAPSHOT-1 file of sealed frames;the log is compacted behind it (100 kept)10 the answer{ index, result, events, receipt }, the eventsthe caller is in the audience of, sealed back

Steps 1 and 2 are The write path, step 3 Application execution, step 8’s rows the storage schema. The rest is below.

Proposing: the leader runs, everyone applies

Section titled “Proposing: the leader runs, everyone applies”

Group::propose serializes proposals on a group behind one lock and hands the builder the applied state, on a node that believes it leads (its Raft metrics say so; otherwise openraft’s core answers, refusing a follower at once). It does not first confirm leadership with a majority: the entry’s own replication does that, and base_index makes a stale build harmless.

The builder records as base_index the index of the app’s last entry (AppState::last_index), and apply compares it with the index of the last entry the machine applied. Blank entries (a new leader’s first) and membership changes do not count: they change no app state, and counting them on one side only made the first call after every election conflict once. Regression: the_first_call_after_a_leader_change_commits_without_a_rerun in crates/node/tests/cluster.rs; calimero_build_conflicts_total in /ops/metrics counts the conflicts a node’s entries met. If they differ, other entries came in between (a new leader’s, or a write that was not yet applied when the call ran) and the diff was computed against replaced state: the entry applies as a no-op with conflict: true, records no idempotency key, and the node that proposed it runs the call again (Node::call loops). A client that gets 503 for a conflict that could not be resolved retries with the same idempotency key.

So a leader that has lost its majority without knowing it may still run a call on old state and append it, and nothing wrong follows. Either client_write answers NotLeader, or the entry never commits (the new leader’s log replaces it), or the next leader had received it and commits it: then, if anything the build did not see was applied ahead of it, it applies as the no-op above. Regression: a_stale_build_by_a_deposed_leader_never_applies in crates/log/tests/cluster.rs.

The base check covers the state, not the code. A call’s component is prepared from the bundle the leader reads before it takes the propose lock, so an upgrade that applies in between would let the old code run against the upgraded state on a base that matches. The build therefore checks that the state still names the bundle it prepared (AppState::bundle); if not, it answers a conflict without appending, and Node::call runs the call again on the new bundle.

A proposal that answers without appending (Proposal::Reply: an idempotent replay, an xcall already delivered, a read-only member’s refused write) is a read, so it is confirmed: the leader runs a read-index round (openraft’s ensure_linearizable) before answering, and builds again on the confirmed state if entries committed in between. Only a machine that checks the base it was built on may use propose; the system group’s commands are built on no state (Group::write) and apply judges them.

Followers never run WASM for an entry. They write the diff’s keys into the app’s state, append the events to the app’s event ring, queue the xcalls and actions in the outbox, and record the idempotency answer, all in one RocksDB batch together with the new applied index. The leader’s run is the only run.

In a Raft group:

  • There is one order. An entry is committed only once a majority has it, and every replica applies committed entries in index order.
  • A replica that is behind is only behind. It holds a prefix of the same log. The leader sends it the entries it lacks, or a snapshot if they were compacted.
  • A minority cannot write. A node cut off from the majority cannot commit, so it refuses writes (503) instead of accepting ones the rest of the group would later have to reconcile. A strong read takes its index from a leader that has just confirmed it leads, for the same reason (Reads).

There is therefore no sync handshake, no verdict and no merge on the replication path.

Raft keeps the logs equal, not what each replica made of them: a replica that applies an entry differently (a row it misread, an entry it skipped, code that is not deterministic) or whose disk changes a row holds another state, and nothing in the protocol notices. So each replica of an app group keeps a digest of its state (calimero_log::digest): the sum, modulo 2^256, of SHA-256(label ‖ len(key) ‖ key ‖ value) over every row of the group. A sum depends on the rows alone, not on the order they were written, how entries were batched, or whether they came by entries or a snapshot, so replicas at the same index hold the same digest exactly when they hold the same rows.

  • Kept as entries apply. A write moves the digest by the row it writes less the row it replaces (one read of that row, never a scan); a read that fails fails the batch. The digest after every entry whose index is a multiple of DIGEST_EVERY (16) is recorded, the last DIGEST_KEPT (16) of them, so replicas that batched differently meet at the same indexes.
  • Stored with the applied index (the group’s meta row digest), in the same batch. A start whose stored digest is not at the applied index (rows changed outside the log, StateTx::commit) takes it again from the rows. A snapshot install and a backup restore take it from the rows they write.
  • Compared by the leader. Every minute (DIGEST_COMPARE_EVERY), the leader of each app group asks the other replicas for their digests (POST /internal/digests, only of groups the caller is a member of, at most DIGEST_GROUPS_PER_ASK an ask) and compares them at the latest index both recorded. A difference is logged with the group, the peer and the index, and shows in calimero_group_diverged (Observability, runbook).
  • Across releases: replicas compare only digests of the same DIGEST_VERSION, raised with any change to a row’s stored shape; a peer on a release before digests answers 404 and is not compared.

The system group is not covered: its machine keeps its state in memory and writes no rows as it applies.

Setting (calimero_log::Config) Default
Heartbeat 200 ms
Election timeout 1.2–2.4 s
Snapshot after 1000 applied entries
Log entries kept behind a snapshot 100
Snapshot chunk on the wire 1 MiB (SNAPSHOT_CHUNK_BYTES)
Snapshot install timeout 10 s per chunk RPC

A frozen or partitioned leader is replaced after an election timeout. A call forwarded to an app’s leader waits at most FORWARD_TIMEOUT (5 s) before the forwarding node retries at whoever leads now; calls without a client idempotency key get a node key first (node:…, remembered 10 minutes), so the retry replays a call that did commit instead of running it twice. The process suite’s frozen-leader (crates/node/tests/fault/suite.py) is the regression test.

An app group’s state lives on disk, in its group’s rows of the state column (Storage schema); the machine keeps only a small Head in memory (bundle, executor, the component’s host records: timers, ring bounds, active runs, the last write’s time). Each applied batch commits its writes and the applied index together, so a restarted node resumes from where it stopped instead of replaying the log.

The system group’s machine keeps its state in memory and is checkpointed with every snapshot; a restart restores the checkpoint and replays the log after it.

The commit index is written without a sync: a crash can lose it or leave an older one. On start openraft takes the applied index when the record is behind it, replays the log up to the record when it is ahead, and learns the rest from the leader, so no entry applies twice or is skipped (a_lost_or_older_commit_index_is_learned_again and an_in_memory_machine_applies_each_entry_once_after_losing_its_commit_index in crates/log/tests/cluster.rs).

A group’s Raft core on a node stops when that node cannot go on safely: a write the disk refused (EIO, ENOSPC), a machine that panicked in an entry, an entry it could not apply. openraft treats each as fatal to the core, so the group itself starts it again. As the core stops, the node drops what it applied in memory (a batch that never committed, half an entry) and reads the group’s state and applied index back from the disk: the last committed batch for an app group, the last checkpoint for the system group. A stopped replica then answers like one that lags (a read that waits for an index waits for it to be applied again), and the node forwards its app’s calls and views to the other replicas. After RESTART_FIRST (1 s) it starts a new core on the disk’s state, which catches up from the log and the leader. A core that stops again within RESTART_MAX (5 min) waits twice as long before the next try, up to that, with up to a quarter of jitter, so a fault that persists (a full disk, an entry this release cannot decode) keeps the group stopped without a hot loop. Each stop is logged, and calimero_group_stopped and calimero_group_restarts_total show it (Observability). Tests: a_batch_that_fails_to_commit_stops_the_group_which_reloads_and_restarts, a_machine_that_panics_stops_the_group_which_reloads_and_restarts and a_fault_that_outlasts_a_restart_backs_off_and_shows_in_the_status in crates/log/tests/cluster.rs; a_node_whose_system_group_stopped_on_a_disk_error_rejoins_on_its_own in crates/node/tests/cluster.rs.

A snapshot is a file of sealed frames under <data-dir>/snapshots, written from a consistent RocksDB view of the group’s rows, never the whole state in memory. A follower that needs one receives the file in chunks. Every node of a cluster holds the same cluster key K_c, whatever release it runs, so a follower opens the leader’s frames as they are. Installing writes the group’s rows under the next generation (group ‖ generation ‖ key) and switches generations in one batch, so a crash mid-install leaves the previous state intact. calimero-log’s tests stream a ~6 MiB state to a new node this way.

openraft builds a snapshot beside the installs and applies of the same group, so a build reads its generation, its index and its RocksDB view under the one lock an install switches them under, and records its file only if the generation is still the one it read: a build that an install or a backup restore overtook is made again, and never replaces the newer snapshot or deletes its file (machine::build_races in crates/log).

The system group’s app directory decides which nodes host which app groups. The reconcile loop on every node starts the groups listed for it and stops the rest, deleting their data from that node.

  • Creation. CreateApp assigns the next group number. The first replica in the instance’s replica list initializes the group with all replicas as voters; whichever replica becomes leader runs init and appends the componentInit entry. The first replica stays one that holds the group: a move keeps the current replicas first in every list it records, and the nodes it adds after them, so a node that has not joined yet never forms a group of its own (initializing a node that already holds the group does nothing).
  • Late start. A Raft RPC for a group a node does not run yet goes through the engine’s group factory: the node starts it if the directory lists it as a replica, and answers “not hosted here” otherwise.
  • Release gate. Before any Raft RPC (entries or snapshot) leaves for a peer, Node::may_send checks that the peer’s attested mero-tee release is one approved to hold that app’s data (Deployment governance). The system group goes to every attested peer. The receiving node makes the same check, and that the sender is a member of the group (api::internal_raft). Both ends run it for every heartbeat of every app group, so neither scans the directory: the system machine keeps an index from group to app and from node to its apps, and caches each (group, release) decision, all derived from the applied state and dropped when the next system entry applies or a snapshot restores (SystemMachine::app_of_group, apps_on, group_may_hold). Nothing of it is stored or replicated. reconcile reads the same index, so a pass costs this node’s groups, not groups times apps.

POST /ops/apps/{app}/replicas { replicas } moves an app to a new replica set (at least three when the cluster has three nodes). It runs on the app group’s leader; any other replica answers 421 NOT_LEADER with details.leaderUrl before writing anything. The leader:

  1. refuses any target node whose attested release the app’s owner has not approved;
  2. records the union of old and new replicas in the directory, so every node involved starts the group (the current replicas first, in their order, then the new ones, so the first replica is still one that formed or joined the group);
  3. hands each new node the app’s bundle and every blob the app’s state marks (bytes that live outside the log, and that a cluster grown from one node holds only on that node), so the new replica can run and serve the app without the old ones;
  4. adds each new node as a learner and waits for it to catch up;
  5. changes the voter set to the new replicas (joint consensus);
  6. records the new replica list (the replicas that stay first, in their order), after which reconcile stops the group on nodes that left.

A rebalancer may repeat or race a move. A move to the replica set the app already has (as a set, in any order) writes nothing and answers unchanged: true. While a move of an app runs on its leader, a second move of that app is refused 409 MOVE_IN_PROGRESS before it writes anything, so two moves never mix their replica sets in the directory; the move runs to the end even if its caller disconnects. The guard is the leader’s: a move sent to a new leader while the old one is still moving is refused by the group’s own membership change instead.

A node started without --bootstrap waits. Once the operator registers it (POST /ops/nodes), the system leader adds it to the system group as a learner; it catches up from the log or a snapshot, learns the directory, and from then on placement and replica moves can put apps on it.

Learners become voters up to 3. On every reconcile pass, a system leader with fewer than 3 voters promotes registered learners that are caught up (at most PROMOTE_MAX_LAG, 16 entries, behind its log) so that the system group has exactly 3 voters, in one membership change (joint consensus). It never stops at 2: two voters tolerate no failure and make every system write wait on both, so a cluster started on one node keeps that one voter until two learners are ready, then goes from 1 to 3. Learners in a zone no voter is in come first (zone from POST /ops/nodes), then learners with no zone, then the rest. The leader never grows the system group past 3 by itself; a fourth node stays a learner (5 voters would be an operator’s choice, not implemented yet). Removing a voter still promotes a registered learner while fewer than 3 are left (DELETE /ops/nodes/{id}).

So a cluster can start on one node and grow: until the second and third nodes are voters, the first is a single point of failure for every system write and every access check (sys_fresh asks the system leader); from then on any one node, the first included, can fail. Apps placed while the cluster was one node have one replica; a replica move (ops placement, then ops move) spreads them. Growing a cluster draws the whole sequence.

The store names its format (calimero_log::STORE_FORMAT, 2) in a sealed node row, settled when the engine opens it: an empty store takes it, any other format is refused (Format versions). No earlier store, and no log.redb store from before RocksDB, is read: a rollout’s new nodes start empty and receive their data by snapshot.