Replication
Every group — the system group and each app instance’s group — is a Raft log
on openraft 0.9, run by calimero-log. One
Engine per node hosts all of that node’s groups over one RocksDB database;
a Group<M> is one log with its state machine M.
What a log entry is
Section titled “What a log entry is”The engine orders opaque command bytes. The node gives them meaning:
- A system entry is one
SystemCmd(The system group). - An app entry is one
ComponentCmd(crates/node/src/component.rs), JSON tagged byop:
op |
Written when | What applying it does |
|---|---|---|
componentInit |
The instance’s first leader runs init |
Applies the outcome; records the bundle, its compute component and the executor; marks the app initialized or records why init failed |
componentCall |
A client’s call, an xcall delivery, or an action’s ctx.call ran |
Applies the outcome; remembers the idempotency answer |
componentTick |
A timer fired | Applies the outcome if the slot is later than the timer’s last applied one |
componentUpgrade |
An upgrade (code only) | Switches the bundle and its compute component |
componentBlob, componentFile |
A blob or a sealed file was uploaded for the app (its bytes are in the blob store already) | The app holds it, idle until a value refers to it |
componentBlobForget, componentBlobRevoke, componentBlobsCollected |
The owner deleted a blob or revoked its grants; the leader collected forgotten bytes | Drops the hold, the grants, or the queue entry |
componentActionEnd, componentXcallDone |
An action ended or timed out; the leader delivered or dead-lettered the outbox’s head | Records it; a failed action runs onActionFailed |
componentBarrier |
The leader settles which earlier entries stand | Nothing |
componentRestored |
A restored group’s first entry after the backup’s rows | Ids continue past every term the backup carries |
computeAttempt, computeFinal |
The leader assigned a compute run to a job node; the run ended | Records the attempt; registers the outputs and applies the result method’s outcome (Compute actions) |
pipelineStart, pipelineStep, pipelineCancel |
A pipeline run starts, a step is decided, a run is cancelled | Moves the pipeline run’s record |
A componentCall entry records the method, arguments, caller, idempotency key,
the passkey consent for a confirm method, the leader’s timestamp, the xcall it
delivers if any, the ids the run drew (idTerm, idSeq), the base index
it ran against, and the outcome: status, result or error, the storage
diff (each write as key → value or a delete, in the order they last
happened), the events with their audiences, the xcalls and actions it started,
the outbound HTTP exchange it recorded, the files it wrote and the compute runs
it started or cancelled.
An entry is format 1 (component::ENTRY_FORMAT): it carries no marker. A
release that changes what an entry means writes {"v":N, first, and only once
every replica of the group reads format N; a replica that finds a newer
format applies the entry as a no-op that names it, so it never applies part of
one (Format versions). An SDK 0.12 app’s
entries (AppCmd, wasmer) apply as no-ops: that runtime is gone.
Entries are capped at MAX_ENTRY_BYTES, 1 MiB. openraft gives each append RPC
one heartbeat interval, so a much larger entry would never reach a follower and
would stall everything behind it. Large content goes to the
blob store and the entry carries its hash. Group::write
refuses a larger command (WriteError::TooLarge) before anything is appended.
A log entry’s life
Section titled “A log entry’s life”Steps 1 and 2 are The write path, step 3 Application execution, step 8’s rows the storage schema. The rest is below.
Proposing: the leader runs, everyone applies
Section titled “Proposing: the leader runs, everyone applies”Group::propose serializes proposals on a group behind one lock and hands the
builder the applied state, on a node that believes it leads (its Raft metrics
say so; otherwise openraft’s core answers, refusing a follower at once). It
does not first confirm leadership with a majority: the entry’s own replication
does that, and base_index makes a stale build harmless.
The builder records as base_index the index of the app’s last entry
(AppState::last_index), and apply compares it with the index of the last
entry the machine applied. Blank entries (a new leader’s first) and membership
changes do not count: they change no app state, and counting them on one side
only made the first call after every election conflict once. Regression:
the_first_call_after_a_leader_change_commits_without_a_rerun in
crates/node/tests/cluster.rs; calimero_build_conflicts_total in
/ops/metrics counts the conflicts a node’s entries met. If they differ, other entries came in between (a new leader’s,
or a write that was not yet applied when the call ran) and the diff was computed
against replaced state: the entry applies as a no-op with conflict: true,
records no idempotency key, and the node that proposed it runs the call again
(Node::call loops). A client that gets 503 for a conflict that could not be
resolved retries with the same idempotency key.
So a leader that has lost its majority without knowing it may still run a call
on old state and append it, and nothing wrong follows. Either client_write
answers NotLeader, or the entry never commits (the new leader’s log replaces
it), or the next leader had received it and commits it: then, if anything the
build did not see was applied ahead of it, it applies as the no-op above.
Regression:
a_stale_build_by_a_deposed_leader_never_applies in crates/log/tests/cluster.rs.
The base check covers the state, not the code. A call’s component is prepared
from the bundle the leader reads before it takes the propose lock, so an
upgrade that applies in between would let the old code run against the
upgraded state on a base that matches. The build therefore checks that the
state still names the bundle it prepared (AppState::bundle); if not, it
answers a conflict without appending, and Node::call runs the call again on
the new bundle.
A proposal that answers without appending (Proposal::Reply: an idempotent
replay, an xcall already delivered, a read-only member’s refused write) is a read,
so it is confirmed: the leader runs a read-index round (openraft’s
ensure_linearizable) before answering, and builds again on the confirmed
state if entries committed in between. Only a machine that checks the base
it was built on may use propose; the system group’s commands are built on no
state (Group::write) and apply judges them.
Followers never run WASM for an entry. They write the diff’s keys into the app’s state, append the events to the app’s event ring, queue the xcalls and actions in the outbox, and record the idempotency answer, all in one RocksDB batch together with the new applied index. The leader’s run is the only run.
Why replicas cannot diverge
Section titled “Why replicas cannot diverge”In a Raft group:
- There is one order. An entry is committed only once a majority has it, and every replica applies committed entries in index order.
- A replica that is behind is only behind. It holds a prefix of the same log. The leader sends it the entries it lacks, or a snapshot if they were compacted.
- A minority cannot write. A node cut off from the majority cannot commit,
so it refuses writes (
503) instead of accepting ones the rest of the group would later have to reconcile. A strong read takes its index from a leader that has just confirmed it leads, for the same reason (Reads).
There is therefore no sync handshake, no verdict and no merge on the replication path.
State digests
Section titled “State digests”Raft keeps the logs equal, not what each replica made of them: a replica that
applies an entry differently (a row it misread, an entry it skipped, code that
is not deterministic) or whose disk changes a row holds another state, and
nothing in the protocol notices. So each replica of an app group keeps a
digest of its state (calimero_log::digest): the sum, modulo 2^256, of
SHA-256(label ‖ len(key) ‖ key ‖ value) over every row of the group. A sum
depends on the rows alone, not on the order they were written, how entries
were batched, or whether they came by entries or a snapshot, so replicas at the
same index hold the same digest exactly when they hold the same rows.
- Kept as entries apply. A write moves the digest by the row it writes less
the row it replaces (one read of that row, never a scan); a read that fails
fails the batch. The digest after every entry whose index is a multiple of
DIGEST_EVERY(16) is recorded, the lastDIGEST_KEPT(16) of them, so replicas that batched differently meet at the same indexes. - Stored with the applied index (the group’s meta row
digest), in the same batch. A start whose stored digest is not at the applied index (rows changed outside the log,StateTx::commit) takes it again from the rows. A snapshot install and a backup restore take it from the rows they write. - Compared by the leader. Every minute (
DIGEST_COMPARE_EVERY), the leader of each app group asks the other replicas for their digests (POST /internal/digests, only of groups the caller is a member of, at mostDIGEST_GROUPS_PER_ASKan ask) and compares them at the latest index both recorded. A difference is logged with the group, the peer and the index, and shows incalimero_group_diverged(Observability, runbook). - Across releases: replicas compare only digests of the same
DIGEST_VERSION, raised with any change to a row’s stored shape; a peer on a release before digests answers404and is not compared.
The system group is not covered: its machine keeps its state in memory and writes no rows as it applies.
Timing
Section titled “Timing”Setting (calimero_log::Config) |
Default |
|---|---|
| Heartbeat | 200 ms |
| Election timeout | 1.2–2.4 s |
| Snapshot after | 1000 applied entries |
| Log entries kept behind a snapshot | 100 |
| Snapshot chunk on the wire | 1 MiB (SNAPSHOT_CHUNK_BYTES) |
| Snapshot install timeout | 10 s per chunk RPC |
A frozen or partitioned leader is replaced after an election timeout. A call
forwarded to an app’s leader waits at most FORWARD_TIMEOUT (5 s) before the
forwarding node retries at whoever leads now; calls without a client
idempotency key get a node key first (node:…, remembered 10 minutes), so the
retry replays a call that did commit instead of running it twice. The process
suite’s frozen-leader (crates/node/tests/fault/suite.py) is the regression
test.
State, restarts and snapshots
Section titled “State, restarts and snapshots”An app group’s state lives on disk, in its group’s rows of the state column
(Storage schema); the machine keeps only a small Head
in memory (bundle, executor, the component’s host records: timers, ring
bounds, active runs, the last write’s time). Each applied
batch commits its writes and the applied index together, so a restarted node
resumes from where it stopped instead of replaying the log.
The system group’s machine keeps its state in memory and is checkpointed with every snapshot; a restart restores the checkpoint and replays the log after it.
The commit index is written without a sync: a crash can lose it or leave an
older one. On start openraft takes the applied index when the record is
behind it, replays the log up to the record when it is ahead, and learns the
rest from the leader, so no entry applies twice or is skipped
(a_lost_or_older_commit_index_is_learned_again and
an_in_memory_machine_applies_each_entry_once_after_losing_its_commit_index
in crates/log/tests/cluster.rs).
A group’s Raft core on a node stops when that node cannot go on safely: a
write the disk refused (EIO, ENOSPC), a machine that panicked in an entry, an
entry it could not apply. openraft treats each as fatal to the core, so the
group itself starts it again. As the core stops, the node drops what it applied
in memory (a batch that never committed, half an entry) and reads the group’s
state and applied index back from the disk: the last committed batch for an app
group, the last checkpoint for the system group. A stopped replica then answers
like one that lags (a read that waits for an index waits for it to be applied
again), and the node forwards its app’s calls and views to the other replicas.
After RESTART_FIRST (1 s) it starts a new core on the disk’s state, which
catches up from the log and the leader. A core that stops again within
RESTART_MAX (5 min) waits twice as long before the next try, up to that, with
up to a quarter of jitter, so a fault that persists (a full disk, an entry this
release cannot decode) keeps the group stopped without a hot loop. Each stop is
logged, and calimero_group_stopped and calimero_group_restarts_total show it
(Observability). Tests:
a_batch_that_fails_to_commit_stops_the_group_which_reloads_and_restarts,
a_machine_that_panics_stops_the_group_which_reloads_and_restarts and
a_fault_that_outlasts_a_restart_backs_off_and_shows_in_the_status in
crates/log/tests/cluster.rs;
a_node_whose_system_group_stopped_on_a_disk_error_rejoins_on_its_own in
crates/node/tests/cluster.rs.
A snapshot is a file of sealed frames under <data-dir>/snapshots, written
from a consistent RocksDB view of the group’s rows, never the whole state in
memory. A follower that needs one receives the file in chunks. Every node of
a cluster holds the same cluster key K_c, whatever release it runs, so a
follower opens the leader’s frames as they are. Installing writes the group’s rows under the next generation
(group ‖ generation ‖ key) and switches generations in one batch, so a crash
mid-install leaves the previous state intact. calimero-log’s tests stream a
~6 MiB state to a new node this way.
openraft builds a snapshot beside the installs and applies of the same group,
so a build reads its generation, its index and its RocksDB view under the one
lock an install switches them under, and records its file only if the
generation is still the one it read: a build that an install or a backup
restore overtook is made again, and never replaces the newer snapshot or
deletes its file (machine::build_races in crates/log).
Groups come and go with the directory
Section titled “Groups come and go with the directory”The system group’s app directory decides which nodes host which app groups. The reconcile loop on every node starts the groups listed for it and stops the rest, deleting their data from that node.
- Creation.
CreateAppassigns the next group number. The first replica in the instance’s replica list initializes the group with all replicas as voters; whichever replica becomes leader runsinitand appends thecomponentInitentry. The first replica stays one that holds the group: a move keeps the current replicas first in every list it records, and the nodes it adds after them, so a node that has not joined yet never forms a group of its own (initializing a node that already holds the group does nothing). - Late start. A Raft RPC for a group a node does not run yet goes through the engine’s group factory: the node starts it if the directory lists it as a replica, and answers “not hosted here” otherwise.
- Release gate. Before any Raft RPC (entries or snapshot) leaves for a
peer,
Node::may_sendchecks that the peer’s attested mero-tee release is one approved to hold that app’s data (Deployment governance). The system group goes to every attested peer. The receiving node makes the same check, and that the sender is a member of the group (api::internal_raft). Both ends run it for every heartbeat of every app group, so neither scans the directory: the system machine keeps an index from group to app and from node to its apps, and caches each (group, release) decision, all derived from the applied state and dropped when the next system entry applies or a snapshot restores (SystemMachine::app_of_group,apps_on,group_may_hold). Nothing of it is stored or replicated.reconcilereads the same index, so a pass costs this node’s groups, not groups times apps.
Replica moves
Section titled “Replica moves”POST /ops/apps/{app}/replicas { replicas } moves an app to a new replica set
(at least three when the cluster has three nodes). It runs on the app group’s
leader; any other replica answers 421 NOT_LEADER with details.leaderUrl
before writing anything. The leader:
- refuses any target node whose attested release the app’s owner has not approved;
- records the union of old and new replicas in the directory, so every node involved starts the group (the current replicas first, in their order, then the new ones, so the first replica is still one that formed or joined the group);
- hands each new node the app’s bundle and every blob the app’s state marks (bytes that live outside the log, and that a cluster grown from one node holds only on that node), so the new replica can run and serve the app without the old ones;
- adds each new node as a learner and waits for it to catch up;
- changes the voter set to the new replicas (joint consensus);
- records the new replica list (the replicas that stay first, in their order), after which reconcile stops the group on nodes that left.
A rebalancer may repeat or race a move. A move to the replica set the app
already has (as a set, in any order) writes nothing and answers
unchanged: true. While a move of an app runs on its leader, a second move of
that app is refused 409 MOVE_IN_PROGRESS before it writes anything, so two
moves never mix their replica sets in the directory; the move runs to the end
even if its caller disconnects. The guard is the leader’s: a move sent to a new
leader while the old one is still moving is refused by the group’s own
membership change instead.
Joining nodes
Section titled “Joining nodes”A node started without --bootstrap waits. Once the operator registers it
(POST /ops/nodes), the system leader adds it to the system group as a
learner; it catches up from the log or a snapshot, learns the directory,
and from then on placement and replica
moves can put apps on it.
Learners become voters up to 3. On every reconcile pass, a system leader
with fewer than 3 voters promotes registered learners that are caught up (at
most PROMOTE_MAX_LAG, 16 entries, behind its log) so that the system group has
exactly 3 voters, in one membership change (joint consensus). It never stops
at 2: two voters tolerate no failure and make every system write wait on both,
so a cluster started on one node keeps that one voter until two learners are
ready, then goes from 1 to 3. Learners in a zone no voter is in come first
(zone from POST /ops/nodes), then learners with no zone, then the rest. The
leader never grows the system group past 3 by itself; a fourth node stays a
learner (5 voters would be an operator’s choice, not implemented yet). Removing
a voter still promotes a registered learner while fewer than 3 are left
(DELETE /ops/nodes/{id}).
So a cluster can start on one node and grow: until the second and third nodes
are voters, the first is a single point of failure for every system write and
every access check (sys_fresh asks the system leader); from then on any one
node, the first included, can fail. Apps placed while the cluster was one node
have one replica; a replica move (ops placement, then ops move) spreads them.
Growing a cluster draws the whole
sequence.
Upgrading the on-disk format
Section titled “Upgrading the on-disk format”The store names its format (calimero_log::STORE_FORMAT, 2) in a sealed node
row, settled when the engine opens it: an empty store takes it, any other
format is refused (Format versions).
No earlier store, and no log.redb store from before RocksDB, is read: a
rollout’s new nodes start empty and receive their data by snapshot.