Storage schema

Everything a node persists is in its data directory (--data-dir, by default {home}/{node}/data):

<data-dir>/
├── store/          one RocksDB database for every group on this node
├── snapshots/      sealed snapshot files, one current per group
└── log.redb.migrated   (only if the node was upgraded from the previous format)

The database is opened with tuned column and table options (calimero-log’s tuning): compression, bloom filters, one bounded block cache (128 MiB). Log appends, votes and blobs are synced writes; RocksDB shares one WAL sync between the groups writing at once, and groups never queue behind one another.

Columns

Column	Key	Value
log	group (u64 BE) ‖ index (u64 BE)	A Raft entry
meta	group ‖ name	vote, committed, purged, applied (last applied log id and membership), generation, snapshot (the current snapshot’s metadata and file)
state	group ‖ generation (u32 BE) ‖ machine key	The state machine’s rows
blobs	name	Bundles, uploaded releases and app blobs, by hex(SHA-256)

Sealing

Every value is sealed with AES-256-GCM under the storage key K_r, with a random 96-bit nonce stored in front, and bound as associated data to its column tag, its group and its key: "cl" ‖ tag ‖ group ‖ key. A state row’s associated data names its machine key but not its generation, so installing a snapshot can move sealed rows to the next generation without opening them. A value copied to another row, group or column fails to open.

Two further keys are derived from K_r (HMAC-SHA256(K_r, label)):

• calimero.log.hidden-keys.v1 keys the HMAC that hides row keys;
• calimero.log.snapshot.v1 seals snapshot frames.

Hidden rows

A machine can hide a key (StateTx::put_hidden). The row is then stored at tag ‖ HMAC(group ‖ tag ‖ key) (33 bytes), and the value, before sealing, is u32 BE key length ‖ key ‖ value. The disk then shows neither which keys an app uses nor how they sort; reading a hidden key needs K_r.

An app group’s rows

An app group’s machine key space (crates/node/src/app.rs, rows):

Row	Hidden	Holds
k + the app’s key	yes	The app’s storage: what its runs write, private spaces, writer sets and blob marks (the host’s \0calimero.* keys)
i + caller|key	yes	Idempotency records: the answer replayed for a repeated key, and its expiry
h	no	The Head: bundle, initialized and init error, last applied index, pending follow-ups, timer ticks, event-ring and expiry-ring bounds, and a component’s host records (among them lastWriteMs, the timestamp of its last write, left out until the first)
e ‖ n (u64 BE)	no	Event n of the event ring (the last 10 000)
r ‖ class ‖ n	no	The idempotency keys in the order they expire, per TTL class (client keys, node keys)

Only the Head is kept in memory; runs read the app’s keys through the group’s state (KvView), so an app’s state is never loaded whole.

The system group keeps its state in memory and writes one checkpoint row (\0calimero.log.checkpoint) with each snapshot.

Snapshot files

A snapshot is snapshots/<…>.snap: a magic line (CALIMERO-SNAPSHOT-1), then frames of up to about 1 MiB of rows each, each frame sealed with associated data naming the group, the snapshot id, the frame’s position and whether it is the last. It is written from a consistent RocksDB view, so a large state is streamed rather than held in memory. A follower receiving one writes it to a file first, then installs it as the next generation in batches of about 4 MiB and switches generations in one batch (Replication).

Backups

A backup is a copy of a group’s current snapshot file under {backup-dir}/{app}/{index:020}.{group}.{snapshot}.snap, with its sealed manifest beside it (….snap.manifest, backup format 1; a lone snapshot file is format 0) (The operator). backup::find checks a backup against its manifest; backup::latest opens the newest one with K_c and returns its index and rows. A restore writes a backup’s rows into a new group (Group::replace_state): each row is opened and sealed again for that group, and a hidden key is hidden again under it.

Migration from the previous format

A data directory holding the previous single redb file (log.redb) is migrated when the engine opens it: entries, votes, commit and purge marks, blobs, and each machine’s old whole-state snapshot, which the app machine imports into its rows (the app’s keys, events, idempotency records with their expiry ring). The old file is kept, renamed log.redb.migrated.

Inspecting a stopped node

merod inspect [--storage-key-file F] summarizes the store’s layout and sealed sizes; with the key, it also opens each group’s metadata (vote, applied index, membership). It opens the store for writing, so stop the node first.