Storage Layer

calimero-store + calimero-store-rocksdb + calimero-dag + calimero-storage

column families

20+

group key prefixes

crates

AES-GCM

optional encryption

Purpose

The storage layer provides a column-family key-value abstraction over RocksDB. The core Database trait exposes has, get, put, delete, iter, and apply(Transaction). Keys are typed via generic_array, giving compile-time guarantees on key size and column assignment. An optional AES-GCM encryption layer transparently encrypts values at rest.

pub trait Database {
    fn has(&self, col: Column, key: &[u8]) -> Result<bool>;
    fn get(&self, col: Column, key: &[u8]) -> Result<Option<Slice>>;
    fn put(&self, col: Column, key: &[u8], value: &[u8]) -> Result<()>;
    fn delete(&self, col: Column, key: &[u8]) -> Result<()>;
    fn iter(&self, col: Column) -> Result<DBIterator>;
    fn apply(&self, tx: Transaction) -> Result<()>;
}

Column Architecture

All persistent data is partitioned into 10 column families. Each maps to a dedicated RocksDB column family with independent compaction and bloom filters. The Group column is the most complex, containing 20+ logical key prefixes for governance state.

Membership

Resources

OpLog / Hashing

Lifecycle

Informal group::… labels in the diagram map to the typed keys and single-byte prefixes in Storage Schema (Group column): group::info → GroupMeta (0x20); group::member → GroupMember (0x21); group::role → role data on GroupMember values (0x21); group::context → GroupContextIndex (0x22) / reverse index ContextGroupRef (0x23); group::upgrade → GroupUpgradeKey (0x24); group::signing_key → GroupSigningKey (0x25); group::cap → GroupMemberCapability (0x26); group::visibility → GroupContextVisibility (0x27); group::allowlist → GroupContextAllowlist (0x28); group::settings (defaults) → GroupDefaultCaps (0x29), GroupDefaultVis (0x2A); migration markers → GroupContextLastMigration (0x2B); group::nonce → GroupLocalGovNonce (0x2C); group::alias (member) → GroupMemberAlias (0x2D), group/context names → GroupAlias (0x2E), GroupContextAlias (0x2F); group::oplog → GroupOpLog (0x30); group::oplog_head → GroupOpHead (0x31); member–context links → GroupMemberContext (0x32), GroupContextMemberCap (0x33). Some cells (e.g. group::invite, group::state_hash) are illustrative and do not match a single prefix—use the schema tables as the source of truth.

Key Model

All keys are statically typed via Key<T>, a newtype over GenericArray<u8, T::Size>. The AsKeyParts / FromKeyParts traits define how a key is decomposed into column + byte components, giving compile-time column assignment.

pub struct Key<T: KeyParts>(GenericArray<u8, T::Size>);

pub trait AsKeyParts {
    type Size: ArrayLength<u8>;
    const COLUMN: Column;
    fn as_key(&self) -> Key<Self>;
}

pub trait FromKeyParts: AsKeyParts {
    fn from_key(key: Key<Self>) -> Self;
}

Key Types by Column

calimero-dag

A generic, in-memory causal DAG used for both context state deltas and governance operation logs. Provides topological ordering, pending queues for out-of-order delivery, and missing-parent detection for catch-up.

crates/dag

CausalDelta<T>

pub struct CausalDelta<T> {
    id: DeltaId,
    parents: Vec<DeltaId>,
    payload: T,
    timestamp: HLC,
    expected_root_hash: Hash,
}

Each delta records its causal parents, forming a partial order. The expected_root_hash enables fast consistency checks — a peer can verify it arrived at the same state after applying a delta.

DagStore<T>

pub struct DagStore<T> {
    deltas: HashMap<DeltaId, CausalDelta<T>>,
    applied: HashSet<DeltaId>,
    pending: Vec<CausalDelta<T>>,
    heads: HashSet<DeltaId>,
}

Tracks all known deltas, which have been applied, which are pending (missing parents), and the current DAG head set. New deltas promote heads automatically.

DeltaApplier<T> trait

pub trait DeltaApplier<T> {
fn apply_delta(&mut self, delta: &CausalDelta<T>) -> Result<()>;
fn restore_applied_delta(&mut self, delta: &CausalDelta<T>) -> Result<()>;
}

Key Operations

Topological Ordering

Before applying, all pending deltas are sorted in topological order (parents before children). This ensures deterministic replay regardless of arrival order.

Pending Queue

If a delta's parents haven't been seen yet, it enters the pending queue. When missing parents arrive, queued deltas are automatically drained and applied.

restore_applied_delta

Used during node restart to rebuild the in-memory DAG from persisted deltas without re-executing the payload (state is already in storage).

get_missing_parents

Returns the set of delta IDs referenced as parents but not yet received. Used by the sync protocol to request specific deltas from peers.

calimero-storage

Provides CRDT collections used by the WASM runtime for conflict-free replicated state. Each collection implements the Mergeable trait for automatic conflict resolution during sync.

crates/storage

Map

Observed-remove map. Concurrent puts to the same key are resolved by LWW using HybridTimestamp. Deletions are tracked as tombstones until causally stable.

Set

Observed-remove set. Add/remove conflicts resolved in favor of add (add-wins semantics). Internally backed by a Map with unit values.

LwwRegister

Last-writer-wins register. Stores a single value with a HybridTimestamp. On merge, the value with the highest timestamp wins.

Core Traits

pub struct HybridTimestamp {
    wall_clock: u64,
    logical: u32,
    node_id: NodeId, // tiebreaker
}

pub trait Mergeable {
    fn merge(&mut self, other: &Self);
}

The Mergeable trait is the fundamental building block — any type that implements it can be used as a CRDT value in the storage layer. The runtime's host functions call merge when applying remote deltas.

RocksDB Implementation

The calimero-store-rocksdb crate provides the concrete Database implementation backed by RocksDB.

crates/store/rocksdb

Column Family Mapping

Each Column enum variant maps 1:1 to a RocksDB column family. CFs are created at DB open time. Each has independent compaction, bloom filters (10 bits/key), and block cache partitions.

WriteBatch Transactions

The Transaction type accumulates puts and deletes, then is atomically committed via WriteBatch. Guarantees all-or-nothing semantics for multi-key operations like delta application.

Snapshot Iteration

Iterators are backed by RocksDB snapshots for consistent point-in-time reads. Prefix iteration uses set_iterate_range for efficient scans within a column family.

Pinned Gets

Uses get_pinned_cf for zero-copy reads where possible. The returned Slice borrows directly from the block cache, avoiding allocation for large values.

CRDT Collections

The calimero-storage crate provides application-level CRDT collections built on top of the storage layer. These are what SDK applications use for state management.

Available Collections

UnorderedMap

Key-value store with LWW (Last-Write-Wins) semantics per entry. Entries can be inserted, updated, and removed independently across nodes.

Vector

Ordered append-only list. Items are pushed to the end. Positional inserts and removes use index-based CRDT logic.

Counter

G-Counter (grow-only). Each node increments its own slot; value() returns the sum across all nodes. Naturally commutative and idempotent.

Merge Semantics

All collections implement the Mergeable trait. When state deltas arrive from peers, the storage layer calls merge() on each affected entry. The merge is:

Commutative — merge(A, B) = merge(B, A)
Associative — merge(merge(A, B), C) = merge(A, merge(B, C))
Idempotent — merge(A, A) = A

These properties guarantee eventual consistency regardless of message ordering or duplication.