Testing strategy

Calimero Core is tested in layers that trade scope for speed. Most of the node is covered by real clusters in one process: three nodes, sealed channels, real WASM, driven only through the client. Multi-process behaviour (crashes, freezes, restarts, upgrades) is covered by the process suite, which runs one merod process per node, and data written by the last release by the upgrade path.

Layer	Where	What it proves	Speed
Unit tests	#[cfg(test)] in each crate	One module’s logic: config merging, key sources, quotas, presence rules, outbound gates	Milliseconds
Raft engine	crates/log	Replication, failover, restart from disk, learner catch-up, snapshot streaming, no plaintext on disk, the log.redb migration	Seconds
Cluster	crates/node/tests/cluster.rs	The node end to end in one process: sign-up, installs, calls, views, events, orgs and teams, upgrades, blobs, governance, replica moves	Seconds to minutes
CLI	crates/merod/tests/cli.rs, crates/meroctl/tests/cli.rs	merod init/config/run/inspect through the binary; every meroctl command group against a local cluster	Seconds to minutes
App logic	TestApp in crates/sdk/src/testing (cargo test --manifest-path apps/<app>/Cargo.toml)	An app’s methods, events and errors on the real engine	Milliseconds to seconds
Scenario files	apps/*/scenarios/, workflows/**/scenarios/, run by crates/node/tests/scenarios.rs	An app or a flow end to end on an in-process cluster, written as data: orgs, teams, invites, packages, upgrades, blobs, calls as several accounts	Seconds per file
Process suite	crates/node/tests/fault/suite.py	Separate merod run --dev processes: a crash, a freeze and a restart, a frozen leader, orgs and keys, signed packages, a rolling upgrade under load, the kv-store soak	Minutes
Upgrade path	crates/node/tests/fault/upgrade_path.py	A cluster on the latest published release’s merod, its data written through the public API, rolled onto this tree’s merod as production rolls it, then read back, extended and read again after a restart	About two minutes, plus the builds

Use both ends. A green cluster test does not prove a process restart or a frozen leader behaves, and a green scenario does not localize a logic bug.

Fixtures

The cluster and CLI tests build the components they load on first use (calimero_app_build::build_component_cached, cached under $CARGO_TARGET_DIR/app-components; the wasm32-wasip2 target, which rust-toolchain.toml lists): the kv fixture apps/v2-fixtures/kv, the other apps/v2-fixtures/*, apps/kv-store-with-shared-storage and apps/rooms. No build step comes first. KV_STORE_WASM points meroctl’s tests at another component.

Running the tests

cargo test -p calimero-node                 # the cluster tests
cargo test -p calimero-node --test cluster teams_work_like_v1_subgroups -- --nocapture
cargo test -p merod -p meroctl                    # the CLI tests
cargo test -p calimero-log
cargo test -p calimero-client -p calimero-identity -p calimero-sealed
cargo test -p calimero-outbound --features outbound-testing
cargo test -p calimero-kms-client --features mock-attestation
cargo test -p calimero-tee-attestation --features mock-attestation
cargo test --workspace                            # everything

The real-process fault tests start merod processes on ports 7600-7699 and need release merod and meroctl (cargo build --release -p merod -p meroctl):

python3 -I crates/node/tests/fault/rooms_fault.py --iterations 200 --seed 1   # Rooms under leader kills
python3 -I crates/node/tests/fault/compute_e2e.py                              # compute actions: 3 replicas, 2 job nodes

cargo-nextest runs the workspace faster (cargo nextest run --workspace, then cargo test --workspace --doc for the doctests it skips).

Caution

Cluster tests start several nodes on free local ports and use real WASM; run them on Linux. A test that fails with “build kv-store for wasm32 first” is missing a fixture, not broken.

Writing a cluster test

crates/node/tests/cluster.rs builds clusters with the real run, mock attestation and a soft authenticator, and drives them only through calimero-client, as an SDK user would. Write a new test there when the behaviour crosses the API: start the cluster, sign up, do the thing, assert on what the client sees. Name it for the behaviour (teams_work_like_v1_subgroups, a_node_added_to_a_running_cluster_hosts_replicas).

App logic: TestApp

calimero-sdk’s testing feature (crates/sdk/src/testing) runs an app as the real calimero:app component on the real wasmtime engine with an in-memory host: TestApp::new, as_(account).call(..) / .view(..), events_for, tick, advance, mock_http, take_xcalls / deliver, and run_scenario for the shared JSON scenarios below. A component app’s own tests/ use it (cargo test --manifest-path apps/<app>/Cargo.toml); the app author’s view is Testing apps.

Scenarios

Scenario files are end-to-end tests written as data: each runs on an in-process cluster of its own (crates/node/tests/harness/cluster.rs: real run, mock attestation, sealed channels, the real WASM, a package registry on loopback), driven only through calimero-client. One test binary runs every file it finds:

cargo test -p calimero-node --test scenarios                       # all of them
SCENARIO_FILTER=apps/blobs/,app-migration/02 cargo test -p calimero-node --test scenarios
cargo test -p calimero-node --test scenarios -- fugue-collab        # a filter argument does the same
SCENARIO_SHARD=3/4 cargo test -p calimero-node --test scenarios -- --test-threads 2
cargo test -p calimero-node --test scenarios -- --list              # what would run

It has its own test main (harness = false): --test-threads N (default 2) scenarios run at once, each on its own cluster; SCENARIO_SHARD=k/n keeps every n-th file of the sorted list starting at the k-th, so n shards split the files evenly and deterministically. A file listed in crates/node/tests/scenarios-skip.txt (<path> <reason>) is reported ignored until --include-ignored (or SCENARIO_INCLUDE_SKIPPED=1).

A run fails when it discovers no scenario, or when a filter matches none. Every run also checks that each *.json (other than .org.json) under any directory named scenarios is discovered or listed, with its reason, in crates/node/tests/scenarios-exclude.txt, so no scenario sits on disk unrun without saying why; a stale or reasonless entry fails too.

Variable	Default	What it does
SCENARIO_FILTER	all	Comma-separated substrings; a path containing any of them runs
SCENARIO_SHARD	none	k/n: the k-th of n shards (1-based)
SCENARIO_WAIT_CAP_SECS	2	Longest a wait step sleeps, unless it says "exact": true: the cluster replicates synchronously, and the ported waits were for v1’s gossip
SCENARIO_FUZZY_SECONDS	10	How long a fuzzy_test runs (soak lengths are the Python suite’s)
SCENARIO_TIMEOUT_SECS	600	A scenario still running after this, plus its own fuzzing (each fuzzy_test step’s SCENARIO_FUZZY_SECONDS), fails
SCENARIO_REPEAT_CAP	none	Most iterations of a repeat step
CORE_WASM_DIR	$CARGO_TARGET_DIR/workflow-wasm	Where {{env.CORE_WASM_DIR}} fixtures live
SCENARIO_FILES	none	Comma-separated cluster scenario files to run instead of the discovered ones (crates/node/tests/previous-release/kv-store.json, which needs the previous release’s bundle in PREVIOUS_RELEASE_DIR: scripts/fetch-release-app.py)

After each call the runner waits, up to 10 s, until every replica has applied it. A call that does not settle fails its step, naming each replica behind, the index it waited for and the index that replica had reached. The runner never makes up a refusal: every refusal a step expects is the node’s or the app’s.

Fixtures are built on first use when missing: <crate>.wasm by building that app’s component, and the blobs size sweep’s static/generated/blob-<N>mib.bin as random bytes in memory.

App-level scenarios

apps/<app>/scenarios/<name>.json is the step array of calimero_sdk::testing::scenario (call, view, tick, advance, removeMember; as, expect, save, $var, @account, events), the same file the app’s own tests/scenarios.rs runs in the harness. Beside it, <name>.org.json is the organization it needs:

{ "admins": ["ada"], "members": ["bob", "cy"], "guests": ["gus"], "teams": { "sales": ["bob"] }, "outsiders": ["mallory"] }

The owner is always owner; an omitted key is empty, and no file means the owner alone. outsiders are accounts a step acts as that hold no place in the org: they sign up and join nothing, so the node refuses them. On the cluster an as that is neither in the org nor a declared outsider fails the scenario before it runs, so a mistyped name cannot become an outsider whose refusal passes. (The SDK’s TestApp harness still treats an unknown name as an outsider.) On the cluster the runner builds the app (a component, with calimero_app_build::build_component_cached), signs every account up, creates the organization, invites each account with its role, creates the teams and adds their members, deploys the app in the organization and runs the steps: step i on node i % 3, as its as account’s own client. @name is that account’s id; a string that is exactly a team’s name in args is that team’s id on the cluster (and a team id in a result reads back as its name). The app clock starts at 1_700_000_100_000 (a 15-minute slot boundary, the harness tests’ time zero); advance moves every node’s clock, tick waits until the timer’s due slot is applied everywhere. Events are each account’s live stream from the log’s start.

A migration scenario, apps/<app>/migrations/<version>.json, is the same step array with one { "op": "upgrade" } step: the app’s own tests/migrations.rs runs it with run_migration, from the component in apps/<app>/releases/<version>.mpk to the crate’s current build (Testing apps). The cluster runner does not discover these files (they are not in a scenarios directory); a node-level upgrade is a cluster scenario’s upgrade_app step.

Cluster scenarios

apps/<app>/scenarios/cluster/**/*.json and workflows/**/scenarios/**/*.json hold node-level steps a harness cannot run. A file is an object:

{
  "name": "Blobs across nodes",
  "description": "...",
  "comment": ["what the YAML's header said: v1 steps dropped in the port, fixes"],
  "cluster": { "nodes": 3, "settings": "[outbound]\nenabled = true\n..." },
  "accounts": ["alice", "bob"],
  "variables": {},
  "steps": [
    { "type": "create_org", "account": "alice", "org_name": "Blobs", "outputs": { "org": "orgId" } },
    { "type": "deploy", "account": "alice", "org": "{{org}}",
      "wasm": "{{env.CORE_WASM_DIR}}/kv_store.wasm", "outputs": { "app": "contextId" } },
    { "type": "upload_blob", "account": "bob", "app": "{{app}}", "file_path": "static/test.txt",
      "outputs": { "blob": "blob_id" } },
    { "type": "download_blob", "account": "alice", "app": "{{app}}", "blob_id": "{{blob}}",
      "expected_size": 13 },
    { "type": "assert", "statements": ["is_set({{blob}})"] }
  ]
}

It is merobox’s cloud-v2 workflow format as JSON, so a workflow converts mechanically (scripts/workflow-to-scenario.py FILE.yml [--delete]; it drops base_port and binary and refuses what it cannot carry over, naming why). cluster.settings is the TOML every node starts with (its [outbound] table; the test build has outbound-testing). Every account in accounts is signed up with its own soft passkey on node 1 and talks to node index % nodes (an explicit "node": N, 1-based, overrides).

Conventions, as merobox’s runner had them:

• {{name}} placeholders: a string that is one placeholder keeps the value’s type; one inside text is spliced in (json.dumps of a non-string). Dotted paths reach into values ({{accounts.alice.account_id}}, {{env.HOME}}), {{a ?? b}} takes the first that resolves, and an unresolved one fails the step. Set for you: accounts.<name>.{account_id,public_key}, public_key_<name>, env.*, ops_token.
• outputs: { var: path } stores a path of the step’s result ("" the whole result); a call’s or view’s result also has rpc.output (its result, as v1’s JSON-RPC envelope had it).
• expected_error: CODE makes a refusal with that API code the step’s success, expected_failure: true any refusal, allow_failure: true either outcome; an output whose path starts with error then holds the refusal’s text. expect compares a step’s result; retry_for: S retries a failing step for S seconds; max_seconds: S fails one that succeeds too late.
• assert statements: is_set(A), is_empty(A), contains(A, B), not_contains(A, B), regex(A, P), equal(A, B) / not_equal(A, B) (also equals / not_equals), A op B for == != >= > <= < (numeric when both sides are numbers); non_blocking: true reports without failing. json_assert: json_equal, json_not_equal, json_subset (and unprefixed). Operands are evaluated as merobox did, with Python’s semantics (crates/node/tests/harness/py.rs): str of a map is its Python repr, 1 == 1.0, a whole {{name}} keeps its type and names a variable directly (no dotted paths), and an unresolved one fails its statement.
• repeat { count, steps } sets iteration (from 0) and iteration_node (from 1). fuzzy_test { operations: [{ name, weight, steps }], accounts, success_threshold, operation_delay_ms } picks weighted operations at random for SCENARIO_FUZZY_SECONDS, with {{random_account}}, {{random_int(a, b)}}, {{uuid}}, {{timestamp}} and, after a call, {{fuzzy_<arg>}}; it passes at success_threshold percent.

Step types (fields as in merobox; each acting step names an account, or an api_key, or as_key, an account’s public key):

Area	Steps
Apps	deploy (wasm or package + version/service; org, access, init_args, confirm_methods (ignored: a component declares its own), app_name) answers appId, contextId, bundle, memberPublicKey; upload_bundle (hash); create_app; upgrade_app (version/package/bundle/migrate, approved with the account’s passkey when the node asks); delete_app; migration_status (apps)
Packages	install_package (mpk: published to the cluster’s registry, then POST /packages/install; answers package, version), packages, uninstall_package
Calls	call (args, confirm, idempotency_key), view (stale, min_index), wait_for (equals/contains, timeout, interval), events (from, count, timeout)
Accounts	whoami (adds publicKey), add_device (another passkey on the account; device then acts as it), create_api_key
Orgs	create_org, invite (role, teams), redeem_invite, remove_member, set_role, capabilities, member_capabilities
Teams	create_team (team_name, join, parent), team_add (members, role, lead, capabilities), team_remove, team_update (join, metadata, team_name, parent), team_get, team_list, team_join, team_delete
Blobs	upload_blob (file_path: beside the scenario or any directory above it; answers blobId, blob_id, size), download_blob (expected_size, expected_sha256), list_blobs, blob_info, delete_blob
Presence	publish_presence (seq, state), presence
Approvals	transfer_app, approve_release, approve_outbound, upgrade_org, upgrade_run, retry_upgrade_run
Raw	http (method, path, body; a 412 CONFIRM_REQUIRED answered with the passkey)
Flow	wait (seconds, capped; exact), set (values), assert, json_assert, repeat, fuzzy_test, find_leader

Node process faults (kill_node, pause_node, resume_node, start_node, restart_node, ops) are not steps here: a scenario naming one is refused. They need real processes, and run in the process suite.

Converted from merobox workflows

Core’s and the apps’ functional workflows (142 YAML files under workflows/** and apps/*/workflows/**, written for merobox, the external runner core used before) were converted to cluster scenarios by scripts/workflow-to-scenario.py and run in process; merobox is no longer used. What needed separate processes (merobox’s cloud-v2 examples and its fuzzy-load soak) moved to the process suite. The table below records the v1 workflows deleted earlier rather than ported.

Workflows deleted in the migration

A v1 workflow whose subject has no counterpart on a cluster was deleted rather than kept unrun. Each, and why nothing replaces it (or what does):

Workflow	Its subject, and why a cluster has none
apps/blobs/workflows/blob-presence-multi-holder.yml	DHT provider discovery among user nodes holding a blob: a cluster’s nodes all keep an app’s blobs (blob-cross-node-transfer, blob-cross-node-sizes)
apps/blobs/workflows/blob-presence-tee-serves.yml	a ReadOnlyTee availability node prefetching blobs: every cluster node is the TEE and keeps the app’s blobs
apps/fugue-collab/workflows/fugue-collab-figure2.yml	the Fugue paper’s Figure 2, three isolated replicas typing at positions of their own unsynced views and merged after: a cluster orders every edit against one current text, so there is no concurrent merge (the other fugue-collab workflows cover editing)
apps/fugue-collab/workflows/fugue-collab-large-paste.yml	a paste merged across a libp2p partition: a cluster has no partitioned user replica (concurrent edits: the other fugue-collab workflows)
apps/fugue-collab/workflows/fugue-collab-partition-heal.yml	two sides of a network partition typing at the same position and merging when it heals: a cluster has no partitioned user replica and no merge
apps/kv-store/workflows/ephemeral-open-subgroup-e2e.yml	presence over v1’s WS/SSE for a member inheriting an open subgroup, checked by a script on v1’s endpoints (the cluster test presence_reaches_every_node_and_expires covers presence)
apps/kv-store/workflows/ephemeral-presence-e2e.yml	presence delivered over v1’s WS/SSE, replayed on subscribe and expired, checked by Node scripts on v1’s endpoints (the port had already dropped them, leaving a bootstrap): the cluster test presence_reaches_every_node_and_expires (crates/node/tests/cluster.rs) covers presence
apps/nested-crdt-test/workflows/nested-crdt-offline-catchup.yml	an offline node catching up over v1’s HashComparison sync: a cluster replicates by its Raft log
apps/rich-collab/workflows/rich-catchup.yml	offline and replaced nodes catching up over v1’s HashComparison sync: a cluster replicates by its Raft log
apps/scaffolding-e2e/workflows/account-namespace-left-unfollows.yml	v1 devices following and unfollowing an account’s namespaces: a device is a passkey of the account and follows nothing
apps/scaffolding-e2e/workflows/account-pairing-application-scope.yml	pairing scoped by namespace and application: a passkey signs in to the whole account
apps/scaffolding-e2e/workflows/account-pairing-missed-publish.yml	a partitioned device missing the pairing’s one-shot publish of its link and scope key: no keys are delivered to devices
apps/scaffolding-e2e/workflows/account-registry-replicates.yml	the v1 account namespace recording each device’s scope: an account’s passkeys are cluster system state
apps/scaffolding-e2e/workflows/account-revoke-lost-device.yml	revoking a device with a proof minted from the account root’s recovery phrase: an account has passkeys, no root key
apps/scaffolding-e2e/workflows/account-revoked-device-misses-new-restricted-subgroup.yml	a revoked device’s key excluded from a subgroup opened later (key delivery): removing a passkey ends its sessions (account-device-revoke-lockout)
apps/scaffolding-e2e/workflows/account-revoked-device-repairs.yml	a revoked node re-minting its device and re-pairing: a removed passkey is gone; signing in again is a new passkey (add_device)
apps/scaffolding-e2e/workflows/account-sibling-projected-by-founder.yml	a device carried into a namespace by a sibling device: a device is a passkey of the account and needs no carrying
apps/scaffolding-e2e/workflows/admit-without-credential.yml	v1’s admission handler publishing a keyholder’s signed join without a node credential: an invite token is redeemed by a signed-in person
apps/scaffolding-e2e/workflows/delegated-authorship.yml	a keyholder writing through a relay under a signed warrant (delegated authorship): there are no relays, the node runs every call for the caller it authenticated
apps/scaffolding-e2e/workflows/delegated-invitation-routing.yml	routing a join to an admitter named in the invitation: there are no admitters, any node redeems an invite
apps/scaffolding-e2e/workflows/delegated-proof.yml	delegated execution by a signed request (CallerProof) to a node the caller has no account on: every call is the authenticated caller’s own
apps/scaffolding-e2e/workflows/delegated-session.yml	a device key obtaining a session on a node it does not own (delegated sessions): a passkey signs in to any node
apps/scaffolding-e2e/workflows/group-held-flip-applies-when-member-added.yml	a governance op held until the subgroup key arrives: teams are system state with no keys to deliver
apps/scaffolding-e2e/workflows/group-held-flip-nonmember-keeps-up.yml	a sealed governance op a non-member cannot settle stalling its governance: teams are system state ordered by one log
apps/scaffolding-e2e/workflows/group-join-mesh-not-ready.yml	a join failing closed while no admitter is reachable over the mesh, then recovering: there are no admitters, any node redeems an invite (38-namespace-invite-join-regression covers joining)
apps/scaffolding-e2e/workflows/group-leave-context.yml	leave_context, a node-local opt-out that stops a node syncing one context: a member holds no replica to stop; access follows the organization and team
apps/scaffolding-e2e/workflows/kad-routing-from-identify.yml	libp2p Kademlia routing among user nodes: a cluster’s peers are its attested nodes
apps/scaffolding-e2e/workflows/member-removed-partition-window-reconcile.yml	v1’s reconcile rolling back a write a removed member made in a partition window the admin had not seen: a cluster orders the write and the removal in one log, so there is no window to reconcile
apps/scaffolding-e2e/workflows/namespace-restore-kicked-member.yml	restoring a namespace a node removed from its own disk: a cluster keeps no per-member copy to remove
apps/scaffolding-e2e/workflows/namespace-restore-without-invitation.yml	restoring a namespace a node removed from its own disk: a cluster keeps no per-member copy to remove
apps/scaffolding-e2e/workflows/offline-account-root.yml	a node joining under a certificate its account root signed elsewhere (the root kept offline): an account has passkeys, no root key
apps/scaffolding-e2e/workflows/public-intents.yml	a relay serving delegated execution with –public-intents: there are no relays
apps/scaffolding-e2e/workflows/relay-device-login.yml	a fleet relay in proxy auth mode behind mero-auth serving device-key login: there are no relays, and every node authenticates its callers
apps/scaffolding-e2e/workflows/sync-resilience-lossy-link.yml	v1 sync recovering from packet loss on a user node’s link: a cluster replicates by its Raft log
apps/scaffolding-e2e/workflows/sync-resilience-network-partition-symmetric.yml	v1 sync and relay reservations across a NAT partition: a cluster replicates by its Raft log
apps/scaffolding-e2e/workflows/sync-resilience-network-partition.yml	v1 sync and relay reservations across a NAT partition: a cluster replicates by its Raft log
apps/scaffolding-e2e/workflows/sync-resilience-peer-pause-symmetric.yml	v1 sync after a paused peer re-dials its relay: a cluster replicates by its Raft log
apps/scaffolding-e2e/workflows/sync-resilience-peer-pause.yml	v1 sync after a paused peer re-dials its relay: a cluster replicates by its Raft log
apps/scaffolding-e2e/workflows/sync-resilience-peer-restart-symmetric.yml	v1 sync after a restarted peer re-registers its relay reservation: a cluster replicates by its Raft log
apps/scaffolding-e2e/workflows/sync-resilience-peer-restart.yml	v1 sync after a restarted peer re-registers its relay reservation: a cluster replicates by its Raft log
apps/scaffolding-e2e/workflows/tee-admission-mode-relay.yml	a namespace’s TEE admission mode (RelayTee vs ReadOnlyTee): the cluster is the TEE and there are no relays
apps/tee-cards/workflows/tee-cards-failover.yml	failover between TEE authorities elected per trigger: the cluster is the TEE and its leader runs every trigger (tee-cards covers the game)
apps/tee-cards/workflows/tee-cards-late-tee.yml	a TEE admitted later opening a deck sealed before (the namespace TEE key): the cluster is the TEE and holds the app’s key
apps/tee-cards/workflows/tee-cards-rotate.yml	rotating the namespace TEE key when a TEE is removed: the cluster is the TEE and holds the app’s key
workflows/app-migration/26-owner-driven-authored.yml	identity-gated entries migrating lazily at each owner’s next write: an upgrade migrates the app’s one state at once (16 covers UserStorage)
workflows/app-migration/28-get-migration-status.yml	the rollup of members’ migration heartbeats: no member holds its own copy; an upgrade run reports the apps (00, 14)
workflows/app-migration/36-chained-catchup.yml	a member left several releases behind replaying the upgrade ladder lazily: no member replica falls behind, each upgrade migrates at once
workflows/app-migration/37-stranded-resync.yml	a member stranded by an unobtainable intermediate release, recovered by resync: no member replica falls behind
workflows/app-migration/42-activate-upgrade-on-an-idle-node.yml	activating an upgrade on a member’s idle node (POST …/upgrade/activate): the cluster installs and switches every node
workflows/app-migration/43-retarget-before-activate.yml	a member’s activation refused after the admin retargeted: there is no per-member activation
workflows/app-migration/45-frontend-two-releases.yml	each release serving its own frontend, checked by a v1 script on the app host: the node serves a package’s frontend from its own origin ({label}.{apps_domain}, routed by Host), which no runner step addresses; the cluster test app_frontends_are_served_and_their_sessions_are_bound (crates/node/tests/cluster.rs) covers serving a release’s frontend there, one release only
workflows/app-migration/46-frontend-proxy-mode.yml	an app host on a node in proxy auth mode passing no token: a cluster node always authenticates
workflows/app-migration/47-missing-releases-before-activate.yml	a member installing the releases it missed before activating: the cluster installs releases on every node
workflows/app-migration/48-member-installs-only-the-newest-release.yml	a member that installed only the newest release replaying the ladder: the cluster installs and migrates at once
workflows/auth-seam.yml	v1’s client tokens minted by the auth frontend (POST /admin/client-key): a cluster signs in with passkeys and scopes API keys (the process suite’s orgs-and-keys)

Process suite

crates/node/tests/fault/suite.py runs the tests that need one merod process per node, so a test can SIGKILL, SIGSTOP, restart or upgrade one node: a crashed, a frozen and a restarted node (kv-failover); a frozen leader under a 10 s write budget (frozen-leader); organizations, scoped API keys and passkey confirmations (orgs-and-keys); signed packages from a registry on loopback (packages); a rolling upgrade of every node under a keyed write load (rolling-upgrade); the kv-store soak (kv-soak); and, nightly, rooms_fault.py’s randomized nemesis (rooms-fault). Each test starts its own 3-node merod run --dev cluster (rv2.py) on ports 7600-7699 (+ WORKFLOW_PORT_OFFSET), drives it over plain HTTP with API keys and through meroctl, and stops it; a failed test keeps its directory under target/fault-suite, with each node’s nodeN.log.

Set	What runs
default (PRs)	every test but rooms-fault, once, a 30 s soak
--smoke	the release-image set: no rolling-upgrade (one image, one build) and no rooms-fault; with --merod PATH, against another binary
--nightly	every test three times, a 900 s soak, 50 rooms-fault iterations

--only a,b and --iterations N pick and repeat tests. Its README.md (beside it) lists each test’s checks and every input:

cargo build --release -p merod -p meroctl
scripts/install-mero-app-build.sh   # once: puts mero-app-build on PATH
mero-app-build apps/v2-fixtures/kv --out target/app-components/kv.wasm
for v in 1.0.0 1.1.0; do
  python3 -I scripts/pack-component.py --wasm target/app-components/kv.wasm \
    --package com.calimero.kv-store --app-version "$v" --out "dist/com.calimero.kv-store-$v.mpk"
done
python3 -I crates/node/tests/fault/suite.py --list
python3 -I crates/node/tests/fault/suite.py                        # the PR set
python3 -I crates/node/tests/fault/suite.py --only kv-failover,frozen-leader
python3 -I crates/node/tests/fault/suite.py --smoke --merod /tmp/merod

The kv fixture is the component apps/v2-fixtures/kv; the two signed releases (package com.calimero.kv-store, dev key) are for packages and land in dist/. A node behaviour that only shows across processes (the bounded forward to a frozen leader is the example) gets its regression test here.

Soak latency and throughput

The suite’s kv-soak (kv-store’s fuzzy load: the operations and weights of workflows/fuzzy-tests/kv-store/scenarios/fuzzy-test.json, four members) times every call it sends to the cluster (one round trip each). With --soak-report file.json (or SOAK_REPORT) it appends a report to that file: per operation, how many ran and failed, operations and calls per second over the run, and the p50/p95/p99 (and mean, max) latency of its calls in milliseconds, the same per call kind (call set, call get), and the whole run’s (the JSON format merobox’s fuzzy_test wrote, so older reports still render). crates/node/tests/fault/soak.py report file.json (or scripts/soak-summary.sh, which CI runs) prints it as a table:

python3 -I crates/node/tests/fault/suite.py --only kv-soak --soak-seconds 30 --soak-report soak.json
python3 -I crates/node/tests/fault/soak.py report soak.json

A 30-second kv-store run against a debug merod on a shared 4-vCPU Linux container (the shape of the nightly table, not a reference):

Operation	Ops	Pass %	Ops/s	Calls	Calls/s	p50 ms	p95 ms	p99 ms
cross_node_write_read	7	100.0	0.23	14	0.46	19.5	21.8	21.8
list_entries	29	100.0	0.96	29	0.96	43.3	74.1	85.3
random_read	28	100.0	0.93	28	0.93	11.6	16.2	20.0
set_and_verify	96	100.0	3.18	192	6.36	17.8	29.3	31.9
write_burst	56	100.0	1.86	672	22.28	25.1	30.9	33.5
all (30s)	216	100.0	7.16	935	31.00	22.1	31.8	51.1

The calls run one at a time, so a call’s latency is what one caller sees and calls/s is one caller’s rate, not the cluster’s capacity; the cluster bench (below) measures that. The PR job’s numbers (a fresh release build on a shared runner) are for comparing runs of the same job; the release image’s runs measure the binary that ships. The pass rate (99 %) gates; latency gates nothing.

Upgrade path

Every Calimero Cloud cluster on a release must be able to roll forward to every later core release without losing data. crates/node/tests/fault/upgrade_path.py proves it for the release pinned in .github/upgrade-from, the latest published one, against this tree:

1. The old cluster. Three merod run --dev nodes and a job node start on the release’s own merod, downloaded from its GitHub release and checked against its SHA256SUMS (scripts/fetch-release.sh). The nodes attest a mero-tee release of their own (--tee-release), the operator decides releases (--release-approvals operator, --operator-key), and packages come from a registry on loopback.
2. Seed. With the release’s own meroctl, over the public API: accounts with passkeys, sessions and API keys (a read-only one scoped to an app); an organization with members, a team, an invite not yet redeemed and an app team; the release’s signed kv package (kv-store-test-fixture.mpk) installed and an app of it with a history of writes, overwrites, removals and a note; plain blobs, one referenced from the app’s state; the compute fixture, built from the release’s tag and signed with its mero-sign (scripts/build-release-fixtures.sh), with a sealed file of three segments and a compute run finished on the job node.
3. Roll forward, as mdma’s rollout does it (rollout.py): the operator approves the new release (POST /ops/releases), three nodes on this tree’s merod join one at a time (POST /ops/nodes, each a system-group learner), every app moves onto them (POST /ops/apps/{app}/replicas, until GET /ops/placement shows it), and the old nodes are removed one at a time while the system group has a leader and its voters (DELETE /ops/nodes/{id}). Every decision is signed with the operator key. Once the first app has moved, apps sit on both releases and everything is read back (mixed).
4. Verify, extend, restart. With this tree’s meroctl, every value is read back on the new nodes, the saved sessions included (issued by a node that is gone, refreshed by the client); then new data is written: an account, a key, the old invite redeemed, a team, the app upgraded to this tree’s kv (signed as kv 2.0.0, its history kept), a blob, a new sealed file, and compute runs on a new job node over the old sealed files and the new one. One new node is restarted from its data directory and everything is read again.

What is written and checked lives in cases, upgrade_cases/NN_*.py, each a Case with seed, mixed, verify and extend steps (the module’s docstring says when each runs); a new kind of stored data gets a case. Run it locally, as CI does, with scripts/upgrade-path.sh (it builds this tree’s merod, meroctl and kv, fetches the release and builds its fixture), or with the binaries in hand:

scripts/fetch-release.sh 1.0.0-rc.15 target/upgrade-from
scripts/build-release-fixtures.sh 1.0.0-rc.15 target/upgrade-from
python3 -I crates/node/tests/fault/upgrade_path.py --from-dir target/upgrade-from \
  --to-merod target/release/merod --to-meroctl target/release/meroctl \
  --kv-next target/upgrade-path-fixtures/kv-next.mpk --only kv-package,blobs

The rule. A change that turns the upgrade path red has broken every existing cluster. It merges only with one of:

• a reader or migration, so the data the old release wrote keeps working (the usual answer: the new node reads both formats, as calimero-log reads the old log.redb);
• a declared hard cutover: the PR replaces the version in .github/upgrade-from with hard-cutover: <PR> <why>, marks it in CHANGELOG.md (“no interop with earlier nodes …; existing clusters are recreated”), and mdma adds it to mdma_common/release_compat.py’s CUTOVERS, whose epochs refuse to roll a cluster across it. The job reports the cutover instead of running until the next release puts a version back.

Never silently: #4884 (sealed files to segments) shipped “one-way” and forced a wipe of every cluster holding sealed files; run from 1.0.0-rc.14, this test fails on exactly that (no such file in this app for the old sealed files on the new nodes).

Keeping the pin current. .github/upgrade-from names the latest published release: once a release’s assets are up, the pin moves to it, at the latest in the next release PR (CONTRIBUTING.md, “Cutting a release”); the job warns when a newer version is tagged. A pinned tag that does not exist yet is reported and not run; a tag whose release lacks an artifact fails the job.

Deep cases: every functionality

Beside the representative data set (upgrade_cases/10_representative.py), the cases upgrade_cases/20_*.py and on take each user-visible functionality in turn: accounts and passkeys, sessions, activity, the recovery code, API and upgrade keys; organizations (roles, capabilities, invites, ownership, deletion, the recovery key); teams and app teams; packages, publisher bindings, app upgrades with a state migration and organization upgrade runs; app instances (history, access kinds, xcall, search, presence, transfer and move, usage); sealed files and upload sessions; governance and the operator’s state. Each is a DeepCase (crates/node/tests/fault/upgrade_deep.py), whose steps the harness runs:

Step	Runs	What it does
populate(w)	once, on the old release	seeds through the public API and answers a JSON-able record of what it expects
check(w, s)	right after seeding, in the mixed cluster, on the new nodes, after the restart	reads everything back and judges access both ways: what was allowed is allowed, what was refused (a removed member, a closed team, a revoked key) is refused with the same code
during_mix(w, s)	in the mixed cluster	check by default; xcall also relays between two apps that may sit on different releases
exercise(w, s)	on the new nodes	writes something new of the kind (and updates s for the checks after it)

check runs right after seeding as well, so a check that fails after the roll failed because of the upgrade, not because it was wrong. A read compares as a subset (a new release may add a field, never drop or change one), without the fields that move on their own (VOLATILE: sign-in counts, replicas, write times). A case that fails to seed is reported by each later step and the run goes on, so one run lists every breakage. The cases deploy the release’s own kv component and packages built from the release’s tag with its mero-sign (scripts/build-upgrade-deep-fixtures.sh, after build-release-fixtures.sh).

New functionality ships with a case. A PR that adds a user-visible functionality adds a DeepCase for it, with since set to the release it will ship in. Rolled from an older release, such a case is seeded on the rolled cluster instead (in extend) and checked from then on, so it runs from the PR that adds it, and becomes a full upgrade case once the pinned release has the functionality. To add one:

1. Put a @case class deriving DeepCase in the upgrade_cases/ file of its area (or a new NN_area.py), with a unique name, its area, since, and one-line seeds and verifies for the matrix.

2. populate signs its own people up (person(w, name), names no other case uses) and keeps every id and expected value in the record; check reads them back and records access decisions with decisions({...}), compared with same_decisions; exercise writes on the new release.

3. Paste the output of python3 -I crates/node/tests/fault/upgrade_deep.py --matrix between the matrix markers below; the job fails while they differ (--check-matrix).

4. Run it: scripts/upgrade-path.sh, or with the fixtures in hand:

   scripts/build-upgrade-deep-fixtures.sh target/upgrade-from   # after build-release-fixtures.sh
   python3 -I crates/node/tests/fault/upgrade_path.py --from-dir target/upgrade-from \
     --to-merod target/release/merod --to-meroctl target/release/meroctl \
     --kv-next target/upgrade-path-fixtures/kv-next.mpk --only <name>

What cannot be tested this way yet is listed in NOT_COVERED (upgrade_deep.py) with the reason, and appears in the matrix as not covered; a case is never skipped.

Coverage matrix

Area	Case	Seeded on the old release	Checked in the mixed cluster, after the roll and after a restart	Covered
Representative	accounts-and-keys	Accounts with passkeys, meroctl sessions and API keys.	read back, extended, read again	yes
Representative	organization	An organization with members, a team with a member, an open invite and app teams.	read back, extended, read again	yes
Representative	kv-package	The release’s signed kv package, an app instance of it in the org with history, an app team, a read-only key; upgraded on the new release to a build of this tree.	read back, extended, read again	yes
Representative	blobs	Plain blobs of the kv app, one referenced from its state.	read back, extended, read again	yes
Representative	files-and-compute	Sealed files (one of three segments) of a compute app from a signed package, and a compute run finished on the old release; on the new one a run reads the old files.	read back, extended, read again	yes
Accounts	account-passkeys	an account with three passkeys: one labelled, one removed	the passkey list and labels; every kept passkey signs in on each node, the removed one is refused; a label set on the new release	yes (since 1.0.0-rc.15)
Accounts	account-sessions	sessions opened on old nodes, one revoked; an account signed out everywhere	a refresh token an old node issued refreshes on every node; its session token is accepted while that node is in the cluster and refused once it is removed; the revoked session’s and the signed-out account’s refresh tokens are refused; a session revoked on the new release ends	yes (since 1.0.0-rc.15)
Accounts	account-activity	sign-ins and passkey approvals (an API key made, a confirm method called)	every seeded activity entry is still listed; an approval and a sign-in on the new release add theirs	yes (since 1.0.0-rc.15)
Accounts	account-recovery-code	an account recovery code made on the old release	recoveryCode stays set; the code begins a recovery on every node and a wrong code is refused; removed on the new release, it is refused	yes (since 1.0.0-rc.15)
Keys	api-keys	API keys: read-only for one app, one method of one app, with a TTL, revoked; an alias	each key allowed and refused exactly as before (reads, writes, its method, its app, another app); the revoked key stays refused; the key list and the alias; a key made and one revoked on the new release	yes (since 1.0.0-rc.15)
Keys	upgrade-keys	an upgrade key (an API key that may only move one app to releases of its package)	the key still cannot call or read the app; on the new release it upgrades the app to the next release, and is refused another app	yes (since 1.0.0-rc.15)
Organizations	org-members-roles	an organization with metadata, an admin, members (one granted CREATE_TEAM), a guest, member metadata, a removed member, a member who left, an organization app	the organization as the owner, a member and a guest read it; the members’ security view; who reaches the app (members yes; removed, departed and outsider no; the guest as before) and who may read security and write metadata; CREATE_TEAM makes a team on the new release, a plain member is refused; metadata written there	yes (since 1.0.0-rc.15)
Organizations	org-invites	invites: a multi-use one used once, an unused single-use one, a used-up one, a revoked one, a guest invite into one team (app-only access)	the invite list; the revoked and used-up invites are refused; the guest reaches its team’s app only; the unused and multi-use invites made on the old release redeem on the new one	yes (since 1.0.0-rc.15)
Organizations	org-ownership	an organization whose owner handed it to a member	the roles after the hand-over; the former owner is refused owner-only actions; the new owner changes a role on the new release	yes (since 1.0.0-rc.15)
Organizations	org-deleted	an organization with an app and a team, deleted	the organization and its app stay gone; the owner makes a new one on the new release	yes (since 1.0.0-rc.15)
Organizations	org-recovery-key	an organization with a recovery key, and a recovery a member started with it on the old release	the recovery key and the pending recovery; finishing early and a recovery signed with another key are refused; the owner keeps the organization; the owner cancels it on the new release and the member starts it again	yes (since 1.0.0-rc.15)
Teams	teams	an open team; a closed team with a lead, a member, a read-only member and a removed member; a private child team; an app on each	each team as before; who reads and writes each team’s app (the closed team excludes non-members, read-only reads only, the removed member is refused, the open team admits the organization, the child only its members); a member added on the new release writes	yes (since 1.0.0-rc.15)
Teams	app-teams	two packages’ instances in an organization given app teams (ScopeApps): one closed keeping one member, one open	the app-team list; the kept member and the open team’s members reach their apps, the member who lost the closed one is refused it; the closed one opened again on the new release admits them	yes (since 1.0.0-rc.15)
Packages	packages	two signed releases installed from the registry and the second uninstalled again; an app on the first with data; an app from a dev bundle (raw component)	the package and its versions; both apps answer from the new nodes; on the new release the uninstalled version installs again and the app upgrades to it keeping its data	yes (since 1.0.0-rc.15)
Packages	publisher-binding	a package signing key bound to an organization (its publisher)	the binding and its generation as anyone reads them; a second key bound on the new release	yes (since 1.0.0-rc.15)
Upgrades	app-upgrade-migration	notes 1.0.0 and 2.0.0 (state version 1 to 2) installed; an app upgraded to 2.0.0, one left on 1.0.0, two upgraded by an organization-wide upgrade run	each app’s release and notes; the run’s status as a member reads it; the app left on 1.0.0 upgrades on the new release keeping its note and gains 2.0.0’s methods; a new run there	yes (since 1.0.0-rc.15)
Apps	app-state-history	an app with 40 writes (sets, overwrites, removals) and app metadata	every entry through each node, the length, the full event history from index 0 (the event stream’s replay), the app record, its storage figures and ABI; a write on the new release extends the history after the old events	yes (since 1.0.0-rc.15)
Apps	app-access-kinds	an organization’s app of each access kind: owner, organization, listed accounts, public, team	who reads and writes each (the owner, a member, a listed account, an outsider, no credential), allowed and refused as before; access changed on the new release	yes (since 1.0.0-rc.15)
Apps	xcall	two apps of an organization; writes relayed from one to the other (xcall), one refused by the callee, one dead-lettered	the relayed writes, what onXcallFailed recorded, the xcall list and its dead letters on every replica; a relay lands in the mixed cluster (across releases while the two apps sit on different ones) and on the new release	yes (since 1.0.0-rc.15)
Apps	full-text-search	notes in an app’s full-text index (title and body), one dropped	each search answers the same notes through every node (each rebuilds its index); a note added on the new release is found	yes (since 1.0.0-rc.15)
Apps	presence	presence published in an app (ephemeral: it expires by design, so nothing is carried)	presence publishes and reads back through every node, in the mixed cluster and after	yes (since 1.0.0-rc.15)
Apps	app-transfer-move	an app handed to another account (transfer); an app moved to another organization with its members	the owner and organization of each, their data; the former owner refused; the moved app’s members reach it in its new organization; handed back on the new release	yes (since 1.0.0-rc.15)
Metering	usage-and-storage	an organization’s app with writes; the account’s and the organization’s usage	the app’s per-day usage history (earlier days unchanged, today’s figures only grow), the account’s usage counts and the organization’s usage	yes (since 1.0.0-rc.15)
Blobs and files	sealed-files-sessions	files an app wrote itself, an upload session begun with one of its parts sent, a plain blob deleted (the representative set uploads files of several segments)	each app-written file’s size and SHA-256 as the app reads it; the half-done session expects its next part while its leader (an old node) leads and answers UPLOAD_UNKNOWN once the app moved (documented: the client begins again); a session begun on the new release completes with the right digest; the deleted blob stays gone	yes (since 1.0.0-rc.15)
Governance	outbound-approvals	an organization’s outbound approval for a package and an account’s own	each approval as its holder reads it (or the same refusal, where governance gives it to the operator); every release the cluster approved stays approved after the roll	yes (since 1.0.0-rc.15)
Ops	ops-state	account quotas changed, a per-organization quota override, a compute pool	quotas, overrides, pools, the cluster’s sign-up invites and policy as before; a quota changed and a backup written on each new node	yes (since 1.0.0-rc.15)
Accounts	Account recovery started and finished with a new passkey	-	-	no: the start takes a WebAuthn registration made outside a session, which neither meroctl nor the suite’s standard-library Python can make (P-256); the cluster tests cover it on one build. The recovery code (account-recovery-code) is covered
Accounts	Sign-up through a cluster sign-up invite, and issuing one	-	-	no: meroctl’s login takes no invite and has no command to issue one (an operator-signed request); the policy and the invite list are read back (ops-state)
Accounts	Consent-page approvals (/consent/requests)	-	-	no: a consent request is approved in a browser on the node’s consent page; the cluster test consent_page.rs covers it on one build
Apps	Timers (#[app::every]) firing across the roll	-	-	no: the fixture’s timer is hourly on the nodes’ real clock, which a process cluster cannot move; the cluster tests move the app clock
Packages	Served frontends and served origins (custom domains)	-	-	no: the suite’s packages carry no frontend, and the node routes a frontend by Host under the apps domain, which plain HTTP on 127.0.0.1 does not reach; the cluster test app_frontends_are_served_and_their_sessions_are_bound covers serving
Governance	Owner-held release approvals and release deadlines	-	-	no: the rolled cluster gives release approvals to the operator (--release-approvals operator), as Calimero Cloud does; the operator’s approval of the new release is the rollout’s first step
Deployments	Deployer grants and deployments (/deployers, /deployments, /ops/deployers)	-	-	no: not seeded yet: a grant is an operator-signed or passkey-approved decision these cases do not make yet; the cluster tests (deployers.rs, deployments.rs) cover them on one build
Packages	Registered web origins of a package (/package-origins)	-	-	no: not seeded yet: an origin is added with its manager’s passkey approval and is read only by the served-frontend path above
Blobs and files	Revoking a published blob’s grants (/blobs/{id}/revoke)	-	-	no: not seeded yet: a grant is a URL the app’s share view mints, which these cases do not follow
Ops	Restoring a backup onto the new release	-	-	no: POST /ops/backup is exercised on the new nodes (ops-state); a restore is a recovery drill, not a rollout

Replaying a release’s data directory

The rollout moves state between nodes; it does not show that this tree applies an old release’s log the same way. crates/node/tests/upgrade_replay.rs does, in process and in seconds (it runs with the other calimero-node tests): each crates/node/tests/fixtures/upgrade/<release>/ holds the data directory of a one-node cluster as that release’s merod run --dev left it (accounts with two passkeys and a recovery code, scoped API keys, an organization with a removed member, a team and an invite, apps of three access kinds with history, a note and a blob) and expected.json, every answer that release gave to a fixed list of reads (GET /auth/me, the organization, its team and invites, the apps, their storage and ABI, views as members, a read-only key and an outsider, /ops/quotas, /sign-up). The data set stays under the log’s snapshot interval, so the directory holds no system snapshot (the test asserts it) and this tree rebuilds the whole system state by applying every entry the old release logged. The test opens each directory, sends every read again (the same status, a body holding every field of the old one), then writes.

A release’s fixture is written once, with that release’s binaries, and kept: the PR that moves .github/upgrade-from to a release adds its fixture too.

scripts/fetch-release.sh 1.0.0-rc.15 target/upgrade-from
python3 -I crates/node/tests/fault/upgrade_fixture.py --from-dir target/upgrade-from \
  --out crates/node/tests/fixtures/upgrade/1.0.0-rc.15
cargo test -p calimero-node --test upgrade_replay

In CI

.github/workflows/e2e.yml runs on every PR that touches code:

• job scenarios (K/4) (scenarios): cargo test -p calimero-node --test scenarios with SCENARIO_SHARD=K/4, every app-level and cluster scenario, each on its own in-process cluster; the runner builds each app’s component itself. A fuzzy_test runs 10 s, nightly 900 s;
• job process suite (merod processes) (process-suite): a release build of merod and meroctl, the kv component and two signed releases of it, then suite.py (nightly --nightly, which also builds the Rooms component). The soak’s latency and throughput (below) go in the job summary; the artifact process-suite holds the results, the soak report and failed tests’ node logs;
• job compute actions, real processes (compute-e2e): crates/node/tests/fault/compute_e2e.py starts three replicas and two merod run --job-node processes of a release build, installs a signed .mpk that carries the fixture’s app and compute components, and drives compute runs through a job node’s loss, a cancel, the caps and the collection of files no state references; it also checks that the job nodes were given only the compute component, and runs compute-redact end to end: an admin’s policy, a member’s 38 MB CSV of invented people, a job node killed mid-run, then the shared parts downloaded through a grant and compared with mero-compute’s output (logs are the artifact compute-e2e-logs on failure);
• job clients against merod (clients): mero-js’s test:cloud-v2 and calimero-client-py’s cluster tests against this tree’s merod. A change that breaks a client goes green with the client’s fix: a same-named branch there, or sdk-ref: <ref> (mero-js) / py-ref: <ref> (Python) in the PR body.

ci-checks.yml’s Upgrade path (from the pinned release) job (upgrade-path) runs the upgrade path through scripts/upgrade-path.sh on every PR that reaches merod’s, meroctl’s or the kv fixture’s dependency closure, the test or its scripts, and on master; the required Rust check waits on it. The job summary has the rollout’s steps and each case’s result; the artifact upgrade-path holds the results and, for a failed run, every node’s log and data.

ci-checks.yml’s Rust tests job runs each component app’s own cargo test (its tests/harness.rs and its app-level scenarios on TestApp), then clippy with -D warnings on the host and on wasm32-wasip2 and the app’s release wasm32-wasip2 build. The apps are found by their calimero-sdk dependency.

Nothing has to list a scenario for it to run: the runner discovers every file on disk, and one it must not run yet is listed, with its reason, in crates/node/tests/scenarios-skip.txt and reported as ignored.

Those test a fresh build. Two jobs test the merod that ships, the binary inside the Docker image (the suite starts processes, so it is copied out with docker create and docker cp):

• A PR’s own image (.github/workflows/release.yml, job E2E on the merod image, on PRs and the merge queue whenever the PR builds the image). Release Merod container builds the amd64 image as it does for master, loads it instead of pushing it, and hands it on as the artifact merod-image; Image E2E fixtures builds a release meroctl, kv-store’s WASM and two signed releases of it alongside the binaries. The job runs suite.py --smoke against the image’s merod: kv-failover, frozen-leader, orgs-and-keys, packages and a 30 s kv-soak. Master pushes its image instead.
• The latest release’s image, nightly (e2e.yml, jobs Latest release image (release-image) and process suite on release V (release-process-suite); on demand with the release_image input: latest, a tag of ghcr.io/<owner>/merod such as release-v2-edge, or an image reference). It pulls ghcr.io/calimero-network/merod:<version> for the newest published release (a prerelease included) and runs the process suite of the commit the image was built from (its org.opencontainers.image.revision), with that commit’s meroctl and fixtures, so a test added after the release does not fail it: every test but the upgrade, with a 900 s soak. A release whose commit has no crates/node/tests/fault/suite.py is reported in the summary and not run. On the release branch the nightly tests release-v2-edge instead, the image of the branch’s latest merge (Release branch). The same nightly runs every scenario at that commit, in four shards (jobs scenario fixtures (release V) (release-fixtures) and scenarios on release V (K/4) (release-scenarios)), with that commit’s runner and fixtures. The runner is in process, so what it tests is the release’s node source, the source its merod was built from; the binary itself is the process suite’s. A release whose commit has no crates/node/tests/scenarios.rs is reported in the summary and not run.

To run the smoke set against a binary taken out of an image locally (with the fixtures built as in Process suite):

docker create --name m ghcr.io/calimero-network/merod:release-v2-edge && docker cp m:/usr/local/bin/merod /tmp/merod && docker rm m
python3 -I crates/node/tests/fault/suite.py --smoke --merod /tmp/merod

Flamegraphs

Nightly (or with the flamegraph input), job merod flamegraph (kv-store soak) builds merod with the profiling profile (optimized, with symbols) and -C force-frame-pointers=yes, runs the process suite’s kv-soak for five minutes, and records node 1 with perf record -g for most of it (scripts/soak-flamegraph.sh; the runner sets kernel.perf_event_paranoid=-1). inferno renders the samples; the artifact merod-flamegraph holds merod-flamegraph.svg, the folded stacks merod.folded (compare two nights with inferno-diff-folded old.folded new.folded | inferno-flamegraph > diff.svg) and the profiled run’s soak report. Locally, on Linux:

cargo install inferno --version 0.12.8 --locked
RUSTFLAGS="-C force-frame-pointers=yes" cargo build -p merod -p meroctl --profile profiling
sudo sysctl -w kernel.perf_event_paranoid=-1
MEROD_BINARY=target/profiling/merod MEROCTL_BINARY=target/profiling/meroctl SOAK_SECONDS=120 \
  scripts/soak-flamegraph.sh /tmp/flame

PERF=/usr/lib/linux-tools/<version>/perf points it at a perf whose wrapper does not match the running kernel; PROFILE_DELAY, PROFILE_SECONDS and PROFILE_HZ set when and how it samples, and a second argument names another node.

Release branch

Calimero 1.0 was developed on release-v2-parity, which has since been merged into master (1.0.0-rc.1). CI still treats that branch as a release branch, with its own image tags:

When	What runs	What publishes
PR into the branch	Every check a PR into master runs (no check’s pull_request trigger leaves the branch out; abi-guard.yml and platforms.yml name it)	Nothing; Release Merod container builds the PR’s image and E2E on the merod image tests it
Merge (push) into the branch	ci-checks.yml, e2e.yml, release.yml, benchmarks.yml, docs-ci.yml, crypto-vectors.yml, workflow-lint.yml, codeql.yml; ci-checks and E2E runs are never cancelled, and the ci-checks, wasm-apps and storage-cost caches save, as on master	The multi-arch merod image as ghcr.io/calimero-network/merod:release-v2-edge and :release-v2-<short sha>, labelled with the commit (org.opencontainers.image.revision); the release binaries as the run’s artifacts-<target> artifacts, kept 14 days; the criterion, search and cluster benchmark reports as the base the branch’s PRs compare against
Nightly	e2e.yml with nightly: true (the 15-minute soak, the process suite on release-v2-edge, the flamegraph), fuzz.yml (30 minutes per target) and platforms.yml with nightly: true (macOS; a failure opens “Nightly platform build is failing on release-v2-parity”)	Nothing

The branch never publishes a version: no edge, latest, prerelease or version tag, no GitHub Release, crates, App Bundle, Homebrew tap update or SDK bump PRs. release.yml gates each of those on master. The clients job tests the clients’ own release-v2-parity branches when nothing is paired (sdk-ref:, py-ref: or a same-named branch), since their master has no cluster tests.

GitHub runs schedule only from the default branch, so the branch’s nightly runs only when dispatched. To run one by hand:

gh workflow run e2e.yml --ref release-v2-parity -f nightly=true
gh workflow run e2e.yml --ref release-v2-parity -f release_image=release-v2-edge   # the 30 s soak, plus the process suite on that image

docs-site.yml stays master’s: it deploys the public documentation site to GitHub Pages, which has one site per repository, so the branch’s docs are built and link-checked (docs-ci.yml) but not deployed.

What scenario tests cannot catch

Every node in a scenario run is the same build on fresh state, and so is every node in the process suite (its rolling upgrade moves to a copy of the same binary unless --upgrade-merod names another), so neither tests data written by an older version. The upgrade path does, for the data its cases write, from the last release only. A change to an on-disk or wire format also needs its own migration test, as calimero-log‘s migration_tests.rs does for the log.redb store, and a case in the upgrade path when it stores a kind of data none of its cases writes. What else tests across releases, and what a change that breaks one declares, is in Compatibility: ci-checks’ Compatibility job (no route, field, WIT or ABI version dropped) and Apps of the previous release (the last release’s kv bundle on this tree’s node).

Nor do they run two releases in one cluster, so they cannot catch an entry an older node skips. A new command or field ships with a feature gate (The cluster version): add a Feature with the release it ships in, name it in SystemCmd::features, and give it a case in uses in crates/node/src/system/cluster_version.rs’s tests. There, a_feature_is_never_proposed_while_an_older_node_holds_the_log replays every feature through the proposal gate on a cluster holding an older node (modelled as a release that skips what it cannot decode, as SystemMachine::apply does) and checks that nothing is proposed and every replica keeps the same state, then that each feature works once the older node leaves and the leader raises the cluster version.

Proving a fix

1. Reproduce first, at the layer a user hits it. A cluster test for API behaviour, a unit test for a rule, a scenario file for an app’s or a flow’s steps across accounts, a process-suite test for a multi-process flow (a crash, a freeze, an upgrade). Watch it fail for the reason the bug describes.

2. Fix it and rerun the same test. Keep it as the regression test; the before and after output goes in the PR description.

3. Reason about old data. For anything touching the store, snapshots, the sealed wire format or signed formats, work through existing data and mixed-version clusters explicitly.

Benchmarks and cost gates

Storage cost gates, scaling guards and criterion benches are described in docs/benchmarking.md. .github/workflows/benchmarks.yml runs them on every push to master or the release branch and on a PR labelled run-benchmarks: the criterion benches per crate (calimero-log’s append and commit, calimero-sealed’s request round trip, the storage and runtime benches), compared with critcmp against the base commit’s push run on the PR’s base branch, plus two reports that are not criterion: tools/search-bench (the search engine) and the cluster throughput bench, whose JSON the comparison also diffs.

Cluster throughput

crates/node/benches/cluster_throughput.rs drives a 3-node cluster in one process (the harness of tests/cluster.rs, in tests/harness/cluster.rs, with merod’s Raft timing) through the sealed client, calling the kv fixture’s set on the app’s leader:

cargo bench -p calimero-node --bench cluster_throughput -- --secs 15 --out cluster.json

Two runs on a 4-vCPU Linux container (bench profile with LTO off; the machine was shared with other work, so treat rows as ±25%):

Scenario	Calls/s	Per app	p50	p99
(a) 1 app, 1 in flight	148 / 185		6.3 / 5.2 ms	15 / 8 ms
(b) 1 app, 64 in flight	205 / 250		285 / 253 ms	519 / 317 ms
(c) 2 apps, 64 each, one leader node	322 / 349	160 + 162 / 176 + 173	348 / 340 ms	780 / 495 ms
(c’) 2 apps, 64 each, two leader nodes	264 / 298	119 + 146 / 139 + 159	436 / 423 ms	2963 / 3671 ms

One synced RocksDB write (rocksdb_write/synced in calimero-log’s bench, and timed again by this bench) took 0.16 to 0.19 ms on this disk; unsynced, 2 µs.

Appends without a leadership round. Group::propose used to confirm leadership with a majority (ensure_linearizable) before every build; it now does so only for a proposal that answers without appending, and concurrent sys_fresh callers share one read-index round. Before and after on the same machine, two runs each, --secs 10 (other work shared the machine, so ±25% still holds):

Scenario	Calls/s before → after	p50 before → after	p99 before → after
(a) 1 app, 1 in flight	219 / 191 → 205 / 233	4.5 / 5.1 → 4.8 / 4.2 ms	6.6 / 7.2 → 7.9 / 6.3 ms
(b) 1 app, 64 in flight	238 / 233 → 293 / 298	270 / 273 → 216 / 212 ms	337 / 301 → 259 / 247 ms
(c) 2 apps, 64 each, one leader node	368 / 415 → 497 / 465	343 / 305 → 252 / 277 ms	453 / 382 → 327 / 365 ms
(c’) 2 apps, 64 each, two leader nodes	452 / 460 → 481 / 471	286 / 274 → 259 / 259 ms	2273 / 2322 → 368 / 362 ms

calimero-log‘s bench on a 3-node group: one proposal in flight 689 / 633 → 573 / 528 µs, 64 in flight 36.8 / 38.1 → 33.4 / 31.8 ms (1 node: unchanged, it had no round to save). A single caller (a) gains nothing measurable; the queue behind the propose lock (b, c) moves about a fifth faster, and the seconds-long p99 with two leader nodes (c’) is gone.

• The limit is per app, not per node. A group proposes one call at a time (propose_lock in Group::propose is held while the leader runs the WASM and until the entry commits), so 64 calls in flight on one app only queue: (b) is 1.35x (a), and its p50 is about 50 times (a)’s. A second app on the same leader node adds its own stream: (c) is 1.4 to 1.6x (b) (2x in an earlier run), and the two apps share it evenly; a per-node limit would hold (c) at about 1x. Two leader nodes did no better than one here, and their p99 was seconds: 4 cores ran all three nodes and 128 callers.
• The disk is not the limit. A call needs the leader’s synced append and a follower’s (in parallel), so about 2 syncs: 1 ÷ (2 × 0.17 ms) is about 3,000 calls/s per app, against the 150 to 300 measured. A call’s 4 to 7 ms is the sealed HTTP round trip, the fresh system read, the WASM run and the Raft round trip of its append, the last two inside the propose lock. On a disk whose sync takes 10 to 25 ms (macOS, where Rust’s sync_data is F_FULLFSYNC, which redb ran on every commit), the same formula gives 20 to 50 writes/s per app, the range of the ~38/s a macOS build on redb reported.

Fuzzing

fuzz/ fuzzes what a node reads from untrusted bytes (sealed_request, passkey_verify, session_token, quote_parse, bundle_open). fuzz.yml runs a changed target for a minute on a PR and every target for 30 minutes nightly; scripts/fuzz.sh <target> [seconds] runs one locally.

Where to go next

Development workflowBuild, run local nodes, and the checks before a PR.

Testing appsTestApp and scenarios from the app author's side.