Testing strategy

Calimero Core is tested in layers, each trading scope for speed. Fast in-process unit tests cover application and CRDT logic; a deterministic simulator covers the sync protocol without WASM or a network; merobox workflows cover the real multi-node system end to end. This page maps the layers and gives the exact commands CI enforces, so you can reproduce the gate before opening a PR.

Use both ends of the ladder. A green cargo test does not prove a networked flow works, and a green E2E does not localize a logic bug.

Layer	Where	What it proves	Speed
App unit tests (TestHost)	crates/sdk/src/testing.rs	Your app’s call / view / event / migrate logic, in isolation	Milliseconds
Storage convergence	crates/storage/src/testing.rs	A Mergeable / #[app::state] type converges under concurrent, reordered ops	Sub-second
Sync simulator	crates/node/tests/sync_sim/	The sync protocol converges under loss, latency, reordering, and partitions	Fast, deterministic
Fuzzing	fuzz/	Parsers of inbound bytes survive hostile input	Coverage-guided, time-boxed
Integration / e2e (merobox)	apps/*/workflows/, workflows/	The real multi-node system: install, call, sync, govern, migrate	Slow (Docker)

App unit tests with TestHost

TestHost lets you exercise an #[app::state] type as ordinary Rust — no wasm32 build, no node, no containers. State lives in an in-memory mock store; events and logs are captured; the executor / context identity is configurable. You drive mutations with call, reads with view, and inspect events() / logs(). Migrations are supported too: TestHost::migrate, plus assert_migrate_converges and assert_absorb_replay_converges for the determinism properties the migration model rests on.

let mut app = TestHost::new(|| MyApp::init());
app.call(|s| s.increment("a".into())).unwrap();
let value = app.view(|s| s.get("a".into())).unwrap();
assert_eq!(value, 1);
assert_eq!(app.events().len(), 1);

What TestHost deliberately does not model

The harness covers normal call / view / event / log / identity / migrate flows. It does not model:

• Cross-context calls (env::xcall), networked blob announce / fetch, and env::ed25519_verify — these have no in-process equivalent and panic if invoked. Cover them with merobox workflows.
• Event-handler dispatch. TestHost captures the events your methods emit, but it does not run the node’s handler loop, so registered handlers do not fire automatically — you drive methods directly through call.

It is also one-live-harness-per-thread: construct a fresh TestHost at the top of each #[test] (it resets all mock state on construction), and set a non-owner identity explicitly when asserting a rejection path — the default identity satisfies owner checks and would silently pass.

This is the same harness app developers use; the app-author-facing guide is at /build/testing/.

Storage convergence (property tests)

crates/storage/src/testing.rs provides converge / converge_app: assert in a single #[test] that a state type converges under concurrent, reordered operations. It drives N in-memory replicas (each with its own store and executor identity), applies the registered ops on every replica in a per-replica shuffled order, gossips the resulting StorageDeltas in shuffled order, and asserts every replica lands on the same Merkle root hash.

converge_app(TeamMetricsApp::init)
    .replicas(3)
    .ops(|s| { let _ = s.record_win("liverpool".into()); })
    .assert_all_replicas_equal();

Note

This proves your app state converges end-to-end, which is usually not your hand-written Mergeable::merge — collection fields converge via the storage layer’s per-child-entity merge during delta apply, and inline scalars via the root entity’s HLC last-writer-wins. Shared / Authored / User / Frozen storage need the node’s signing identity (delta apply verifies signatures), which this bare harness does not provide — test those with merobox. The run takes a process-global lock, so it self-serializes and does not need #[serial].

The deterministic sync simulator

crates/node/tests/sync_sim/ is a discrete-event framework for the sync protocol. It reuses the real calimero-storage Index / Interface::apply_action and Merkle tree, the real hash computation, and the real select_protocol() — so hash propagation and protocol selection behave identically to production — while replacing WASM execution, network I/O, time, and concurrency with a NetworkRouter (with fault injection), a discrete SimClock, seeded RNG, and sequential event processing. Seeds make every failure reproducible.

let mut rt = SimRuntime::with_seed(42);

Read crates/node/tests/sync_sim/AGENT_GUIDE.md before touching the framework. The framework itself lives under sync_sim/ (change with review); add new protocol-behavior tests under sync_scenarios/, and compliance tests under sync_compliance/.

The neighbouring binaries in crates/node/tests/ (sync_scenarios, network_simulation, dag_*) exercise sync, DAG and readiness convergence across several nodes in one process, without Docker. They are the fastest way to reproduce a sync or ordering bug.

Fuzzing

fuzz/ holds coverage-guided fuzz targets over inbound bytes: bundles, wire frames, signed ops, CRDT sync payloads and provider records. Run one with scripts/fuzz.sh <target> [seconds]; the targets and seeds are described in fuzz/README.md.

Integration and e2e (merobox)

The widest layer runs real merod nodes in Docker via merobox, driven by declarative YAML workflows. Each scenario file describes a node topology and a sequence of install / call / assert steps. These are the tests that exercise everything TestHost and the simulator leave out — real networking, cross-context calls, governance across peers, late joiners, partitions, and multi-node migrations. They run in CI under the e2e-rust-apps.yml, app-migration-e2e.yml, and sync-regression.yml workflows, building a local merod:local image plus the WASM apps under apps/. A scenario joins the e2e-rust-apps.yml suite by an entry in .github/e2e-scenarios.yml; CI packs those entries into groups that each share one runner, and cleans every container, network, volume and data directory between the scenarios of a group.

The scenarios live in:

• apps/scaffolding-e2e/workflows/: group membership, key delivery, kick and rejoin, subgroups, leave, late joiners, sync resilience under partition, mesh soak.
• apps/blobs/workflows/: blob announce, fetch and sizes across nodes.
• workflows/sync-tests/, workflows/app-migration/ and workflows/fuzzy-tests/: sync regression, v1 to v2 migrations, and fuzzy load.

Run one locally with merobox bootstrap run <scenario.yml>, then merobox stop --all. The YAML format is described in apps/AGENTS.md.

CI builds the nodes from the PR’s own code (.github/actions/build-local-merod produces merod:local), so a green E2E means that code passed. A failed group re-runs all its scenarios. Its step summary names the scenario that failed, and that scenario’s node logs are under docker-logs/<scenario>/ in the group’s logs-* artifact.

Reproducing a CI flake locally

A flake that only shows up in CI is usually timing, so reproduce it with CI’s build and loop it.

1. Build merod the way CI does: CARGO_PROFILE_RELEASE_LTO=false CARGO_PROFILE_RELEASE_CODEGEN_UNITS=256 cargo build -p merod --release. A debug build runs several times slower, and that alone can hide a race: one ordering bug in account-pairing-application-scope never reproduced on a debug build in 17 runs, and failed 2 runs in 11 on this build.
2. Package it as merod:local with .github/workflows/deps/prebuilt.Dockerfile, as .github/actions/build-local-merod does. For a scenario on the http registry image, serve the bundles with url=$(./scripts/e2e-fixture-registry.sh) and stamp the image with scripts/set-registry-mode.sh merod:local http "$url".
3. Run the scenario from its app directory as scripts/e2e-run-scenario-group.sh does: merobox bootstrap run <scenario.yml> --image merod:local --e2e-mode. Embedded-auth scenarios need MERO_AUTH_ADMIN_USER and MERO_AUTH_ADMIN_PASSWORD set, matching the scenario’s login steps. The node logs land under data/container-logs/ in the app directory, and a stop_all_nodes scenario removes its containers, so copy that directory before the next run.
4. Loop it and keep every run’s logs, so a failing run can be compared against a passing one. If it still will not fail, add CPU load (one busy process per core): a gossip-mesh race in the same scenario needed it.

What E2E cannot catch

Every node in a merobox run is the same build against fresh state. It validates a uniform, new-from-scratch network, and by construction never tests mixed old and new nodes or data persisted by an older version. A green E2E says nothing about backward compatibility or rolling upgrades. A format, derivation, schema or borsh-layout change that must survive existing data or staggered upgrades needs its own migration path and its own test.

Reading logs

merod logs through tracing to stdout. Target a subsystem with RUST_LOG to see the flow you are debugging:

RUST_LOG=debug merod --node node1 run
RUST_LOG=calimero_node::sync=trace,calimero_context=debug merod --node node1 run

• In merobox each node is a container, so read its output with docker logs <container>. To raise verbosity in a scenario, set RUST_LOG on the node in its YAML.
• Scenarios assert on log signals by grepping for an expected line; the sync-resilience-* scenarios, for example, count occurrences of a sync marker.
• Panics are logged by a structured hook (message, thread, location, backtrace), so grep for the panic message rather than for panicked.

Proving a fix

1. Reproduce first, end to end. Write the failing case at the layer a user hits it: a sync_sim or integration test for a logic or ordering bug, a merobox scenario for a networked flow. Watch it fail, and capture the log window around the failure with the specific line or marker that proves the bug.

2. Apply the fix and rerun the same test. Keep it as the regression test. The captured before and after is the evidence the PR description shows.

3. Reason about old data. For anything touching on-disk formats or wire encoding, work through the old-data and mixed-version cases explicitly, because E2E will not.

What runs in CI

Workflows are path-filtered:

Workflow	Triggers on	Runs
e2e-rust-apps.yml	any crates/** and app dirs	The main merobox suite (scaffolding-e2e, xcall, blobs, kv-store): the scenarios in .github/e2e-scenarios.yml, several per runner (scripts/e2e-scenario-groups.py)
sync-regression.yml	crates/node/src/sync/**, storage sync paths	workflows/sync-tests/
app-migration-e2e.yml	migration paths	v1 to v2 app-migration scenarios
sdk-e2e.yml	SDK paths	SDK end to end
fuzzy-load-test.yml	manual, load paths	Fuzzy load
platforms.yml	nightly; PRs reaching Windows-only code (scripts/needs-windows-ci.py); label ci:platforms	Windows tests and e2e, macOS and Windows release builds

Documentation never starts a code workflow. Every paths: filter ends with !**/*.md / !**/*.mdx, re-including only the Markdown some code reads (an include_str! target, a manifest’s guide or readme); scripts/tests/ci-paths-test.py fails a filter that lets docs back in. ci-checks.yml and release.yml carry required checks, so they run on every PR and skip work instead. scripts/ci-scope.py follows each job’s packages through their dependency closure and skips a job the change cannot reach: the storage cost gate, WASM size gate, cargo-mero e2e and app builds run only when their inputs move, and a docs-only PR builds nothing.

The pre-PR gate

These are the exact checks CI enforces in .github/workflows/ci-checks.yml. CI runs them as parallel jobs behind the one required Rust check: lint, tests, feature-gated tests, the app build, and supply-chain checks. Run them all locally with ./scripts/check-like-ci.py, which reads that workflow instead of keeping its own copy; --list shows what it will run. The steps below are the same commands, grouped by job.

1. Check formatting and lint with the no-warnings gate. The --features calimero-storage/testing flag pulls the storage testing harness into the same single pass so it is linted too.

   cargo fmt --check
   cargo clippy --workspace --all-targets --features calimero-storage/testing -- -D warnings

2. Run the tests. CI runs them under cargo-nextest, which runs every test binary in parallel, plus the doctests nextest does not cover. Some networking tests bind real interfaces through a TEST_HOSTS env var; set it locally if host-dependent tests fail. Tests that execute an app’s WASM build it themselves when it is missing or stale, so no app build is needed first.

   TEST_HOSTS=127.0.0.1,0.0.0.0 cargo nextest run --workspace
   cargo test --workspace --doc

3. Build every app. CI builds and bundles all in-repo apps in its own job, and the script ends by checking that every wasm embeds its ABI. The result is cached under a key made of the git trees it is built from (every app and the crates it reaches, cargo-mero, the lockfile, the toolchain, the script and the workflows that run it), and only master saves it, so a PR whose change cannot reach the apps restores the set master already built and checked; e2e’s Build Apps restores the same entry (.github/actions/wasm-apps).

   ./scripts/build-all-apps.sh

4. Run the narrow target checks CI keeps green: the no-chain context-config build and the local-only merod init binary.

   cargo check -p calimero-context-config
   cargo check -p merod --no-default-features

5. Check the dependency policy - only needed if you changed dependencies. CI pins cargo-deny to 0.19.9 and also runs cargo machete for unused dependencies.

   cargo deny check advisories bans licenses sources
   cargo machete

Tip

Building the example apps to WASM needs the target installed once: rustup target add wasm32-unknown-unknown. The full workflow — running local nodes, commit / PR conventions, and style rules — is on the development workflow page.

Where to go next

Development workflowBuild, run local nodes, and the commit / PR conventions the gate sits inside.

Testing your appThe app-author view of TestHost — fast, deterministic in-process unit tests.