Membership & Leave Operations

Voluntary leave, owner-initiated removal, and role-scoped TEE self-purge across namespaces, groups, and contexts

3 leaves
context · group · namespace
Key rotation
on removal (publisher pipeline)
Direct
leave deletes a row
TEE purge
role-scoped hard delete

As-built page. This was originally a proposal; it now documents shipped behavior. For the fleet-specific eviction story (owner kicks a ReadOnlyTee node on HA-disable, the self-purge listener, and the marker-gated startup reconcile) see Fleet HA. Term definitions live in the Glossary. The governing design record is docs/adr/0002-fleet-tee-leave-protocol.md.

Problem this solved

Originally, the only way out of a Calimero group or namespace was to be removed by an admin — MemberRemoved was admin-initiated and there was no self-leave op. Users could not voluntarily exit a workspace they joined; muting a context they were auto-followed into required client-side filtering rather than a real opt-out, because auto-follow re-adds them on the next ContextRegistered event.

Separately, every group records a single "creator / fallback admin" identity (GroupMetaValue.owner_identity) but the role enum did not distinguish that creator from any other admin. There was no clean way to express "this person owns the group, others administrate it."

The shipped design adds three leave operations and an owner identity stored on group metadata that gates self-leave and involuntary removal. The pieces share infrastructure: the stored owner blocks leave_group/leave_namespace and involuntary MemberRemoved; owner-initiated MemberRemoved reuses the group key-rotation pipeline; leave_namespace is the namespace-root MemberLeft plus a cascade through descendant groups.

A separate, role-scoped mechanism layers on top for fleet TEE nodes: when a ReadOnlyTee member is removed, the evicted node hard-deletes its local key material (see TEE self-purge), because a TEE node has no rejoin path under the same identity. Non-TEE removals keep the soft-leave invariant.

Membership Model Recap

Background on the data shapes the leave operations operate over. Names match the code as it exists today.

Hierarchy

  • Namespace. A ContextGroupId with no parent. The trust anchor for everything beneath it. There is no separate Namespace type — a namespace is a root group.
  • Group / subgroup. A ContextGroup with a parent_group_id. Forms a strict tree under a single namespace.
  • Context. A unit of replicated state. Belongs to exactly one group via ContextGroupRef. Members of the group can join, sync, and write to it.

Direct vs inherited membership

Group membership is computed by check_group_membership_path (crates/context/src/group_store/membership.rs:217-291):

  • Direct membership = a stored (member, group) row exists. Created by add_group_members (explicit invitation). Required for Restricted groups; possible but redundant for Open subgroups.
  • Inherited membership = no row exists; the walker concludes membership by walking the parent chain through Open subgroups when the member holds the CAN_JOIN_OPEN_SUBGROUPS capability in the parent.

This distinction is critical for leave operations. Leave deletes a direct row. If the only path to membership is inheritance, there is no row to delete — the member must leave whichever ancestor provides the inheritance.

In the governance-DAG projection layer, this distinction is reflected by the fold rule for RootOp::MemberJoinedOpen: it folds to OpPayload::Noop rather than OpPayload::MemberAdded. No persistent GroupMember node is written for an open-subgroup inheritance join; instead, AclView::is_member_at_cut re-derives membership at query time via the anchor-root membership + subgroup tree + visibility + CAN_JOIN_OPEN_SUBGROUPS capability walk — exactly mirroring the live store. This means inherited access is automatically revoked when the member is removed from the root group and restored if they rejoin, with no over-grant surviving root-anchor removal.

In the governance-DAG projection layer, this distinction is reflected by the fold rule for RootOp::MemberJoinedOpen: it folds to OpPayload::Noop rather than OpPayload::MemberAdded. No persistent GroupMember node is written for an open-subgroup inheritance join; instead, AclView::is_member_at_cut re-derives membership at query time via the anchor-root membership + subgroup tree + visibility + CAN_JOIN_OPEN_SUBGROUPS capability walk — exactly mirroring the live store. This means inherited access is automatically revoked when the member is removed from the root group and restored if they rejoin, with no over-grant surviving root-anchor removal.

This behavior is pinned by three assertions in projection_membership_equivalence.rs: (1) projection_matches_live_across_inherited_join_and_root_removal asserts that after an inherited join, member_at_cut_authoritative returns Some(true); (2) the same test asserts that after a root removal that the live store rejects, member_at_cut_authoritative must not return Some(true) — the over-grant guard; (3) projection_matches_live_across_leave_and_rejoin_inheritance asserts that after a full leave-and-rejoin cycle, member_at_cut_authoritative correctly restores access. Together these tests ensure the authoritative grant resolver agrees with the live store across all three scenarios and never over-grants on root-anchor removal.

Visibility (Open vs Restricted)

A per-group flag (VisibilityMode::{Open, Restricted} in crates/context/config/src/lib.rs:134-137) that controls whether the parent-walk inherits into this subgroup or stops cold:

  • Open: parent members with CAN_JOIN_OPEN_SUBGROUPS inherit membership.
  • Restricted: walks halt; only direct invitations admit members.

At leave-time, visibility doesn't change behavior — it only affects whether a row exists in the first place. Most members of Open subgroups have no row (inherited); most members of Restricted subgroups have one (direct invite).

Roles & capabilities

The GroupMemberRole enum (crates/primitives/src/context.rs) has exactly four variants:

  • Admin, Member, ReadOnly, ReadOnlyTee.
  • There is no Owner role variant. "Owner" is not a member role — it is a single identity stored on group metadata (GroupMetaValue.owner_identity). The owner is whichever member's public key equals that field; they are otherwise an ordinary Admin. Owner status is enforced by identity comparison in the apply handlers (OwnerImmuneFromRemoval, OwnerCannotSelfLeave), not by a role check. See the Owner section.

Per-member capability bits (crates/context/config/src/lib.rs, MemberCapabilities):

  • CAN_CREATE_CONTEXT, CAN_INVITE_MEMBERS, CAN_JOIN_OPEN_SUBGROUPS, MANAGE_MEMBERS, MANAGE_APPLICATION, CAN_CREATE_SUBGROUP, CAN_DELETE_SUBGROUP, CAN_MANAGE_VISIBILITY, CAN_MANAGE_METADATA.

Existing protections

  • Last-admin protection (membership_policy.rs:39-46): MemberRemoved rejects with LastAdmin if it would leave a group with zero admins. Also enforced on demotion.
  • Key rotation on removal (group_governance_publisher.rs, sign_apply_and_publish_inner): publishing MemberRemoved automatically generates a fresh group key via GroupKeyring, wraps it (wrap_for_member) for every remaining member — excluding the removed one — and delivers it. The removed member never receives the new key, so it cannot decrypt anything written after eviction. This is the forward-secrecy mechanism for the namespace's future writes. Owner-initiated removal is the cryptographically-complete leave path.
  • Namespace genesis op (create_group.rs, parent_group_id == None branch): when a namespace root is created, the handler signs, applies, and publishes a RootOp::NamespaceCreated { founder: admin_identity } genesis op via sign_apply_and_publish_namespace_op. A NoPeersSubscribed error from publish is treated as success (apply succeeded locally; publish is best-effort). Any other genesis-apply error triggers a full rollback of all locally-written root rows — meta, founder Admin member row, default caps, group key, and signing key (plus optional name metadata) — via individual idempotent deletes (GroupKeyring::delete_key_by_id, SigningKeysRepository::delete_key, MetaRepository::delete, MembershipRepository::remove_member, CapabilitiesRepository::delete_default, MetadataRepository::delete_group) before returning Err. The namespace identity row is deliberately not rolled back so a retry obtains the same identity/founder. The DAG head does not require rollback because apply_signed_op is head-atomic (the head advances only after a successful apply).

Forward-only invariant — load-bearing

The apply-time membership check is a function of two things only: the signer's identity and the governance-DAG position the signer signed against. It is not a function of the receiver's current local state. This is the forward-only property: a write authored at governance position P stays valid forever, even if the receiver has since applied a MemberRemoved for the signer at a later position P' in its local DAG.

Why this matters

Without forward-only, two peers receiving the same set of ops in different orders would end up with different state:

  • Peer A receives MemberRemoved first, then the pre-removal delta. If "is the signer currently a member?" gates the apply, the delta is rejected.
  • Peer B receives the delta first, then MemberRemoved. The delta applies; the later removal doesn't retroactively undo it.

That divergence cascades: any subsequent delta that reads the (different) state diverges further, context root hashes drift apart, and the network has no honest convergence point. Forward-only removes the receive-order dependence: pre-removal writes always apply on every peer, regardless of when the receiver learns about the removal.

How it's implemented

Every state delta carries a governance_position field that names the governance-DAG heads the author observed at sign time. On receive, the apply path evaluates membership at the signed heads via the unified projection — which resolves the ancestry of those heads and derives the membership state from the folded projection. The resolution visits only the prefix that was already final when the delta was signed; anything causally after the signed cut (including a later MemberRemoved for this signer) is invisible to the resolution and cannot retroactively invalidate the delta.

The namespace-level applier (NamespaceGovernanceApplier in crates/context/src/governance_dag.rs) threads this cut into the apply path directly: DeltaApplier<SignedNamespaceOp>::apply calls apply_signed_namespace_op_at_cut(&delta.parents, &LIVE_FALLBACK_AUTHORIZER) rather than the cut-free apply_signed_namespace_op. Passing delta.parents as the causal cut ensures membership is evaluated at the position the author signed against. LIVE_FALLBACK_AUTHORIZER is a placeholder authorizer that delegates to the live store; membership authorization for all other paths is now handled entirely by the unified projection (EphemeralProjectionAuthorizer / ScopeProjections) — the acl_view_at, MembershipStatus, prefix_walk_membership, MembershipTransition, heads_equal, and MAX_PREFIX_WALK_NODES helpers that previously backed the live DAG-walk resolver have been deleted.

Resolving the current governance cut — ScopeProjections::namespace_current_heads

Current-state membership reads (as opposed to historical signed-position checks) need a "now" frontier: the set of parent hashes that represents the namespace's current DAG head. The static method ScopeProjections::namespace_current_heads provides this:

  1. Resolves the supplied ContextGroupId to its owning namespace by walking the parent chain.
  2. Reads that namespace's DAG head record from the governance store.
  3. Returns the parent hashes of that head — the cut argument passed to membership_status_at (or AclView::is_member_at_cut) for a current-state query.

None return cases. The method returns None when the group cannot be resolved to a namespace (e.g. the group does not exist in the local store) or when the head record itself is unreadable. Callers must fall back to the live membership store (check_group_membership_path) in both cases rather than treating None as an empty cut — an empty cut would falsely deny all access.

Projection-derived member counts and migration cohorts. ScopeProjections::member_identities_with extends this pattern to identity-set reads: given an already-built projection fold it calls cut_ancestry_complete on the fold before reading the ACL view, and returns None if the ancestry is only partially folded (a governance-backfill race). This None propagates to get_group_info (which falls back to MembershipRepository::count + enumerate_inherited) and to collect_migration_cohort (which falls back to the live list ∪ enumerate_inherited union). The private ephemeral_view helper has been removed; the subtree ephemeral path now calls ephemeral_fold directly and performs the cut_ancestry_complete check itself before reading the view.

Ephemeral per-query projection — ScopeProjections::member_now_ephemeral

Context-layer handlers that do not hold a cached ScopeProjections instance use ScopeProjections::member_now_ephemeral to obtain a one-shot membership verdict from the governance DAG without requiring a pre-built projection. On each call it:

  1. Resolves the supplied group to its owning namespace.
  2. Folds the full governance DAG for that namespace via collect_namespace_ops.
  3. Reads the DAG heads after the fold (not before), so a concurrent governance-op advance cannot leave the cut naming ops the fold hasn't processed.
  4. Calls member_at_cut at those heads and returns the boolean verdict.

None return cases. The method returns None — never deciding against the member — when: (a) the namespace is unresolvable, (b) collect_namespace_ops reports a store fault (surfaced with a tracing::warn), or (c) the cut-completeness guard determines the recorded heads don't cover the full fold. Callers must treat None as undecidable and fall back to the live store rather than denying access.

Unified membership gate — ScopeProjections::member_now_checked

ScopeProjections::member_now_checked is the single decision point used by all context-layer membership gates (get_cascade_status, get_group_upgrade_status, list_group_contexts, list_all_groups). Its semantics:

  • Projection wins when decidable. It calls member_now_ephemeral first. When the projection returns a verdict it acts on that verdict, then cross-checks MembershipRepository::is_member best-effort. A transient live error does not override a projection verdict.
  • Live wins when projection is undecidable. When member_now_ephemeral returns None, member_now_checked falls back to MembershipRepository::is_member as authoritative and propagates any live error normally.
  • Divergence telemetry. When the projection and live store both return definite but conflicting answers, member_now_checked emits a tracing::warn with marker=unified_projection_divergence and plane=membership-query. The projection verdict is still used as the final answer.

Direct use of MembershipRepository in get_cascade_status.rs, get_group_upgrade_status.rs, list_all_groups.rs, and list_group_contexts.rs is eliminated; those handlers now call member_now_checked exclusively and no longer import MembershipRepository.

Behavior change in list_all_groups. Previously, a membership-check error in the per-group loop would propagate and abort the entire response. Now the loop catches errors from member_now_checked with a tracing::warn and continues to the next group, silently omitting groups whose membership cannot be determined. This matches the existing skip behavior for a missing node identity and avoids a single inaccessible group blocking the whole listing.

Op-store reconstruction root — ScopeProjections::scope_root_from_op_store and shadow_compare_op_store

ScopeProjections::scope_root_from_op_store(scope_id, store) provides an independent view of a scope's governance root derived entirely from the persisted op log rather than the maintained in-memory projection:

  1. Loads all ops for the given ScopeId via load_scope_ops.
  2. Replays them into a fresh ScopeProjections instance (causal ordering recovered through DagStore<Op> + UnifiedApplier).
  3. Returns the resulting scope_root computed with a fixed zero entities_root.

Returns Ok(None) when the scope has no persisted ops; Err on store faults. This is intended for completeness verification before C2.2b — not for hot-path membership decisions.

ScopeProjections::shadow_compare_op_store(namespace_id, store) is an observe-only companion that compares the maintained projection's scope_root against the op-store reconstruction. On mismatch it emits a tracing::warn with marker=unified_op_store_divergence and truncated hex of both roots. This log marker is the e2e gate signal for C2.2b readiness. The method silently skips (no divergence reported) when the scope is unfolded, the op-store is empty for it, or the store read fails — none of those conditions count as divergence. calimero_op / ScopeId types stay inside the context crate and are not leaked into the node crate.

Convergence root — ScopeProjections::scope_root_for and group_scope_root_ephemeral

ScopeProjections::scope_root_for(scope) computes a single SHA-256 digest over the folded state of a governance scope: SHA-256(entities_root ‖ acl_hash ‖ governance_hash). This gives callers a cheap way to detect whether two nodes have converged on the same membership/governance state without enumerating members.

  • Returns None when the scope has not been folded yet (i.e. it has no entry in the projection). Callers must treat None as "cannot compare" — never as a divergence signal. False divergences from mid-backfill nodes are the main hazard here.
  • ACL component is currently an empty placeholder because SetWriters ops are not yet routed through the governance DAG. The hash over ACL state will become meaningful at C2 when that routing lands.

The companion free function ScopeProjections::group_scope_root_ephemeral(group_id, store) builds an ephemeral projection from the raw store (no NodeState required), adds an ancestry-completeness guard via cut_ancestry_complete, and delegates to scope_root_for. It returns None when the governance DAG cut is not fully folded, preventing false divergence reports from nodes that are still mid-backfill. Both symbols are public API on ScopeProjections (crates/context/src/scope_projections.rs).

Behavior is pinned by scope_root_for_moves_on_hash_neutral_membership_change_and_is_none_when_unfolded: the test holds entities_root fixed, ingests an AdminChanged op then a causally-chained MemberAdded op, and asserts that scope_root_for returns different values before and after, and returns None for an unfolded scope.

Batch role resolution — ScopeProjections::member_roles_for

Where namespace_current_heads answers "is this member present?", member_roles_for answers "what role does the projection assign to each of these members?". It folds the ACL view exactly once for the supplied batch and returns a BTreeMap<PublicKey, GroupMemberRole>:

  • Direct members keep their folded role from the projection.
  • Inherited{via_admin:true}Admin.
  • Inherited{via_admin:false} → the member's stored role at the anchor group (mirroring live's list ∪ enumerate_inherited semantics).
  • Abstain (empty map) when the ancestry is not fully folded, or when the group has no entry in view.groups (materialized-fallback parity — Restricted subgroup or undecryptable member ops). An empty return means "projection has no opinion"; callers must not treat it as "all roles are wrong".

list_group_members calls member_roles_for once for all live members after the existing identity-set shadow. Each live member's role is compared to its projected role; mismatches are emitted as unified_projection_divergence log events with plane=membership-role. Members absent from the returned map (projection abstains) are skipped — presence divergence is already owned by the identity shadow. The handler still returns the live list unchanged.

Operator note. membership-role divergence warnings in logs indicate a pre-flip validation failure. They will block the eventual flip of list_group_members from the live store to the projection as the authoritative source. Note that acl_view_at and MembershipStatus (the former live DAG-walk resolver types) are no longer available; divergence detection now operates entirely through ScopeProjections projection methods.

Full effective-member enumeration — ScopeProjections::member_entries_with

member_entries_with returns a Vec<(PublicKey, GroupMemberRole)> for every effective member of a group at the current cut, derived entirely from the already-folded auth_cut_context view. It reuses member_identities_in_view_with_ctx for the identity set and resolves each member's role via member_path_at_cut.

The method returns None (signalling: defer to the live store) in three cases:

  • The cited ancestry is not yet fully folded — a partial fold could produce an incomplete member list, so the caller falls back rather than silently truncating.
  • The target group has no folded direct-membership entry in the view — the projection does not cover this group yet.
  • A validated identity resolves to MemberPathAtCut::None or has a missing inherited anchor row — this indicates a fold inconsistency. The method logs a unified_projection_divergence warning and abandons the entire enumeration rather than returning a partial list.

The all-or-nothing fallback contract means callers never observe a silently truncated member list: either the projection supplies the complete set or the live store is used in its entirety.

At-cut effective role — ScopeProjections::role_at_cut_for_group

ScopeProjections::role_at_cut_for_group(member, group_id, cut) resolves the effective GroupMemberRole of a member in a group at a specific governance cut. It gates on cut_ancestry_complete via auth_cut_context (returning None immediately when ancestry is incomplete) and then delegates to member_path_at_cut to handle three cases:

  • Direct membership — returns the folded role from the projection (the same derivation already validated by the membership-role divergence plane).
  • Inherited{via_admin:true} — always resolves to Admin, mirroring the live semantics for anchor-via-admin inheritance.
  • Inherited{via_admin:false} — returns the member's stored role at the anchor group. Returns None rather than guessing when the anchor row is missing, preventing spurious divergence reports.

The method returns None — signalling "projection has no opinion, defer to live store" — in four cases: ancestry is not yet fully folded, the member is absent from the group at the cut, the auth_cut_context call cannot resolve the namespace, and the anchor row is missing for an inherited-via-anchor path. Callers must not treat None as a role denial.

Shared-context identity resolution — ScopeProjections::member_identities_in_view_with_ctx

member_identities_in_view_with_ctx accepts caller-supplied root and default_cap_base instead of re-reading MetaRepository / CapabilitiesRepository internally. member_identities_in_view delegates to it unchanged (existing callers are unaffected). member_entries_with passes the values it already obtained from auth_cut_context, ensuring the candidate-identity filter and the per-member role resolution use a single consistent store snapshot with no window for a concurrent governance write to make them disagree. Prefer member_identities_in_view_with_ctx over member_identities_in_view whenever the caller already holds the root and capability values — it avoids a redundant store read and eliminates the TOCTOU window.

Projection-first list_group_members with stable pagination

The list_group_members handler now reads from the projection authoritatively when a folded proj_ctx is available. It falls back to the live MembershipRepository::list ∪ enumerate_inherited union only when member_entries_with returns None. Previously the handler always read from the live store and compared the projection in shadow mode; the shadow scaffolding methods shadow_member_enum_with and member_roles_for have been deleted as they had no remaining callers.

Both the projection path and the live-store fallback path now apply a members.sort_by_key(|(identity, _)| *identity) sort by PublicKey before the skip(offset).take(limit) slice. The projection returns a BTreeSet-ordered (PublicKey-sorted) result while the live store returns insertion/store order; without the explicit sort, a mid-backfill request served by the projection followed by a next-page request served by the live store could skip or repeat members. The sort makes pagination order stable and path-independent across a mid-backfill transition.

member_now_checked / member_now_checked_with

ScopeProjections::member_now_checked and its shared-fold variant member_now_checked_with are the two query-gate helpers that call namespace_current_heads and then delegate to AclView::is_member_at_cut. Their behavior follows a strict two-branch contract:

  • Projection returns Some — the result is acted on directly. There is no secondary MembershipRepository::is_member live read and no unified_projection_divergence cross-check. The projection verdict is authoritative when it is available.
  • Projection returns None (unresolvable group or unreadable head) — fall back to the live membership store (check_group_membership_path) as before.

The earlier best-effort live cross-check (which compared the projection verdict against the live resolver and emitted a warning/error on divergence) has been removed from both functions. The None→live fallback path is preserved unchanged.

At-cut admin & capability accessors — is_admin_at_cut, is_admin_or_capability_at_cut, is_last_admin_at_cut

Five additional methods on ScopeProjections expose admin/capability checks against a historic DAG cut, mirroring what the live authorization gate does at apply time:

  • auth_cut_context(group_id, cut) — shared setup used by the public accessors. Resolves the namespace for the supplied group, gates on cut_ancestry_complete (returns None immediately if the cited ancestry is not yet fully folded), builds the AclView folded to the cut, and returns the genesis root tuple plus default-capability base.
  • is_admin_at_cut(signer, group_id, cut) — returns Some(true) if the signer holds an admin role at the cut, Some(false) if they do not, and None when ancestry is incomplete. Callers must defer to the live gate on None — incomplete ancestry is not equivalent to unauthorized.
  • is_admin_or_capability_at_cut(signer, group_id, cut, cap_bits) — admin OR capability-bit check, mirroring the live is_authorized_with_capability. Same None semantics as above. This method is a required method on the AtCutAuthorizer trait; all implementors must provide it.
  • is_last_admin_at_cut(member, group_id, cut) — answers, at the op's parent cut (pre-mutation), whether removing or demoting the given member would orphan the group's admin set. Mirrors live logic exactly: a member counts as admin if they hold a direct Admin-role row or are the genesis admin; "another admin" requires a distinct Admin-role row (the genesis admin alone does not satisfy it). Returns None when ancestry is incomplete, deferring to the live last-admin guard.

    Causal-position semantics. Because this method reads the DAG state the signer observed at sign time (the parent cut), not the receiver's current local state, it can legitimately disagree with a live query made after concurrent ops have advanced the DAG — this is the causal-honor property and is expected behavior, not a divergence.
  • membership_path_at_cut(member, group_id, cut) — returns Some(AtCutMembershipPath) indicating whether the member has a direct row, an inherited path, or no membership at the cut, and None when ancestry is incomplete. Delegates to ScopeProjections::membership_path_at_cut, which calls member_path_at_cut on the folded projection via auth_cut_context and maps the result (projecting away the role/anchor detail). Callers must defer to the live gate on None.

All public accessors return None (not Some(false)) when cited ancestry is not fully folded, ensuring callers do not treat an incomplete fold as a denial verdict.

AtCutAuthorizer trait and implementations

The AtCutAuthorizer trait abstracts over how an at-cut admin/capability check is resolved. It declares four required methods: is_admin_at_cut, is_admin_or_capability_at_cut, is_last_admin_at_cut, and membership_path_at_cut. All return Option<…> with the same None = undecidable semantics.

Empty-cut contract. Every method on this trait must return None when the supplied parents slice is empty. Violating this contract would cause a genesis-state fold (no MemberAdded or role ops processed) to produce Some(false), falsely rejecting genesis ops that the live store correctly permits.

  • LiveFallbackAuthorizer — the placeholder implementation used by the current namespace-envelope applier stage. Returns None for all four methods, delegating all decisions to the live store.
  • EphemeralProjectionAuthorizer — the projection-backed implementation. Holds a Mutex<Option<(ContextGroupId, Arc<FoldedProjection>)>> cache. The private folded(group_id) method checks the cache first; on a cache miss it calls ScopeProjections::ephemeral_projection and stores the result. If the fold fails (store or DAG-head fault) it emits a tracing::warn and returns None. All four at-cut methods go through folded, so a single apply call reuses the same folded DAG rather than re-folding per check. is_last_admin_at_cut delegates to ScopeProjections::is_last_admin_at_cut and guards against an empty parents slice with an immediate None return, consistent with the trait-level empty-cut contract. membership_path_at_cut delegates to ScopeProjections::membership_path_at_cut, which calls the projection's member_path_at_cut walk via auth_cut_context and maps the result to AtCutMembershipPath (projecting away the role/anchor detail). An empty parents slice returns None immediately, consistent with the empty-cut contract.

GroupGovernanceApplier — live gates only, no authorizer injection

The standalone GroupGovernanceApplier::apply continues to call apply_local_signed_group_op (live-fallback) and does not accept an injected AtCutAuthorizer. A code comment documents why: a SignedGroupOp's parent_op_hashes belong to the per-group op log, not the namespace governance log that EphemeralProjectionAuthorizer is keyed by. Passing those hashes to the projection authorizer would resolve against the wrong id-space and silently no-op, producing a false "undecidable" verdict on every group op. The real group-auth projection shadow therefore fires only on the namespace-envelope group-op path, not through the standalone applier.

UnifiedApplier — projection-backing DeltaApplier<Op>

UnifiedApplier is the C2.0 struct that bridges DagStore<Op> and ScopeProjections. It implements DeltaApplier<Op>: its apply method acquires a write lock on the inner Arc<std::sync::RwLock<ScopeProjections>>, calls ScopeProjections::ingest_op with the delta's payload, then releases the lock. The lock is a std::sync::RwLock (not a Tokio lock) and is never held across an .await; lock poison is recovered rather than propagated, matching the stance already documented on NodeState::read_scope_projections. Two constructors are provided:

  • UnifiedApplier::new() — creates a private, standalone projection. Use this for replay pipelines and tests.
  • UnifiedApplier::with_projection(Arc<RwLock<ScopeProjections>>) — attaches to the node's live shared projection. This is the C2.1 dual-write seam: governance ops processed by the existing live pipeline are simultaneously ingested into the shared projection.

A projection() accessor exposes the Arc for inspection or cross-checking. pub mod unified_applier is exported from crates/context/src/lib.rs, making UnifiedApplier part of the crate's public API.

Behavior is pinned by dag_releases_out_of_order_ops_and_the_projection_converges: a four-op causal chain spanning the admin, membership, ACL, and data planes is fed through DagStore<Op> + UnifiedApplier in four delivery orders (in-order, full reverse, and two interleaved). The test asserts that every op is applied (none stuck pending) and that scope_root matches the result of direct in-order folding in all cases — confirming causal ordering and projection convergence regardless of receive order.

Shadow authorization check in apply_signed_namespace_op

After a governance op is successfully applied by the live gate, the handler runs a non-blocking shadow authorization check for observability. The check is implemented in two parts:

  • apply_auth_requirement(op) maps the just-applied SignedNamespaceOp to an (ContextGroupId, ApplyAuthReq) pair. ApplyAuthReq is either Admin or AdminOrCap(bits). The mapping covers:
    • Root-level AdminChanged, PolicyUpdated, GroupReparentedAdmin
    • GroupCreatedAdminOrCap(CAN_CREATE_SUBGROUP)
    • Group-level MemberAdded / MemberRemovedAdminOrCap(MANAGE_MEMBERS)
    • SubgroupVisibilitySetAdminOrCap(CAN_MANAGE_VISIBILITY)
    • MemberRoleSet, MemberCapabilitySet, DefaultCapabilitiesSetAdmin
    Explicitly excluded from the mapping (returning None): MemberJoined / MemberJoinedAt (inviter authority; MemberJoinedAt additionally enforces deterministic expiry — apply rejects when joined_at > expiration_timestamp), MemberJoinedOpen (inheritance proof), MemberLeft (self-authored), GroupDeleted and TransferOwnership (owner-gated), and other metadata/context-registration ops.
  • The handler then calls the corresponding at-cut accessor against the op's delta_parents cut (the DAG state the signer observed, not the post-apply cut). If the projection returns Some(false) — it would have rejected what the live gate accepted — a warn! is emitted with marker=unified_projection_divergence and plane=governance-auth. None (ancestry incomplete) is silently skipped. The check runs outside the projection write lock under a brief read lock.

Live authorization gates are entirely unchanged. The shadow check is log-only and has no effect on whether an op is accepted or rejected.

Concurrent-op convergence test — concurrent_independent_member_adds_converge

The integration test concurrent_independent_member_adds_converge pins the CRDT convergence semantics for concurrent ops authored against the same genesis state. Two admins independently add different members (D and E) against the same genesis state; the ops are applied in opposite orders on two nodes. The test asserts: (1) both ops succeed on both nodes, (2) both nodes converge to the identical four-member set, and (3) replaying a seen op is a no-op — the per-signer nonce window dedups it and the member count does not change. The only apply gate is the per-signer nonce window; there is no state-hash staleness check. The old test (state_hash_prevents_concurrent_op_divergence) asserted that the second concurrent op must fail; that hard-reject behavior no longer exists and the old test has been removed.

Where this can break

Two classes of change would silently re-introduce the divergence described above:

  • Calling the projection with the wrong position. Any caller of the at-cut membership resolver that passes the receiver's current state (or any post-cut heuristic) instead of the signed delta position breaks forward-only. The three production call sites — gossip-receive, governance-pending drain, and snapshot-sync replay — each carry inline comments naming this contract; reviewers should flag any new caller that doesn't pass delta.governance_position verbatim.
  • Expanding the cut resolution to visit descendants. A "performance optimization" that advances the resolved cut beyond the signed heads would observe ops the signer hadn't signed against, including later MemberRemoved ops. The forward-only contract is documented on the public projection at-cut methods; the prefix_walk_forward_only_* regression tests pin the boundary.
  • Reintroducing a state-hash staleness gate. Governance ops no longer carry a meaningful root-hash claim (state_hash has been removed from the synthesized SignedGroupOp in decrypt_and_apply_group_op, and signed_op_to_delta / signed_namespace_op_to_delta now pass [0u8; 32] as expected_root_hash). Any new code that gates apply on a non-zero root-hash match would incorrectly reject concurrent ops and break the CRDT convergence property pinned by concurrent_independent_member_adds_converge. The per-signer nonce window is the sole apply gate for replay deduplication.

Scope

Forward-only applies to MemberRemoved and MemberLeft — operations that revoke authorship. It does not apply to role demotions (MemberRoleSet transitioning a Member to ReadOnly): those are enforced against current state at the receive path, separately from the cross-DAG check, because a member who was demoted retains a different relationship to their pre-demotion writes than a member who was fully removed. See is_read_only_for_context for the live-state role gate.

leave_context — local-only opt-out

Stop syncing a context on this node. No DAG op, no broadcast, no key rotation. Other members never observe the leave; from their perspective the leaver is still in the membership list (which is implicit anyway, computed from group membership + auto-follow).

Storage

Two separate stores live side by side: the synced ContextIdentity row (key at crates/store/src/key/context.rs:120, value at crates/store/src/types/context.rs:117) and a node-local ContextLeftMarker tombstone in a new column.

The marker lives in Column::ContextLocal (a new column specifically for node-local context-scoped state — never synchronized). Keying is identical to ContextIdentity so the (context_id, member_pk) pair is the unit of opt-out:

pub struct ContextLeftMarker(Key<(ContextId, PublicKeyComponent)>);

// Borsh value:
pub struct ContextLeftMarker {
    pub left_at_ms: u64,
}

A separate column instead of a flag on ContextIdentity for two reasons: (a) ContextIdentity is part of the synced membership shape, so adding a node-local field to it would conflate replicated and node-local state; (b) extending the existing Borsh value would break deserialization of pre-existing on-disk rows, requiring a migration. The dedicated column makes the node-local-ness explicit at the storage layer and keeps the synced shape clean.

Flow

  1. Client calls leave_context(context_id).
  2. Handler resolves the local ContextIdentity via find_local_signing_identity — scans the column for the row where this node holds the private key. Falls back to the namespace identity if no local row exists.
  3. In a single batched store handle (so they land in one commit), the handler:
    1. Writes the ContextLeftMarker row in Column::ContextLocal.
    2. Deletes the corresponding ContextIdentity row, which stops sync (the sync layer iterates these rows).
  4. Calls node_client.unsubscribe(&context_id) to leave the gossipsub topic, mirroring join_context's subscribe.
  5. Auto-follow handler (auto_follow.rs:255) checks for the marker via has_left_context on every ContextRegistered event and skips the auto-rejoin if it finds one.

Properties

ScopeLocal only
DAG entryNone
Key rotationNone — leaver still holds the group key
CoordinationNone — cannot fail
ReversibilityTrivial — JoinContextRequest deletes the marker as a side-effect of joining and writes a fresh ContextIdentity row
Other members observeNo
Forward secrecyNot provided — leaver retains the key, can decrypt anything they sync later if they rejoin

What this is not

  • Not a kick. Leaver still appears in any "members of this context" query.
  • Not a cryptographic leave. If the user wants forward secrecy on a context, they must leave the group containing it (which rotates the group key).

leave_group — self-removal, no cascade

Self-removal from a single group via GroupOp::MemberLeft. The leaver publishes the leave op; every receiver deletes the leaver's (member, group) row.

As-built caveat — no key rotation on self-leave. Unlike owner-initiated MemberRemoved, MemberLeft deliberately does not trigger the key-rotation pipeline. The publisher of a self-leave is the leaver, and the leaver cannot generate the fresh group key without also retaining it — which would defeat forward secrecy. So MemberLeft is the governance-level departure (the membership row is removed, peers observe the leave, the leaver is deny-listed) without a re-key. The path to a cryptographically-complete leave is an admin/owner-initiated MemberRemoved, which rotates. A two-phase self-leave rotation (a remaining admin's apply hook publishes the new key) is a tracked follow-up, not yet shipped. See the inline note in crates/governance-store/src/ops/group/member_left.rs.

New op

enum GroupOp {
    // existing variants...
    MemberLeft { member: PublicKey },
}

Distinct from MemberRemoved for audit clarity — "this member chose to leave" vs "this member was removed" are semantically different events even when they have similar effects.

Preconditions

  • Signer equals the member being removed (SelfLeaveOnly) and has a direct row in the group; reject with MemberNotDirect if the signer is only an inherited member — they must leave the anchor instead.
  • Signer is not the group's owner_identity (OwnerCannotSelfLeave; must TransferOwnership first).
  • Removing the signer does not leave the group with zero admins (LastAdmin via ensure_not_last_admin_removal).

Flow

  1. Leaver's client calls leave_group(group_id).
  2. Apply enforces the preconditions above (self-equality, direct-row, owner, last-admin).
  3. Leaver publishes MemberLeft { member: signer } via the governance DAG. Signed by the leaver. Targets this group.
  4. Op broadcasts to all members of the group. Each receiver applies it: cascade_remove_member_from_group_tree drops the leaver's ContextIdentity rows, remove_member deletes the (leaver, group_id) membership row, and the leaver is added to the group deny-list so their state-delta traffic is dropped until they re-join.
  5. The apply emits OpEvent::MemberRemoved; if the leaver's role was ReadOnlyTee it additionally emits OpEvent::TeeMemberRemoved, which drives the leaver's own node to self-purge the group's local key material.
  6. No automatic re-key. As noted above, MemberLeft does not rotate the group key — the leaver still holds the last key it received and can decrypt writes made under it. Forward secrecy on future writes requires the owner-initiated MemberRemoved path (or the deferred two-phase self-leave rotation).

Cascade behavior

Direct rows in subgroups under this groupUntouched. Leaver keeps those memberships. (You can be invited to a child while explicitly leaving the parent.)
Inherited memberships in Open subgroupsResolve themselves. The walker stops finding the leaver once their parent row is gone — no extra work needed.

Failure modes

  • No forward secrecy on self-leave (by design, not a failure). Because MemberLeft never rotates, the leaver retains the ability to decrypt writes made under the key they last held. To revoke that, an admin/owner must issue MemberRemoved (which rotates) or the deferred two-phase self-leave rotation must land.
  • Inherited-only membership. A signer whose membership is purely inherited (no direct row) is rejected with MemberNotDirect; they must leave the anchoring ancestor instead.

Rejoin

  • Restricted group: requires fresh add_group_members from a remaining admin. New keys delivered via the existing membership-add path.
  • Open group (if leaver had a redundant direct row): same. If leaver's path to the group was via inheritance from a parent, that inheritance is automatically restored as soon as they rejoin the parent.

leave_namespace — cascading, heaviest

A namespace is the root group. Leaving it is a single MemberLeft at the namespace root whose apply cascades through every descendant group where the leaver has a direct row. This is the only leave operation that fans out across multiple scopes. As with leave_group, the self-leave op does not rotate keys (same publisher-holds-the-key constraint); row removal + deny-listing only.

Preconditions

  • Signer has a direct row at the namespace root (MemberNotDirect otherwise).
  • Signer is not the owner_identity of the namespace root (OwnerCannotSelfLeave) nor of any subgroup beneath it where they have a direct row (OwnerOwnsSubgroup, reporting the offending scope).
  • Removing the signer doesn't leave any scope (namespace or descendant) admin-less. If it would → LastAdmin reports the offending scope, forcing a successor promotion before retry. All these checks run upfront, before any mutation.

Flow

  1. Client calls leave_namespace(namespace_id).
  2. Apply detects the no-parent (namespace-root) case and walks the subtree:
    • Compute the descendant groups where the signer has a direct row.
    • Run owner check + last-admin check across all of them upfront. If any fails (OwnerOwnsSubgroup, OwnerCannotSelfLeave, LastAdmin) it bails before mutating anything — no half-applied cleanup.
  3. Signer publishes a single MemberLeft { member: signer } at the namespace root.
  4. The apply cascades: for each direct-row descendant it calls cascade_remove_member_from_group_tree, removes the membership row, deny-lists the leaver, and emits OpEvent::MemberRemoved. For each descendant where the leaver's role was ReadOnlyTee it also emits a per-subgroup OpEvent::TeeMemberRemoved. The same removal then runs for the namespace root, emitting a root MemberRemoved (and a root TeeMemberRemoved if the root role was ReadOnlyTee).
  5. No per-scope key rotation. The cascade is row-removal only — there is no KeyDelivery fan-out. Forward secrecy on each scope's future writes is provided by the owner-initiated MemberRemoved rotation pipeline, not by self-leave.
  6. Local cleanup on the leaver's node is role-scoped, not a soft/hard toggle:
    • Non-TEE roles: soft leave. Local rows (identity, signing keys, contexts) are kept so rejoin / keyshare / inheritance-rejoin flows can reuse them.
    • ReadOnlyTee: the emitted TeeMemberRemoved events drive the self-purge listener to hard-delete local key material across the whole subtree and unsubscribe from the namespace gossipsub topic.

Edge case — last member of the namespace

If the leaver is the last remaining member, MemberLeft writes locally but has no peers to broadcast to. The namespace effectively dies on the leaver's node.

If the leaver is also the owner AND last member, they're stuck in the cycle — they must transfer ownership but there's no one to transfer to. Use a dedicated DeleteNamespace op (analogous to DeleteGroup but at root) that bypasses the owner-cannot-leave rule.

Failure modes

  • No forward secrecy on self-leave across any scope (same constraint as leave_group): the cascade removes rows but does not re-key. Use owner-initiated MemberRemoved for a cryptographically-complete eviction.
  • Interrupted TEE self-purge. If the leaver was a ReadOnlyTee node and its local cascade-purge is interrupted (crash, or a signing-key delete error), a durable pending-self-purge marker survives and the startup reconcile_sweep completes the purge on next restart. See TEE self-purge.

Role-scoped TEE self-purge

Removal alone (the MemberRemoved rotation) already provides forward secrecy on a namespace's future writes — the evicted node never receives the rotated key. But a ReadOnlyTee fleet node still holds, on its own disk, the old key material from before the rotation: signing keys, the group encryption keys it last received, its NamespaceIdentity, and the governance op-log. A regular member's local rows are deliberately kept (the soft-leave invariant lets kick-and-readd / rejoin-via-keyshare / inheritance-rejoin reuse them). A ReadOnlyTee node has no rejoin path — re-admission via MemberJoinedViaTeeAttestation derives a fresh attestation pubkey — so that material buys nothing and is pure hygiene debt. The self-purge handler hard-deletes it.

Design record: docs/adr/0002-fleet-tee-leave-protocol.md. Implementation: crates/context/src/self_purge.rs. Fleet-side framing: Fleet HA.

Role-scoped trigger

The apply path emits a generic OpEvent::MemberRemoved for every removal, and — only when the removed member's stored role was ReadOnlyTee — an additional OpEvent::TeeMemberRemoved (see ops/group/member_removed.rs and member_left.rs). The self-purge listener subscribes to the op-apply event channel and reacts only to TeeMemberRemoved:

  • Non-TEE removals (Admin/Member/ReadOnly) keep soft-leave semantics — no purge. Hardening to hard-purge every removal would regress the apps/scaffolding-e2e/workflows/group-{kick,leave}-* workflows that depend on the retained rows.
  • ReadOnlyTee removals trigger a hard purge. The self-detection ("did this op evict me?") is a handler concern, not part of the node-agnostic apply contract — it reads this node's stored namespace identity (per-node state). This mirrors the auto-follow architectural split.

Subgroup vs namespace-root

decide_purge_action (pure store reads) resolves the namespace owning the evicted group, confirms the evicted member is this node's own identity, and then picks one of two actions:

PurgeAction::SubgroupRemoved from one subgroup while still a member of others under the same namespace. Purge only that group's local rows (delete_group_local_rows) plus its context-index and tree-edge rows. Do not unsubscribe from the namespace gossipsub topic — other memberships still need it.
PurgeAction::NamespaceRemoved from the namespace root. Cascade the whole subtree, drop namespace-level state, then unsubscribe from the namespace topic.
PurgeAction::NoneEvent is for another member, or for a namespace this node has no identity in. No action.

What gets hard-deleted

The cascade (cascade_namespace_state) iterates the root plus every descendant group and calls delete_group_local_rows per group. That helper (crates/governance-store/src/local_state.rs) deletes the membership rows, capabilities, metadata, upgrade records, the group op-log + head, the deny-list, and — load-bearing — the private key material:

  • Signing keys. SigningKeysRepository::delete_all_for_group — the 32-byte private signing-key material. Leaking these is the actual forward-secrecy hazard, so this is the step the failure-class gating treats as security-critical.
  • Group encryption keys. GroupKeyring::delete_all_for_group — the AES group keys the node received while admitted (added in PR #2776). Without this the evicted node would retain its copy of past group keys on disk even though the rotation already orphaned them for future writes.

Forward secrecy is therefore split cleanly: future writes are protected by key rotation (re-key excluding the removed member, done by the publisher pipeline at removal); the evicted TEE node's own copies of past keys are deleted here by self-purge. The purge does not — and need not — trigger any rotation itself.

Namespace-level teardown (delete_namespace_local_state) then drops the NamespaceIdentity, the namespace governance head, and the namespace op-log. Dropping the identity and unsubscribing is gated on the signing-key purge succeeding (a best-effort dead-pointer cleanup failure does not block the security-critical finalize).

Durable marker + startup reconcile

TeeMemberRemoved fires once per eviction and an already-evicted identity receives no further removal events, so a missed or partially-failed purge has no event-driven retry. Recovery is a startup sweep, made role-safe by a durable marker (ADR 0002 / #2721):

  • Before the namespace cascade, handle_member_removed writes a durable pending-self-purge marker for the namespace. This is the one site that knows — node-aware and role-aware — that this is a confirmed TEE self-eviction of our identity.
  • On startup, reconcile_sweep enumerates markers only and completes a purge iff (marker present) AND (still no surviving namespace-root membership). If the identity is already gone, or we have been re-admitted, it clears the stale marker without purging; a read error skips (never purge on uncertainty).
  • The marker is essential because, post-eviction, the role row is erased: a role-blind scan of NamespaceIdentity rows could not distinguish evicted-TEE residue from a pending join or non-TEE soft-leave residue, and would false-purge both. The marker is the role/intent gate; the still-evicted check is the safety gate; both must hold.

Owner (stored identity, not a role)

Owner is not a GroupMemberRole variant. It is a single identity stored on group metadata: GroupMetaValue.owner_identity (crates/store/src/key/group/mod.rs). Every group has exactly one Owner, and that member is otherwise an ordinary Admin — the "Owner" status is whatever the apply handlers grant by comparing the signer's public key against owner_identity. The distinction is a small set of exclusive privileges plus immunity from involuntary removal; it is enforced by identity comparison, not by a role bit.

Storage

The struct carries both fields: the legacy admin_identity (now a fallback creator-admin marker for pre-existing groups) and the shipped owner_identity. Newly-created groups initialize owner_identity == admin_identity. Field shape and serialization are stable.

New op

enum GroupOp {
    // existing variants + MemberLeft...
    TransferOwnership { new_owner: PublicKey },
}
  • Signer must equal current owner_identity.
  • new_owner must already be a member of the group.
  • Apply: update owner_identity to new_owner. Old owner becomes a regular admin (retains admin powers; doesn't lose them as a side-effect of transfer).
  • No key rotation (membership unchanged).

Privilege matrix

ActionMemberAdminOwner
Invite members (with CAN_INVITE_MEMBERS)
Create context (with CAN_CREATE_CONTEXT)
Remove non-owner members (MANAGE_MEMBERS)
Promote member → admin
Demote admin → member1
TransferOwnership
DeleteGroup / DeleteContext / DeleteNamespace
Be involuntarily removed (MemberRemoved)✗ — OwnerImmuneFromRemoval
Self-leave via leave_group / leave_namespace✓ (last-admin permitting)✗ — OwnerCannotSelfLeave / OwnerOwnsSubgroup

1 Open question. The existing code allows admins to demote each other (mod.rs:823); restricting to Owner-only would tighten centralization. This proposal preserves the current behavior (admins can demote each other), but the doc author may want to revisit.

Owner per context

Same model. CreateContext records its creator as owner_identity on the context's metadata. The context owner can TransferContextOwnership and DeleteContext. Other than that, regular context membership is governed by the parent group as before.

Interactions with leave operations

  • leave_context: no owner constraint. Local opt-out doesn't touch governance — even an owner can mute a context locally.
  • leave_group: rejected for owner. Error fast-paths to TransferOwnership.
  • leave_namespace: rejected if signer owns the namespace OR any subgroup beneath it. Error lists every owned scope. UI may bundle these into a single multi-transfer flow.

Single-Page Reference

Op Scope DAG? Key rotation? Cascade? Reversible?
leave_context Local only None None n/a Trivial — flip flag
leave_group Single group MemberLeft No (self-leave; deferred) None Re-invite (Restricted) / re-inherit (Open)
leave_namespace Whole subtree MemberLeft at root No (self-leave; deferred) All direct-row scopes Re-invite at root + re-inherit
MemberRemoved (owner/admin) Single group (cascades contexts) MemberRemoved Yes — re-key excludes removed Context identities Re-invite (re-key delivered on add)
ReadOnlyTee removal + self-purge Subgroup or whole namespace MemberRemoved + TeeMemberRemoved event Yes (via removal) + local key delete Subtree on namespace-root None under same identity (fresh attestation)
TransferOwnership Single group/context Op None (membership unchanged) None Transfer back
DeleteGroup (Owner) Single group + descendants Op n/a (everyone gets removed) Yes None — destructive
DeleteNamespace (Owner) Whole subtree Op n/a Yes None — destructive

Implementation Status

This page documents shipped behavior. Each row points at the code that implements it.

SectionStatusCode
Membership Model RecapDocuments existing code
leave_contextImplementedcrates/context/src/handlers/leave_context.rs, auto_follow.rs gate, POST /admin-api/contexts/:id/leave
leave_groupImplementedGroupOp::MemberLeft, crates/governance-store/src/ops/group/member_left.rs, POST /admin-api/groups/:id/leave — apply enforces self-leave + direct-row + owner + last-admin. No key rotation on self-leave (deferred two-phase rotation).
leave_namespaceImplementedSame member_left.rs apply detects the no-parent case and cascades through descendant groups where the leaver has direct rows; multi-scope owner + last-admin checked upfront. No per-scope key rotation.
Key rotation on MemberRemovedImplementedcrates/governance-store/src/group_governance_publisher.rs (sign_apply_and_publish_inner) + GroupKeyring in group_keys.rs — fresh key wrapped for remaining members, excluding the removed one.
Role-scoped TEE self-purgeImplemented (ADR 0002; #2680/#2724/#2725)crates/context/src/self_purge.rs listener + cascade + marker-gated reconcile_sweep; delete_group_local_rows in governance-store/src/local_state.rs deletes signing keys (and, as of #2776, GroupKeyring encryption keys); OpEvent::TeeMemberRemoved emitted from member_removed.rs / member_left.rs.
Startup curative sweep (redrive_stranded_ops_sweep)Implemented (#2848 Part C)crates/context/src/self_purge.rs — one-shot sweep on startup after op_events subscribe; iterates namespaces via NamespaceRepository::iter_identities, finds decryptable-but-unapplied groups via NamespaceRetryService::groups_with_held_key_buffered_ops, re-drives via redrive_encrypted_ops_for_group_counted; converges up to MAX_PASSES=64 with pass-narrowing. auto_follow::spawn called before self_purge::spawn in ContextManager::started (load-bearing ordering — see note above).
Born-Open atomic subgroup create (RootOp::GroupCreated restricted field)Implemented (#2771) — WIRE BREAKcrates/primitives/src/context.rs (RootOp::GroupCreated), crates/governance-store/src/ops/group/group_created.rs (group_created::apply) — restricted: bool field added; apply writes subgroup visibility via CapabilitiesRepository::set_subgroup_visibility before queuing OpEvent::SubgroupCreated; gated on has_subgroup_visibility absence. CreateGroupRequest and REST visibility field updated. Op-adapter projection reads restricted from live op (no longer hardcodes true).
TEE subgroup admission diagnostics (handle_new_subgroup)Implementedcrates/context/src/tee_subgroup_admit.rshandle_new_subgroup now emits debug! on entry (subgroup + namespace IDs) and at each early-return path (Open subgroup skip, non-key-holder skip, no-root-TEE-members skip); warn! when a TEE member has no admission verdict in the root op-log; info! just before admitting a TEE member into a Restricted subgroup. Set RUST_LOG=calimero_context=debug (or debug on the context crate) to surface per-subgroup TEE admission decisions in logs.
ScopeProjections::namespace_current_headsImplementedcrates/context/src/scope_projections.rs — resolves a ContextGroupId to its namespace, reads the namespace DAG head, and returns parent hashes as the current-state cut argument. Returns None on unresolvable group or unreadable head; callers fall back to check_group_membership_path.
ScopeProjections::member_now_ephemeralImplementedcrates/context/src/scope_projections.rs — builds a fresh projection per call: resolves the namespace, folds the DAG via collect_namespace_ops, reads heads after the fold, and returns member_at_cut. Returns None (never decides against the member) on unresolvable namespace, store fault, or incomplete fold.
ScopeProjections::member_now_checkedImplementedcrates/context/src/scope_projections.rs — unified membership gate for all four context-layer handlers. Calls member_now_ephemeral first; falls back to MembershipRepository::is_member when projection is undecidable; emits tracing::warn with marker=unified_projection_divergence / plane=membership-query on definite disagreement. Direct MembershipRepository imports removed from get_cascade_status.rs, get_group_upgrade_status.rs, list_all_groups.rs, and list_group_contexts.rs.
ScopeProjections::scope_root_for + group_scope_root_ephemeralImplementedcrates/context/src/scope_projections.rsscope_root_for computes SHA-256(entities_root ‖ acl_hash ‖ governance_hash) for a folded scope; returns None when the scope is unfolded (callers must treat as "cannot compare", not divergence). group_scope_root_ephemeral builds a fresh ephemeral projection from the raw store, applies an ancestry-completeness guard, and delegates to scope_root_for; returns None mid-backfill. ACL component is currently empty pending C2 SetWriters routing. Pinned by scope_root_for_moves_on_hash_neutral_membership_change_and_is_none_when_unfolded.
ScopeProjections::role_at_cut_for_groupImplementedcrates/context/src/scope_projections.rs — resolves the effective GroupMemberRole at a governance cut via auth_cut_context + member_path_at_cut. Handles direct, inherited-via-admin (Admin), and inherited-via-anchor (anchor role row) cases. Returns None on incomplete ancestry, absent member, unresolvable namespace, or missing anchor row. Prevents spurious membership-role divergence reports by abstaining rather than guessing on missing anchor rows.
UnifiedApplier + pub mod unified_applierImplementedcrates/context/src/unified_applier.rs (exported via crates/context/src/lib.rs) — UnifiedApplier wraps Arc<std::sync::RwLock<ScopeProjections>> and implements DeltaApplier<Op>; apply holds the write lock only across the synchronous ScopeProjections::ingest_op call, never across an .await; lock poison is recovered. new() for standalone/test use; with_projection(arc) as the C2.1 dual-write seam; projection() accessor exposes the Arc. Convergence pinned by dag_releases_out_of_order_ops_and_the_projection_converges (four-op causal chain, four delivery orders, all converge to the same scope_root).
unified_op_storepersist_op + load_scope_opsImplemented (no production call sites yet)crates/context/src/unified_op_store.rs (exported as pub mod unified_op_store from crates/context/src/lib.rs) — persist_op Borsh-serializes an Op and writes it under ScopeUnifiedOpKey in Column::UnifiedOp; idempotent because the key is content-addressed by op id. load_scope_ops performs a seek-then-entries range scan over the scope prefix and deserializes all rows back into Vec<Op>. Rows are returned in key order, not causal order — callers must replay the loaded ops through DagStore<Op> + UnifiedApplier to recover causal ordering and rebuild projections; heads are deliberately not persisted. No production apply or sync path calls these functions yet. Behavior is pinned by persisted_op_log_reloads_and_replays_to_the_same_projection: persists a three-op causal chain (AdminChanged → MemberAdded → SetWriters), reloads from a fresh store handle, replays through DagStore<Op> + UnifiedApplier, and asserts the reconstructed scope_root matches an independent in-memory fold; also asserts loading an empty scope returns zero rows.
ScopeProjections::check_op_store_completenessImplemented (diagnostic-only observe gate; retires at C3 Stage 4)crates/context/src/scope_projections.rs — accepts an already-computed gov-DAG fold (dag_ops) and calls load_scope_ops to compare op-id sets. Emits op_store_incomplete warn on gaps; emits op_store_gate_unavailable and returns None when the store is unreadable. Returns None (unverifiable), Some(vec![]) (verified complete), or Some(ids) (missing ids). Pinned by completeness_gate_flags_governance_ops_missing_from_the_op_store in crates/context/tests/op_store_reconstruction.rs.
ScopeProjections::scope_root_from_op_store + ScopeProjections::shadow_compare_op_storeImplemented (observe-only; C2.2b gate)crates/context/src/scope_projections.rsscope_root_from_op_store loads all persisted ops for a given ScopeId via load_scope_ops, folds them into a fresh ScopeProjections instance, and returns the governance scope_root using a fixed zero entities_root. Returns Ok(None) when no ops exist for the scope; Err on store faults. This is the op-store's independent view of a scope's governance root, intended for completeness verification before C2.2b. shadow_compare_op_store is an observe-only method that compares the maintained (governance-DAG) projection's scope_root against the op-store reconstruction for a given namespace. On mismatch it logs a tracing::warn with marker=unified_op_store_divergence and truncated hex of both roots — this log marker is the e2e gate signal for C2.2b readiness. Silently skips (no divergence reported) when the scope is unfolded, the op-store is empty for it, or the store read fails. calimero_op/ScopeId types remain entirely inside the context crate and are not leaked into the node crate. Behavior is pinned by two additional assertions in the existing persisted_op_log_reloads_and_replays_to_the_same_projection test: one verifying scope_root_from_op_store equals the expected in-memory fold result, and one verifying it returns None for a scope that has no persisted ops.
AtCutAuthorizer trait + LiveFallbackAuthorizer + EphemeralProjectionAuthorizerImplementedcrates/context/src/governance_dag.rsAtCutAuthorizer declares four required methods: is_admin_at_cut, is_admin_or_capability_at_cut, is_last_admin_at_cut, and membership_path_at_cut. All must return None when parents is empty (trait-level empty-cut contract). LiveFallbackAuthorizer implements all four returning None. EphemeralProjectionAuthorizer holds a Mutex<Option<(ContextGroupId, Arc<FoldedProjection>)>> fold cache; the private folded helper reuses a cached fold for the same group or calls ScopeProjections::ephemeral_projection on a miss. All four at-cut methods delegate through folded; each opens with an empty-parents guard. is_last_admin_at_cut delegates to ScopeProjections::is_last_admin_at_cut (in crates/context/src/scope_projections.rs), which mirrors live last-admin logic at the parent cut. membership_path_at_cut delegates to ScopeProjections::membership_path_at_cut, which calls member_path_at_cut on the folded projection via auth_cut_context and maps the result to AtCutMembershipPath. The standalone GroupGovernanceApplier::apply is unchanged (live-only); see the GroupGovernanceApplier note in the at-cut accessors section above.
Owner identity + TransferOwnershipImplementedGroupMetaValue.owner_identity, GroupOp::TransferOwnership (ops/group/transfer_ownership.rs), OwnerImmuneFromRemoval / OwnerCannotSelfLeave gates in MemberRemoved / MemberLeft.
Namespace genesis op (RootOp::NamespaceCreated)Implementedcrates/context/src/handlers/create_group.rs (parent_group_id == None branch) — signs, applies, and publishes RootOp::NamespaceCreated { founder: admin_identity } via sign_apply_and_publish_namespace_op; NoPeersSubscribed from publish treated as success; any other error triggers rollback of all locally-written root rows (meta, founder Admin member row, default caps, group key, signing key, optional name metadata) via GroupKeyring::delete_key_by_id, SigningKeysRepository::delete_key, MetaRepository::delete, MembershipRepository::remove_member, CapabilitiesRepository::delete_default, MetadataRepository::delete_group; namespace identity row deliberately excluded from rollback so retries reuse the same identity/founder; DAG head is head-atomic and does not require rollback.

Startup curative sweep — redrive_stranded_ops_sweep

At node startup, a one-shot curative sweep re-drives any encrypted ops that are buffered but decryptable — that is, the node holds the group key but the op was never applied, typically because the GroupCreated event that would have triggered delivery arrived late or was dropped. Without this sweep such ops remain stranded forever.

Where it runs

redrive_stranded_ops_sweep is called from self_purge::run, after subscribing to op_events and before entering the main event loop. It blocks the event loop until the sweep finishes, but errors are logged and swallowed — the sweep never prevents startup.

Algorithm

  1. Call known_namespace_identities (via NamespaceRepository::iter_identities) to enumerate every namespace this node has an identity for.
  2. For each namespace, call NamespaceRetryService::groups_with_held_key_buffered_ops (the inverse of groups_awaiting_key) to find groups whose key is already held but which still have buffered ops.
  3. Re-drive each group via redrive_encrypted_ops_for_group_counted, which applies as many buffered ops as possible and returns a count of newly applied ops.
  4. Loop until a full pass applies nothing new (cross-signer convergence), bounded by MAX_PASSES = 64. Pass-narrowing means only groups that made progress in the previous pass are revisited, avoiding redundant work.

Spawn ordering — load-bearing

ContextManager::started calls auto_follow::spawn before self_purge::spawn. This ordering is intentional and load-bearing: auto_follow::spawn subscribes to op_events synchronously on the caller thread before spawning its async task, and passes the receiver into auto_follow::run(). Because the curative sweep (inside self_purge::run) may emit OpEvent::ContextRegistered for previously stranded contexts, the auto-follow receiver must exist before those events are broadcast — otherwise they are silently dropped by the broadcast channel. Reversing the spawn order would silently break auto-follow for any context recovered by the startup sweep.

Reviewers: both call sites in ContextManager::started carry inline comments documenting the ordering constraint. Any refactor that reorders auto_follow::spawn and self_purge::spawn must preserve the subscribe-before-sweep invariant.

Relevant symbols

  • redrive_stranded_ops_sweep — orchestrator; lives in crates/context/src/self_purge.rs
  • redrive_buffered_ops_for_group — inner per-group apply loop
  • redrive_encrypted_ops_for_group_counted — wraps the above and returns a count of newly applied ops for pass-narrowing
  • namespace_groups_with_held_key_buffered_ops — store query (inverse of groups_awaiting_key)
  • NamespaceRetryService::groups_with_held_key_buffered_ops
  • NamespaceRepository::iter_identities

Born-Open atomic subgroup create — RootOp::GroupCreated wire change

⚠ WIRE BREAK. RootOp::GroupCreated has a new Borsh field (restricted: bool). Nodes running this version cannot exchange governance ops with nodes running the previous wire format. Nodes must reset their governance op-log before upgrading.

Problem

Previously, the GroupCreated op carried no visibility information. The tee_subgroup_admit subscriber fired on every OpEvent::SubgroupCreated and wrote a transient direct ReadOnlyTee membership row for Open subgroups — a row that would need to be retracted once a SubgroupVisibilitySet(Open) op arrived. This transient row could cause spurious TEE admission flows.

Fix — restricted field on GroupCreated

RootOp::GroupCreated gains a restricted: bool field:

  • restricted: true (default) — subgroup is born Restricted. Legacy behavior is preserved; no wire difference for groups created this way.
  • restricted: false — subgroup is born Open. The apply handler writes the subgroup's visibility key via CapabilitiesRepository::set_subgroup_visibility during apply, before OpEvent::SubgroupCreated is queued. tee_subgroup_admit sees the group as already Open when it reacts, skips it, and no transient direct row is written.

The write is gated on CapabilitiesRepository::has_subgroup_visibility absence so later SubgroupVisibilitySet flips are never clobbered.

API surface changes

  • CreateGroupRequest gains a restricted: bool field (default true).
  • The create-group-in-namespace REST endpoint accepts an optional visibility field ("open" | "restricted", default "restricted").
  • The op-adapter projection reads restricted from the live op instead of hardcoding true.

Relevant symbols

  • RootOp::GroupCreatedcrates/primitives/src/context.rs
  • group_created::applycrates/governance-store/src/ops/group/group_created.rs
  • CapabilitiesRepository::has_subgroup_visibility, CapabilitiesRepository::set_subgroup_visibility
  • CreateGroupRequest, payload_from_root_op

Open Questions & Follow-ups

  1. Two-phase self-leave rotation. MemberLeft currently does not re-key (the leaver can't generate a key without retaining it). A deferred design has a remaining admin's apply hook publish the new key after the leave. Until it lands, owner-initiated MemberRemoved is the only rotating leave.
  2. Local cleanup is role-scoped, not a toggle. The earlier soft-vs-hard default question is resolved by role (ADR 0002): non-TEE removals stay soft; ReadOnlyTee removals hard-purge. No global default knob.
  3. Admin-mutual demote. Admins can demote each other. Whether to restrict this to Owner-only is unresolved; current behavior is preserved.
  4. Old owner after TransferOwnership. Becomes a regular admin (current behavior) or just a member (alternative)?
  5. Subgroup-level self-purge reconcile. The startup reconcile is namespace-level only; a subgroup-only purge has no reconcile retry surface yet (tracked in #2726).
  6. Periodic reconcile. A purely-lagged-drop of TeeMemberRemoved writes no marker and is not recovered until a future eviction; a periodic sweep is an open follow-up.

EphemeralProjectionAuthorizer — empty-parents early return

Both is_admin_at_cut and is_admin_or_capability_at_cut on EphemeralProjectionAuthorizer now return None immediately when the supplied parents slice is empty, deferring unconditionally to the live fallback authorizer. Previously, an empty parents slice was passed through to the fold machinery, which evaluated the query against a genesis-state view. That view contains no granted roles or capabilities, so any capability holder whose grant was established after genesis would be falsely rejected by the projection even though the live store correctly recognises them.

When parents is empty

Two cases produce an empty parents slice at the authorizer call site:

  • Genesis op. The very first op in a namespace's governance log has no predecessor hashes. The causal cut is genuinely empty because no prior ops exist.
  • PermissionChecker constructed without apply-auth context. Some code paths build a PermissionChecker outside of a normal apply flow and pass an empty parents field as a zero value. These callers have no meaningful causal position to resolve against.

In both cases, judging against an empty fold is semantically wrong: the fold has not processed any MemberAdded, MemberRoleSet, or DefaultCapabilitiesSet ops, so the resulting AclView is empty. A query against it would return Some(false) for every capability check, silently blocking operations that the live store would correctly permit.

New behavior

  • If parents.is_empty(), both methods return None immediately, before touching the fold cache or calling ScopeProjections::ephemeral_projection.
  • The None return propagates to the caller as "projection is undecidable," which causes the standard live-fallback path to take effect. No warning is emitted — empty parents is a legitimate undecidable case, not a divergence.
  • When parents is non-empty, behavior is unchanged: the fold cache is consulted (or populated on a miss) and the at-cut check proceeds normally.

Contract clarification

The AtCutAuthorizer contract already specifies that None means "undecidable — defer to the live gate." This change makes the empty-parents case an explicit instance of that contract rather than a silent fall-through to a semantically incorrect verdict. Callers must not treat None as a denial; the live authorizer is always consulted when the projection abstains.

The None-on-incomplete-ancestry guard (from cut_ancestry_complete) is distinct and remains in place: it fires when parents is non-empty but the fold has not yet processed all ops reachable from those parents. The empty-parents guard fires earlier, before the fold is attempted at all.

Affected code

  • crates/context/src/governance_dag.rsEphemeralProjectionAuthorizer::is_admin_at_cut and EphemeralProjectionAuthorizer::is_admin_or_capability_at_cut: both now open with if parents.is_empty() { return None; } before the folded(group_id) call.
  • No changes to ScopeProjections or any call sites — the early return is entirely internal to the authorizer methods. LiveFallbackAuthorizer is unaffected (it already returns None unconditionally for all methods, including the new is_last_admin_at_cut).

Deferred op-store read-flip — ops_for_namespace indirection layer

Status: C2.2b deferred. The unified op-store (load_scope_ops) is not yet the source of truth for projection rebuilds. All projection-rebuild call sites now route through a new stable indirection method, ScopeProjections::ops_for_namespace, which currently delegates to collect_namespace_ops (the governance-DAG walk). The flip to load_scope_ops is explicitly deferred until the late-decrypted membership bug described below is resolved.

ops_for_namespace — the stable indirection point

ScopeProjections::ops_for_namespace(namespace_id, store) is the single dispatch point used by every projection-rebuild site:

  • backfill_namespace in scope_projections.rs
  • member_now_ephemeral / ephemeral_fold in scope_projections.rs
  • The scope_root reconstruction path in scope_projections.rs
  • refresh_projection_for_cut in the node crate

All four previously called collect_namespace_ops directly. They now call ops_for_namespace instead. collect_namespace_ops remains as the underlying implementation and is unchanged; ops_for_namespace is a thin wrapper that will be flipped to call load_scope_ops at C2.2b without requiring changes at any call site.

Why the flip was blocked — late-decrypted membership (resolved by C2.2c)

An encrypted MemberAdded op can arrive at a node before that node holds the group key. At apply time the op is processed as Noop and written to the op-store with a Noop payload. When the group key later arrives — via a keyshare or a KeyDelivery event — the governance-DAG walk (collect_namespace_ops) re-decrypts the op at read time and correctly surfaces it as MemberAdded. Previously, the op-store entry was not updated; it still recorded Noop.

Consequence: if ops_for_namespace were flipped to load_scope_ops before this was fixed, any member whose MemberAdded op was frozen as Noop at write time would silently disappear from the projection after a rebuild. This was confirmed by e2e timeouts on the joiner's post-write sync in the group-3node, scaffolding-e2e, and shared-storage rotation workflows.

Fix: ScopeProjections::repersist_namespace_ops. After any group key is delivered, the key-delivery path calls ScopeProjections::repersist_namespace_ops(namespace_id, store), which re-walks the namespace through collect_namespace_ops (the canonical read-time decryption path) and calls persist_op for every op it finds. Because ops are content-addressed by the same op id, a now-decryptable op (e.g. MemberAdded) overwrites the stale Noop that was frozen at apply time when the key was absent. The method is best-effort (errors are logged but not propagated), idempotent, and bounded by the same MAX_BACKFILL_OPS walk cap used elsewhere.

ScopeProjections::check_op_store_completeness — diagnostic completeness gate

ScopeProjections::check_op_store_completeness(dag_ops, store) is a diagnostic-only observe gate that compares the op-ids from an already-computed gov-DAG fold (dag_ops) against the op-ids loaded from the unified op-store via load_scope_ops. For every op present in the gov-DAG fold but absent from the op-store it emits an op_store_incomplete tracing::warn event carrying missing_count, dag_count, op_store_count, and a hex-encoded sample_missing id. If the op-store cannot be loaded at all it emits a distinct op_store_gate_unavailable marker and returns None so a store fault cannot masquerade as a verified-clean result.

Return semantics:

  • None — op-store was unreadable; completeness is unverified. Callers must not treat this as a clean result.
  • Some(vec![]) — verified complete; every gov-DAG op-id is present in the op-store.
  • Some(ids) — those op-ids are present in the gov-DAG fold but missing from the op-store.

This method is diagnostic-only and has no effect on whether ops are accepted or rejected. It retires at C3 Stage 4 when the authoritative read path flips onto the op-store. The companion regression test completeness_gate_flags_governance_ops_missing_from_the_op_store in crates/context/tests/op_store_reconstruction.rs builds three governance ops, persists only two to the op-store, calls check_op_store_completeness with all three as the dag-fold, and asserts the return is Some(vec![op3.id]); after persisting the third it asserts the return becomes Some(vec![]), confirming the None vs empty-vec distinction works correctly.

Removed: shadow_compare_op_store and scope_root_from_governance_dag

The observe-only shadow_compare_op_store method on ScopeProjections and its companion scope_root_from_governance_dag have been removed, along with the call site in refresh_projection_for_cut that triggered a full governance-DAG fold on every backfill.

The comparison these methods performed was structurally flawed: the maintained projection holds live ingest_op-fed ops that may not yet be written to the op-store. A gap in the op-store could therefore be masked by the live apply path, making the divergence invisible to the cross-check. The marker=unified_op_store_divergence log signal described in earlier sections of this page is no longer emitted; the C2.2b readiness gate is now the op_store_reconstruction_matches_governance_dag_for_late_decrypted_membership regression test described below.

Regression test — op_store_reconstruction.rs

The integration test file crates/context/tests/op_store_reconstruction.rs is the CI regression gate for this fix. The test is named op_store_reconstruction_recovers_late_decrypted_membership_after_key_delivery and is structured as an explicit before/after:

  1. Injects an encrypted MemberAdded op into the governance DAG with the group key absent at write time, persisting the op-store entry as Noop.
  2. Before re-persist: asserts that load_scope_ops reconstruction sees Some(false) for the member (the frozen Noop state).
  3. Calls repersist_namespace_ops to simulate key delivery.
  4. After re-persist: asserts that the op-store reconstruction matches the governance-DAG fold — the member is now Some(true).

The test runs in CI without #[ignore] and serves as the C2.2c readiness gate. The module-level doc comment in the test file explains the full bug chain and the fix.

Summary of changes

  • ScopeProjections::ops_for_namespace introduced; all four projection-rebuild call sites updated from collect_namespace_ops to ops_for_namespace.
  • collect_namespace_ops unchanged — still the implementation behind ops_for_namespace.
  • shadow_compare_op_store and scope_root_from_governance_dag removed from ScopeProjections; marker=unified_op_store_divergence is no longer emitted.
  • refresh_projection_for_cut call site that triggered a shadow fold on every backfill deleted.
  • ScopeProjections::repersist_namespace_ops added; called by the key-delivery path after a group key arrives to overwrite stale Noop op-store entries with their now-decryptable payloads. Best-effort, idempotent, bounded by MAX_BACKFILL_OPS.
  • crates/context/tests/op_store_reconstruction.rs updated: test renamed to op_store_reconstruction_recovers_late_decrypted_membership_after_key_delivery, #[ignore] removed, body restructured into an explicit before/after assertion around a repersist_namespace_ops call.

ScopeProjections::persist_namespace_head_ops — mirroring authored ops into the op-store

After a local authoring step publishes a governance op (GroupCreated, GroupDeleted, MemberJoinedOpen), the unified op-store must be updated so that any subsequent load_scope_ops-based projection rebuild sees the authored op without relying on a peer retransmission. The method ScopeProjections::persist_namespace_head_ops(store, namespace_id) is the single call site for this update.

Why not a heads-only walk

A naive implementation would read the namespace's current DAG heads immediately after publish and persist only those ops. This races: if a concurrent peer op is applied between the local publish call and the heads read, the head record advances past the just-authored op. The authored op is then unreachable from a heads-only walk and is silently omitted from the op-store. The member or group it describes disappears from any rebuild that sources ops from load_scope_ops.

Implementation — full gov-DAG fold via repersist_namespace_ops

persist_namespace_head_ops delegates unconditionally to repersist_namespace_ops, which performs a full collect_namespace_ops-backed fold bounded by MAX_BACKFILL_OPS. This walk is read-time decryption-aware, so an authored op that was encrypted at publish time is decoded here and written to the op-store with its actual payload (not as Noop). The walk captures the authored op regardless of where it landed in the DAG relative to concurrent peer ops, because collect_namespace_ops traverses the full ancestry from the current head rather than just the head itself.

The call is:

  • Idempotent. persist_op is content-addressed by op id; calling it twice for the same op is a no-op.
  • Best-effort. Errors are logged by repersist_namespace_ops (using the same tracing::warn paths as the key-delivery re-persist) and are never propagated. A failure to update the op-store never causes the authoring step to return an error to the caller.
  • Bounded. The walk is capped at MAX_BACKFILL_OPS, the same limit used by every other collect_namespace_ops-backed path. An unusually large namespace cannot cause an unbounded fold at authoring time.

Unified authoring wrapper — sign_apply_and_publish_group_op

Rather than calling persist_namespace_head_ops individually in each handler, all group-op authoring is routed through a single crate-internal async wrapper: crate::sign_apply_and_publish_group_op. It calls calimero_governance_store::sign_apply_and_publish and, on success, resolves the namespace from the group ID via NamespaceRepository::resolve and calls ScopeProjections::persist_namespace_head_ops. A namespace-resolve failure logs a tracing::warn identifying the group ID but does not fail the authoring operation.

Callers must not call calimero_governance_store::sign_apply_and_publish directly for group ops. Using the wrapper ensures the op-store is always updated at the single chokepoint and that namespace-resolve failures are observable via tracing warnings.

Wired call sites

The method is called after the governance op is successfully published or applied, never before. A failure at the authoring step is not affected:

  • create_group — called after GroupCreated is successfully reported to the governance pipeline.
  • delete_group — called after GroupDeleted is applied.
  • join_context — called after the MemberJoinedOpen op is published. The call is skipped (effectively a no-op path) if the publish or apply step errored.
  • join_subgroup_inheritance — called after MemberJoinedOpen is published, following the same skip-on-error pattern.
  • All other group-op handlersadd_group_members, admit_tee_node, create_context (two call sites), delete_context, detach_context_from_group, leave_group, leave_namespace, set_member_auto_follow, set_member_capabilities, set_subgroup_visibility, set_tee_admission_policy, update_member_role, upgrade_group (four call sites), dispatch_cascade, and the generic governance handler in lib.rs all call crate::sign_apply_and_publish_group_op instead of the underlying store function.

Exception — remove_group_members. The key-rotation removal path uses sign_apply_and_publish_removal (not the standard publish function) and cannot use the wrapper. After each removal it inlines the same namespace-resolve + persist_namespace_head_ops logic directly. A namespace-resolve failure logs a tracing::warn identifying the group ID.

Regression test

The test persist_namespace_head_ops_lands_locally_authored_op_in_the_op_store in crates/context/tests/op_store_reconstruction.rs verifies the full round-trip:

  1. An encrypted MemberAdded op is injected into the governance DAG (op-log + DAG head) but not into the op-store, simulating the local-author gap.
  2. The test asserts that load_scope_ops returns no entry for the op before the call — confirming the gap exists.
  3. persist_namespace_head_ops is called.
  4. The test asserts the op now appears in the op-store.
  5. A fresh projection is folded from the op-store contents and member_at_cut is called. The test asserts it returns Some(true) — proving the encrypted payload was decoded correctly, not merely that the op id was recorded.

The test runs without #[ignore] in CI and is the readiness gate for any future flip of ops_for_namespace to source from load_scope_ops at authoring call sites.

Relationship to repersist_namespace_ops

repersist_namespace_ops was introduced to recover late-decrypted MemberAdded ops after key delivery (see the deferred op-store read-flip section above). persist_namespace_head_ops reuses that same mechanism for a different trigger — local authoring — ensuring both the key-delivery path and the authoring path produce a consistent op-store state. The two call sites share the same underlying fold, the same error handling, and the same MAX_BACKFILL_OPS bound.

Atomic op-store write — removal of per-site dual-write scaffolding

Status: shipped. store_signed_operation now persists every governance op atomically at apply time, making all per-site op-store write calls redundant. This section documents what was removed and why, and supersedes the Deferred op-store read-flip and persist_namespace_head_ops sections above where they describe the old per-site scaffolding.

What was removed

  • Receive-handler dual-write. The persist_op call inside apply_signed_namespace_op that was labelled C2.1b dual-write has been deleted. The ingest_op fold into the live projection is deliberately kept; only the now-redundant durable-store write is gone.
  • sign_apply_and_publish_group_op wrapper. The crate-private function in lib.rs that wrapped calimero_governance_store::sign_apply_and_publish and appended a persist_namespace_head_ops call is deleted entirely. All ~15 call sites (add_group_members, admit_tee_node, create_context, delete_context, detach_context_from_group, join_context, leave_group, leave_namespace, set_member_auto_follow, set_member_capabilities, set_subgroup_visibility, set_tee_admission_policy, update_member_role, upgrade_group, dispatch_cascade, and others) now call calimero_governance_store::sign_apply_and_publish directly.
  • ScopeProjections::persist_namespace_head_ops. The thin alias for repersist_namespace_ops that documented the C3 Stage 1/2/3 per-site persist pattern is deleted from scope_projections.rs. All call sites are also removed: create_group, delete_group, join_context, join_subgroup_inheritance, remove_group_members, join_namespace (node crate), reparent_group (admin API), and create_group_in_namespace (admin API).

Why the per-site scaffolding was safe to delete

store_signed_operation is called on every apply path — both local authoring and peer-receive — and now writes the op into the unified op-store as part of the same atomic commit. Because the persist is guaranteed at the apply chokepoint, no handler needs to issue a separate post-publish persist call. The two problems the scaffolding addressed are resolved at source:

  • Local-author gap. A locally authored op is written by store_signed_operation at apply time, before the handler returns. There is no window in which the op could be absent from the op-store after a successful publish.
  • Late-decrypted membership. repersist_namespace_ops (called by the key-delivery path) still overwrites stale Noop entries with their now-decryptable payloads. The atomic write path records the best payload available at apply time; the key-delivery re-persist corrects any entry frozen as Noop before the group key arrived. Both paths remain in place; only the redundant per-site calls are removed.

Race that the old heads-only walk would have introduced

The deleted persist_namespace_head_ops performed a full collect_namespace_ops-backed fold after publish to avoid a heads-only-walk race (a concurrent peer op advancing the head past the just-authored op before the walk ran). The atomic write in store_signed_operation sidesteps this entirely: the authored op is written at apply time under its content-addressed op id, so its presence in the op-store is independent of where the DAG head points afterward.

Surviving use of repersist_namespace_ops

repersist_namespace_ops itself is not deleted. It continues to serve the key-delivery path: when a group key arrives late, the key-delivery handler calls repersist_namespace_ops to re-walk the namespace via collect_namespace_ops and overwrite any op-store entries that were frozen as Noop at apply time because the key was absent. This is the only remaining call site.

Updated test — locally_authored_op_lands_in_the_op_store_atomically

The test previously named persist_namespace_head_ops_lands_locally_authored_op_in_the_op_store is renamed locally_authored_op_lands_in_the_op_store_atomically. Its structure is updated to reflect the atomic-write path:

  1. An encrypted MemberAdded op is passed through store_signed_operation (the atomic path).
  2. The test asserts the op appears in the op-store immediately after that call, with no intervening persist_namespace_head_ops call required — authoring handlers need only call calimero_governance_store::sign_apply_and_publish directly; no post-author mirror step is needed.
  3. The test then calls ScopeProjections::repersist_namespace_ops directly (replacing the now-deleted persist_namespace_head_ops alias) and asserts the op-store state is unchanged — confirming idempotency over an already-present entry.

The test runs without #[ignore] in CI and is the readiness gate confirming that the atomic-write path is sufficient and that the per-site scaffolding carried no load that store_signed_operation does not now cover.