Membership & Leave Operations
Voluntary leave, owner-initiated removal, and role-scoped TEE self-purge across namespaces, groups, and contexts
As-built page. This was originally a proposal; it now documents shipped behavior. For the
fleet-specific eviction story (owner kicks a ReadOnlyTee node on HA-disable, the
self-purge listener, and the marker-gated startup reconcile) see
Fleet HA. Term definitions live in the
Glossary. The governing design record is
docs/adr/0002-fleet-tee-leave-protocol.md.
Problem this solved
Originally, the only way out of a Calimero group or namespace was to be removed by an admin —
MemberRemoved was admin-initiated and there was no self-leave op. Users could not
voluntarily exit a workspace they joined; muting a context they were auto-followed into required
client-side filtering rather than a real opt-out, because auto-follow re-adds them on the next
ContextRegistered event.
Separately, every group records a single "creator / fallback admin" identity
(GroupMetaValue.owner_identity) but the role enum did not distinguish that creator
from any other admin. There was no clean way to express "this person owns the group, others
administrate it."
The shipped design adds three leave operations and an owner identity stored on group metadata that
gates self-leave and involuntary removal. The pieces share infrastructure: the stored owner blocks
leave_group/leave_namespace and involuntary MemberRemoved;
owner-initiated MemberRemoved reuses the group key-rotation pipeline;
leave_namespace is the namespace-root MemberLeft plus a cascade through
descendant groups.
A separate, role-scoped mechanism layers on top for fleet TEE nodes: when a
ReadOnlyTee member is removed, the evicted node hard-deletes its local key
material (see TEE self-purge), because a TEE node has no rejoin path
under the same identity. Non-TEE removals keep the soft-leave invariant.
Membership Model Recap
Background on the data shapes the leave operations operate over. Names match the code as it exists today.
Hierarchy
- Namespace. A
ContextGroupIdwith no parent. The trust anchor for everything beneath it. There is no separateNamespacetype — a namespace is a root group. - Group / subgroup. A
ContextGroupwith aparent_group_id. Forms a strict tree under a single namespace. - Context. A unit of replicated state. Belongs to exactly one group via
ContextGroupRef. Members of the group can join, sync, and write to it.
Direct vs inherited membership
Group membership is computed by check_group_membership_path
(crates/context/src/group_store/membership.rs:217-291):
- Direct membership = a stored
(member, group)row exists. Created byadd_group_members(explicit invitation). Required for Restricted groups; possible but redundant for Open subgroups. - Inherited membership = no row exists; the walker concludes membership by walking the
parent chain through Open subgroups when the member holds the
CAN_JOIN_OPEN_SUBGROUPScapability in the parent.
This distinction is critical for leave operations. Leave deletes a direct row. If the only path to membership is inheritance, there is no row to delete — the member must leave whichever ancestor provides the inheritance.
In the governance-DAG projection layer, this distinction is reflected by the fold rule for
RootOp::MemberJoinedOpen: it folds to OpPayload::Noop rather than
OpPayload::MemberAdded. No persistent GroupMember node is written for
an open-subgroup inheritance join; instead, AclView::is_member_at_cut re-derives
membership at query time via the anchor-root membership + subgroup tree + visibility +
CAN_JOIN_OPEN_SUBGROUPS capability walk — exactly mirroring the live store. This
means inherited access is automatically revoked when the member is removed from the root group
and restored if they rejoin, with no over-grant surviving root-anchor removal.
In the governance-DAG projection layer, this distinction is reflected by the fold rule for
RootOp::MemberJoinedOpen: it folds to OpPayload::Noop rather than
OpPayload::MemberAdded. No persistent GroupMember node is written for
an open-subgroup inheritance join; instead, AclView::is_member_at_cut re-derives
membership at query time via the anchor-root membership + subgroup tree + visibility +
CAN_JOIN_OPEN_SUBGROUPS capability walk — exactly mirroring the live store. This
means inherited access is automatically revoked when the member is removed from the root group
and restored if they rejoin, with no over-grant surviving root-anchor removal.
This behavior is pinned by three assertions in
projection_membership_equivalence.rs: (1)
projection_matches_live_across_inherited_join_and_root_removal asserts that after
an inherited join, member_at_cut_authoritative returns Some(true);
(2) the same test asserts that after a root removal that the live store rejects,
member_at_cut_authoritative must not return Some(true) — the
over-grant guard; (3)
projection_matches_live_across_leave_and_rejoin_inheritance asserts that after a
full leave-and-rejoin cycle, member_at_cut_authoritative correctly restores access.
Together these tests ensure the authoritative grant resolver agrees with the live store across
all three scenarios and never over-grants on root-anchor removal.
Visibility (Open vs Restricted)
A per-group flag (VisibilityMode::{Open, Restricted} in
crates/context/config/src/lib.rs:134-137) that controls whether the parent-walk
inherits into this subgroup or stops cold:
- Open: parent members with
CAN_JOIN_OPEN_SUBGROUPSinherit membership. - Restricted: walks halt; only direct invitations admit members.
At leave-time, visibility doesn't change behavior — it only affects whether a row exists in the first place. Most members of Open subgroups have no row (inherited); most members of Restricted subgroups have one (direct invite).
Roles & capabilities
The GroupMemberRole enum (crates/primitives/src/context.rs) has exactly
four variants:
Admin,Member,ReadOnly,ReadOnlyTee.- There is no
Ownerrole variant. "Owner" is not a member role — it is a single identity stored on group metadata (GroupMetaValue.owner_identity). The owner is whichever member's public key equals that field; they are otherwise an ordinaryAdmin. Owner status is enforced by identity comparison in the apply handlers (OwnerImmuneFromRemoval,OwnerCannotSelfLeave), not by a role check. See the Owner section.
Per-member capability bits (crates/context/config/src/lib.rs, MemberCapabilities):
CAN_CREATE_CONTEXT,CAN_INVITE_MEMBERS,CAN_JOIN_OPEN_SUBGROUPS,MANAGE_MEMBERS,MANAGE_APPLICATION,CAN_CREATE_SUBGROUP,CAN_DELETE_SUBGROUP,CAN_MANAGE_VISIBILITY,CAN_MANAGE_METADATA.
Existing protections
- Last-admin protection (
membership_policy.rs:39-46):MemberRemovedrejects withLastAdminif it would leave a group with zero admins. Also enforced on demotion. - Key rotation on removal (
group_governance_publisher.rs,sign_apply_and_publish_inner): publishingMemberRemovedautomatically generates a fresh group key viaGroupKeyring, wraps it (wrap_for_member) for every remaining member — excluding the removed one — and delivers it. The removed member never receives the new key, so it cannot decrypt anything written after eviction. This is the forward-secrecy mechanism for the namespace's future writes. Owner-initiated removal is the cryptographically-complete leave path. - Namespace genesis op (
create_group.rs,parent_group_id == Nonebranch): when a namespace root is created, the handler signs, applies, and publishes aRootOp::NamespaceCreated { founder: admin_identity }genesis op viasign_apply_and_publish_namespace_op. ANoPeersSubscribederror from publish is treated as success (apply succeeded locally; publish is best-effort). Any other genesis-apply error triggers a full rollback of all locally-written root rows — meta, founder Admin member row, default caps, group key, and signing key (plus optional name metadata) — via individual idempotent deletes (GroupKeyring::delete_key_by_id,SigningKeysRepository::delete_key,MetaRepository::delete,MembershipRepository::remove_member,CapabilitiesRepository::delete_default,MetadataRepository::delete_group) before returningErr. The namespace identity row is deliberately not rolled back so a retry obtains the same identity/founder. The DAG head does not require rollback becauseapply_signed_opis head-atomic (the head advances only after a successful apply).
Forward-only invariant — load-bearing
The apply-time membership check is a function of two things only: the signer's identity
and the governance-DAG position the signer signed against. It is not a function
of the receiver's current local state. This is the forward-only property: a write authored
at governance position P stays valid forever, even if the receiver has since applied
a MemberRemoved for the signer at a later position P' in its local DAG.
Why this matters
Without forward-only, two peers receiving the same set of ops in different orders would end up with different state:
- Peer A receives
MemberRemovedfirst, then the pre-removal delta. If "is the signer currently a member?" gates the apply, the delta is rejected. - Peer B receives the delta first, then
MemberRemoved. The delta applies; the later removal doesn't retroactively undo it.
That divergence cascades: any subsequent delta that reads the (different) state diverges further, context root hashes drift apart, and the network has no honest convergence point. Forward-only removes the receive-order dependence: pre-removal writes always apply on every peer, regardless of when the receiver learns about the removal.
How it's implemented
Every state delta carries a governance_position field that names the governance-DAG
heads the author observed at sign time. On receive, the apply path evaluates membership at the
signed heads via the unified projection — which resolves the ancestry of those heads and derives
the membership state from the folded projection. The resolution visits
only the prefix that was already final when the delta was signed; anything causally after
the signed cut (including a later MemberRemoved for this signer) is invisible to
the resolution and cannot retroactively invalidate the delta.
The namespace-level applier (NamespaceGovernanceApplier in
crates/context/src/governance_dag.rs) threads this cut into the apply path directly:
DeltaApplier<SignedNamespaceOp>::apply calls
apply_signed_namespace_op_at_cut(&delta.parents, &LIVE_FALLBACK_AUTHORIZER)
rather than the cut-free apply_signed_namespace_op. Passing delta.parents
as the causal cut ensures membership is evaluated at the position the author signed against.
LIVE_FALLBACK_AUTHORIZER is a placeholder authorizer that delegates to the live store;
membership authorization for all other paths is now handled entirely by the unified projection
(EphemeralProjectionAuthorizer / ScopeProjections) — the
acl_view_at, MembershipStatus, prefix_walk_membership,
MembershipTransition, heads_equal, and MAX_PREFIX_WALK_NODES
helpers that previously backed the live DAG-walk resolver have been deleted.
Resolving the current governance cut — ScopeProjections::namespace_current_heads
Current-state membership reads (as opposed to historical signed-position checks) need a "now"
frontier: the set of parent hashes that represents the namespace's current DAG head. The static
method ScopeProjections::namespace_current_heads provides this:
- Resolves the supplied
ContextGroupIdto its owning namespace by walking the parent chain. - Reads that namespace's DAG head record from the governance store.
- Returns the parent hashes of that head — the cut argument passed to
membership_status_at(orAclView::is_member_at_cut) for a current-state query.
None return cases. The method returns None when the group cannot
be resolved to a namespace (e.g. the group does not exist in the local store) or when the head
record itself is unreadable. Callers must fall back to the live membership store
(check_group_membership_path) in both cases rather than treating None as
an empty cut — an empty cut would falsely deny all access.
Projection-derived member counts and migration cohorts. ScopeProjections::member_identities_with
extends this pattern to identity-set reads: given an already-built projection fold it calls
cut_ancestry_complete on the fold before reading the ACL view, and returns
None if the ancestry is only partially folded (a governance-backfill race). This
None propagates to get_group_info (which falls back to
MembershipRepository::count + enumerate_inherited) and to
collect_migration_cohort (which falls back to the live list ∪ enumerate_inherited
union). The private ephemeral_view helper has been removed; the subtree ephemeral path
now calls ephemeral_fold directly and performs the cut_ancestry_complete
check itself before reading the view.
Ephemeral per-query projection — ScopeProjections::member_now_ephemeral
Context-layer handlers that do not hold a cached ScopeProjections instance use
ScopeProjections::member_now_ephemeral to obtain a one-shot membership verdict from
the governance DAG without requiring a pre-built projection. On each call it:
- Resolves the supplied group to its owning namespace.
- Folds the full governance DAG for that namespace via
collect_namespace_ops. - Reads the DAG heads after the fold (not before), so a concurrent governance-op advance cannot leave the cut naming ops the fold hasn't processed.
- Calls
member_at_cutat those heads and returns the boolean verdict.
None return cases. The method returns None — never deciding
against the member — when: (a) the namespace is unresolvable, (b)
collect_namespace_ops reports a store fault (surfaced with a tracing::warn),
or (c) the cut-completeness guard determines the recorded heads don't cover the full fold. Callers
must treat None as undecidable and fall back to the live store rather than
denying access.
Unified membership gate — ScopeProjections::member_now_checked
ScopeProjections::member_now_checked is the single decision point used by all
context-layer membership gates (get_cascade_status,
get_group_upgrade_status, list_group_contexts,
list_all_groups). Its semantics:
- Projection wins when decidable. It calls
member_now_ephemeralfirst. When the projection returns a verdict it acts on that verdict, then cross-checksMembershipRepository::is_memberbest-effort. A transient live error does not override a projection verdict. - Live wins when projection is undecidable. When
member_now_ephemeralreturnsNone,member_now_checkedfalls back toMembershipRepository::is_memberas authoritative and propagates any live error normally. - Divergence telemetry. When the projection and live store both return definite but
conflicting answers,
member_now_checkedemits atracing::warnwithmarker=unified_projection_divergenceandplane=membership-query. The projection verdict is still used as the final answer.
Direct use of MembershipRepository in
get_cascade_status.rs, get_group_upgrade_status.rs,
list_all_groups.rs, and list_group_contexts.rs is eliminated; those
handlers now call member_now_checked exclusively and no longer import
MembershipRepository.
Behavior change in list_all_groups. Previously, a membership-check error in
the per-group loop would propagate and abort the entire response. Now the loop catches errors from
member_now_checked with a tracing::warn and continues to the next group,
silently omitting groups whose membership cannot be determined. This matches the existing skip
behavior for a missing node identity and avoids a single inaccessible group blocking the whole
listing.
Op-store reconstruction root — ScopeProjections::scope_root_from_op_store and shadow_compare_op_store
ScopeProjections::scope_root_from_op_store(scope_id, store) provides an independent
view of a scope's governance root derived entirely from the persisted op log rather than the
maintained in-memory projection:
- Loads all ops for the given
ScopeIdviaload_scope_ops. - Replays them into a fresh
ScopeProjectionsinstance (causal ordering recovered throughDagStore<Op>+UnifiedApplier). - Returns the resulting
scope_rootcomputed with a fixed zeroentities_root.
Returns Ok(None) when the scope has no persisted ops; Err on store
faults. This is intended for completeness verification before C2.2b — not for hot-path membership
decisions.
ScopeProjections::shadow_compare_op_store(namespace_id, store) is an observe-only
companion that compares the maintained projection's scope_root against the
op-store reconstruction. On mismatch it emits a tracing::warn with
marker=unified_op_store_divergence and truncated hex of both roots.
This log marker is the e2e gate signal for C2.2b readiness. The method silently skips
(no divergence reported) when the scope is unfolded, the op-store is empty for it, or the store
read fails — none of those conditions count as divergence. calimero_op /
ScopeId types stay inside the context crate and are not leaked into the node crate.
Convergence root — ScopeProjections::scope_root_for and group_scope_root_ephemeral
ScopeProjections::scope_root_for(scope) computes a single
SHA-256 digest over the folded state of a governance scope:
SHA-256(entities_root ‖ acl_hash ‖ governance_hash). This
gives callers a cheap way to detect whether two nodes have converged on
the same membership/governance state without enumerating members.
- Returns
Nonewhen the scope has not been folded yet (i.e. it has no entry in the projection). Callers must treatNoneas "cannot compare" — never as a divergence signal. False divergences from mid-backfill nodes are the main hazard here. - ACL component is currently an empty placeholder because
SetWritersops are not yet routed through the governance DAG. The hash over ACL state will become meaningful at C2 when that routing lands.
The companion free function
ScopeProjections::group_scope_root_ephemeral(group_id, store)
builds an ephemeral projection from the raw store (no
NodeState required), adds an ancestry-completeness guard via
cut_ancestry_complete, and delegates to
scope_root_for. It returns None when the
governance DAG cut is not fully folded, preventing false divergence
reports from nodes that are still mid-backfill. Both symbols are public
API on ScopeProjections
(crates/context/src/scope_projections.rs).
Behavior is pinned by
scope_root_for_moves_on_hash_neutral_membership_change_and_is_none_when_unfolded:
the test holds entities_root fixed, ingests an
AdminChanged op then a causally-chained MemberAdded
op, and asserts that scope_root_for returns different values
before and after, and returns None for an unfolded scope.
Batch role resolution — ScopeProjections::member_roles_for
Where namespace_current_heads answers "is this member present?",
member_roles_for answers "what role does the projection assign to each of these
members?". It folds the ACL view exactly once for the supplied batch and returns a
BTreeMap<PublicKey, GroupMemberRole>:
- Direct members keep their folded role from the projection.
Inherited{via_admin:true}→Admin.Inherited{via_admin:false}→ the member's stored role at the anchor group (mirroring live'slist ∪ enumerate_inheritedsemantics).- Abstain (empty map) when the ancestry is not fully folded, or when the group has no
entry in
view.groups(materialized-fallback parity — Restricted subgroup or undecryptable member ops). An empty return means "projection has no opinion"; callers must not treat it as "all roles are wrong".
list_group_members calls member_roles_for once for all live members
after the existing identity-set shadow. Each live member's role is compared to its projected
role; mismatches are emitted as unified_projection_divergence log events with
plane=membership-role. Members absent from the returned map (projection abstains)
are skipped — presence divergence is already owned by the identity shadow. The handler still
returns the live list unchanged.
Operator note. membership-role divergence warnings in logs indicate a
pre-flip validation failure. They will block the eventual flip of list_group_members
from the live store to the projection as the authoritative source. Note that
acl_view_at and MembershipStatus (the former live DAG-walk resolver
types) are no longer available; divergence detection now operates entirely through
ScopeProjections projection methods.
Full effective-member enumeration — ScopeProjections::member_entries_with
member_entries_with returns a Vec<(PublicKey, GroupMemberRole)> for
every effective member of a group at the current cut, derived entirely from the already-folded
auth_cut_context view. It reuses member_identities_in_view_with_ctx for
the identity set and resolves each member's role via member_path_at_cut.
The method returns None (signalling: defer to the live store) in three cases:
- The cited ancestry is not yet fully folded — a partial fold could produce an incomplete member list, so the caller falls back rather than silently truncating.
- The target group has no folded direct-membership entry in the view — the projection does not cover this group yet.
- A validated identity resolves to
MemberPathAtCut::Noneor has a missing inherited anchor row — this indicates a fold inconsistency. The method logs aunified_projection_divergencewarning and abandons the entire enumeration rather than returning a partial list.
The all-or-nothing fallback contract means callers never observe a silently truncated member list: either the projection supplies the complete set or the live store is used in its entirety.
At-cut effective role — ScopeProjections::role_at_cut_for_group
ScopeProjections::role_at_cut_for_group(member, group_id, cut) resolves the effective
GroupMemberRole of a member in a group at a specific governance cut. It gates on
cut_ancestry_complete via auth_cut_context (returning None
immediately when ancestry is incomplete) and then delegates to member_path_at_cut to
handle three cases:
- Direct membership — returns the folded role from the projection (the same derivation
already validated by the
membership-roledivergence plane). Inherited{via_admin:true}— always resolves toAdmin, mirroring the live semantics for anchor-via-admin inheritance.Inherited{via_admin:false}— returns the member's stored role at the anchor group. ReturnsNonerather than guessing when the anchor row is missing, preventing spurious divergence reports.
The method returns None — signalling "projection has no opinion, defer to live store"
— in four cases: ancestry is not yet fully folded, the member is absent from the group at the cut,
the auth_cut_context call cannot resolve the namespace, and the anchor row is missing
for an inherited-via-anchor path. Callers must not treat None as a role denial.
Shared-context identity resolution — ScopeProjections::member_identities_in_view_with_ctx
member_identities_in_view_with_ctx accepts caller-supplied root and
default_cap_base instead of re-reading MetaRepository /
CapabilitiesRepository internally. member_identities_in_view delegates to
it unchanged (existing callers are unaffected). member_entries_with passes the values
it already obtained from auth_cut_context, ensuring the candidate-identity filter and
the per-member role resolution use a single consistent store snapshot with no window for a concurrent
governance write to make them disagree. Prefer member_identities_in_view_with_ctx over
member_identities_in_view whenever the caller already holds the root and capability
values — it avoids a redundant store read and eliminates the TOCTOU window.
Projection-first list_group_members with stable pagination
The list_group_members handler now reads from the projection authoritatively when a
folded proj_ctx is available. It falls back to the live
MembershipRepository::list ∪ enumerate_inherited union only when
member_entries_with returns None. Previously the handler always read from
the live store and compared the projection in shadow mode; the shadow scaffolding methods
shadow_member_enum_with and member_roles_for have been deleted as they had
no remaining callers.
Both the projection path and the live-store fallback path now apply a
members.sort_by_key(|(identity, _)| *identity) sort by PublicKey
before the skip(offset).take(limit) slice. The projection returns a
BTreeSet-ordered (PublicKey-sorted) result while the live store returns
insertion/store order; without the explicit sort, a mid-backfill request served by the projection
followed by a next-page request served by the live store could skip or repeat members. The sort
makes pagination order stable and path-independent across a mid-backfill transition.
member_now_checked / member_now_checked_with
ScopeProjections::member_now_checked and its shared-fold variant
member_now_checked_with are the two query-gate helpers that call
namespace_current_heads and then delegate to AclView::is_member_at_cut.
Their behavior follows a strict two-branch contract:
- Projection returns
Some— the result is acted on directly. There is no secondaryMembershipRepository::is_memberlive read and nounified_projection_divergencecross-check. The projection verdict is authoritative when it is available. - Projection returns
None(unresolvable group or unreadable head) — fall back to the live membership store (check_group_membership_path) as before.
The earlier best-effort live cross-check (which compared the projection verdict against the live
resolver and emitted a warning/error on divergence) has been removed from both functions. The
None→live fallback path is preserved unchanged.
At-cut admin & capability accessors — is_admin_at_cut, is_admin_or_capability_at_cut, is_last_admin_at_cut
Five additional methods on ScopeProjections expose admin/capability checks against a
historic DAG cut, mirroring what the live authorization gate does at apply time:
auth_cut_context(group_id, cut)— shared setup used by the public accessors. Resolves the namespace for the supplied group, gates oncut_ancestry_complete(returnsNoneimmediately if the cited ancestry is not yet fully folded), builds theAclViewfolded to the cut, and returns the genesis root tuple plus default-capability base.is_admin_at_cut(signer, group_id, cut)— returnsSome(true)if the signer holds an admin role at the cut,Some(false)if they do not, andNonewhen ancestry is incomplete. Callers must defer to the live gate onNone— incomplete ancestry is not equivalent to unauthorized.is_admin_or_capability_at_cut(signer, group_id, cut, cap_bits)— admin OR capability-bit check, mirroring the liveis_authorized_with_capability. SameNonesemantics as above. This method is a required method on theAtCutAuthorizertrait; all implementors must provide it.is_last_admin_at_cut(member, group_id, cut)— answers, at the op's parent cut (pre-mutation), whether removing or demoting the given member would orphan the group's admin set. Mirrors live logic exactly: a member counts as admin if they hold a directAdmin-role row or are the genesis admin; "another admin" requires a distinctAdmin-role row (the genesis admin alone does not satisfy it). ReturnsNonewhen ancestry is incomplete, deferring to the live last-admin guard.
Causal-position semantics. Because this method reads the DAG state the signer observed at sign time (the parent cut), not the receiver's current local state, it can legitimately disagree with a live query made after concurrent ops have advanced the DAG — this is the causal-honor property and is expected behavior, not a divergence.membership_path_at_cut(member, group_id, cut)— returnsSome(AtCutMembershipPath)indicating whether the member has a direct row, an inherited path, or no membership at the cut, andNonewhen ancestry is incomplete. Delegates toScopeProjections::membership_path_at_cut, which callsmember_path_at_cuton the folded projection viaauth_cut_contextand maps the result (projecting away the role/anchor detail). Callers must defer to the live gate onNone.
All public accessors return None (not Some(false)) when cited ancestry
is not fully folded, ensuring callers do not treat an incomplete fold as a denial verdict.
AtCutAuthorizer trait and implementations
The AtCutAuthorizer trait abstracts over how an at-cut admin/capability check is
resolved. It declares four required methods: is_admin_at_cut,
is_admin_or_capability_at_cut, is_last_admin_at_cut, and
membership_path_at_cut. All return Option<…> with the same
None = undecidable semantics.
Empty-cut contract. Every method on this trait must return None when
the supplied parents slice is empty. Violating this contract would cause a
genesis-state fold (no MemberAdded or role ops processed) to produce
Some(false), falsely rejecting genesis ops that the live store correctly permits.
LiveFallbackAuthorizer— the placeholder implementation used by the current namespace-envelope applier stage. ReturnsNonefor all four methods, delegating all decisions to the live store.EphemeralProjectionAuthorizer— the projection-backed implementation. Holds aMutex<Option<(ContextGroupId, Arc<FoldedProjection>)>>cache. The privatefolded(group_id)method checks the cache first; on a cache miss it callsScopeProjections::ephemeral_projectionand stores the result. If the fold fails (store or DAG-head fault) it emits atracing::warnand returnsNone. All four at-cut methods go throughfolded, so a single apply call reuses the same folded DAG rather than re-folding per check.is_last_admin_at_cutdelegates toScopeProjections::is_last_admin_at_cutand guards against an emptyparentsslice with an immediateNonereturn, consistent with the trait-level empty-cut contract.membership_path_at_cutdelegates toScopeProjections::membership_path_at_cut, which calls the projection'smember_path_at_cutwalk viaauth_cut_contextand maps the result toAtCutMembershipPath(projecting away the role/anchor detail). An emptyparentsslice returnsNoneimmediately, consistent with the empty-cut contract.
GroupGovernanceApplier — live gates only, no authorizer injection
The standalone GroupGovernanceApplier::apply continues to call
apply_local_signed_group_op (live-fallback) and does not accept an injected
AtCutAuthorizer. A code comment documents why: a SignedGroupOp's
parent_op_hashes belong to the per-group op log, not the namespace governance log
that EphemeralProjectionAuthorizer is keyed by. Passing those hashes to the
projection authorizer would resolve against the wrong id-space and silently no-op, producing a
false "undecidable" verdict on every group op. The real group-auth projection shadow therefore
fires only on the namespace-envelope group-op path, not through the standalone applier.
UnifiedApplier — projection-backing DeltaApplier<Op>
UnifiedApplier is the C2.0 struct that bridges DagStore<Op> and
ScopeProjections. It implements DeltaApplier<Op>: its
apply method acquires a write lock on the inner
Arc<std::sync::RwLock<ScopeProjections>>, calls
ScopeProjections::ingest_op with the delta's payload, then releases the lock. The
lock is a std::sync::RwLock (not a Tokio lock) and is never held across an
.await; lock poison is recovered rather than propagated, matching the stance
already documented on NodeState::read_scope_projections. Two constructors are provided:
UnifiedApplier::new()— creates a private, standalone projection. Use this for replay pipelines and tests.UnifiedApplier::with_projection(Arc<RwLock<ScopeProjections>>)— attaches to the node's live shared projection. This is the C2.1 dual-write seam: governance ops processed by the existing live pipeline are simultaneously ingested into the shared projection.
A projection() accessor exposes the Arc for inspection or cross-checking.
pub mod unified_applier is exported from crates/context/src/lib.rs,
making UnifiedApplier part of the crate's public API.
Behavior is pinned by
dag_releases_out_of_order_ops_and_the_projection_converges: a four-op causal chain
spanning the admin, membership, ACL, and data planes is fed through DagStore<Op>
+ UnifiedApplier in four delivery orders (in-order, full reverse, and two interleaved).
The test asserts that every op is applied (none stuck pending) and that scope_root
matches the result of direct in-order folding in all cases — confirming causal ordering and
projection convergence regardless of receive order.
Shadow authorization check in apply_signed_namespace_op
After a governance op is successfully applied by the live gate, the handler runs a non-blocking shadow authorization check for observability. The check is implemented in two parts:
apply_auth_requirement(op)maps the just-appliedSignedNamespaceOpto an(ContextGroupId, ApplyAuthReq)pair.ApplyAuthReqis eitherAdminorAdminOrCap(bits). The mapping covers:- Root-level
AdminChanged,PolicyUpdated,GroupReparented→Admin GroupCreated→AdminOrCap(CAN_CREATE_SUBGROUP)- Group-level
MemberAdded/MemberRemoved→AdminOrCap(MANAGE_MEMBERS) SubgroupVisibilitySet→AdminOrCap(CAN_MANAGE_VISIBILITY)MemberRoleSet,MemberCapabilitySet,DefaultCapabilitiesSet→Admin
None):MemberJoined/MemberJoinedAt(inviter authority;MemberJoinedAtadditionally enforces deterministic expiry — apply rejects whenjoined_at > expiration_timestamp),MemberJoinedOpen(inheritance proof),MemberLeft(self-authored),GroupDeletedandTransferOwnership(owner-gated), and other metadata/context-registration ops.- Root-level
- The handler then calls the corresponding at-cut accessor against the op's
delta_parentscut (the DAG state the signer observed, not the post-apply cut). If the projection returnsSome(false)— it would have rejected what the live gate accepted — awarn!is emitted withmarker=unified_projection_divergenceandplane=governance-auth.None(ancestry incomplete) is silently skipped. The check runs outside the projection write lock under a brief read lock.
Live authorization gates are entirely unchanged. The shadow check is log-only and has no effect on whether an op is accepted or rejected.
Concurrent-op convergence test — concurrent_independent_member_adds_converge
The integration test concurrent_independent_member_adds_converge pins the CRDT
convergence semantics for concurrent ops authored against the same genesis state. Two admins
independently add different members (D and E) against the same genesis state; the ops are applied
in opposite orders on two nodes. The test asserts: (1) both ops succeed on both nodes, (2) both
nodes converge to the identical four-member set, and (3) replaying a seen op is a no-op — the
per-signer nonce window dedups it and the member count does not change. The only apply gate is the
per-signer nonce window; there is no state-hash staleness check. The old test
(state_hash_prevents_concurrent_op_divergence) asserted that the second concurrent op
must fail; that hard-reject behavior no longer exists and the old test has been removed.
Where this can break
Two classes of change would silently re-introduce the divergence described above:
- Calling the projection with the wrong position. Any caller of the at-cut membership
resolver that passes the receiver's current state (or any post-cut heuristic) instead of the
signed delta position breaks forward-only. The three production call sites — gossip-receive,
governance-pending drain, and snapshot-sync replay — each carry inline comments naming this
contract; reviewers should flag any new caller that doesn't pass
delta.governance_positionverbatim. - Expanding the cut resolution to visit descendants. A "performance optimization" that
advances the resolved cut beyond the signed heads would observe ops the signer hadn't signed
against, including later
MemberRemovedops. The forward-only contract is documented on the public projection at-cut methods; theprefix_walk_forward_only_*regression tests pin the boundary. - Reintroducing a state-hash staleness gate. Governance ops no longer carry a
meaningful root-hash claim (
state_hashhas been removed from the synthesizedSignedGroupOpindecrypt_and_apply_group_op, andsigned_op_to_delta/signed_namespace_op_to_deltanow pass[0u8; 32]asexpected_root_hash). Any new code that gates apply on a non-zero root-hash match would incorrectly reject concurrent ops and break the CRDT convergence property pinned byconcurrent_independent_member_adds_converge. The per-signer nonce window is the sole apply gate for replay deduplication.
Scope
Forward-only applies to MemberRemoved and MemberLeft — operations
that revoke authorship. It does not apply to role demotions
(MemberRoleSet transitioning a Member to ReadOnly): those are enforced against
current state at the receive path, separately from the cross-DAG check, because a member who
was demoted retains a different relationship to their pre-demotion writes than a member who
was fully removed. See is_read_only_for_context for the live-state role gate.
leave_context — local-only opt-out
Stop syncing a context on this node. No DAG op, no broadcast, no key rotation. Other members never observe the leave; from their perspective the leaver is still in the membership list (which is implicit anyway, computed from group membership + auto-follow).
Storage
Two separate stores live side by side: the synced ContextIdentity row
(key at crates/store/src/key/context.rs:120, value at
crates/store/src/types/context.rs:117) and a node-local
ContextLeftMarker tombstone in a new column.
The marker lives in Column::ContextLocal (a new column
specifically for node-local context-scoped state — never synchronized).
Keying is identical to ContextIdentity so the
(context_id, member_pk) pair is the unit of opt-out:
// Borsh value:
pub struct ContextLeftMarker {
pub left_at_ms: u64,
}
A separate column instead of a flag on ContextIdentity for two
reasons: (a) ContextIdentity is part of the synced membership
shape, so adding a node-local field to it would conflate replicated and
node-local state; (b) extending the existing Borsh value would break
deserialization of pre-existing on-disk rows, requiring a migration.
The dedicated column makes the node-local-ness explicit at the storage
layer and keeps the synced shape clean.
Flow
- Client calls
leave_context(context_id). - Handler resolves the local
ContextIdentityviafind_local_signing_identity— scans the column for the row where this node holds the private key. Falls back to the namespace identity if no local row exists. - In a single batched store handle (so they land in one commit), the
handler:
- Writes the
ContextLeftMarkerrow inColumn::ContextLocal. - Deletes the corresponding
ContextIdentityrow, which stops sync (the sync layer iterates these rows).
- Writes the
- Calls
node_client.unsubscribe(&context_id)to leave the gossipsub topic, mirroringjoin_context's subscribe. - Auto-follow handler (
auto_follow.rs:255) checks for the marker viahas_left_contexton everyContextRegisteredevent and skips the auto-rejoin if it finds one.
Properties
| Scope | Local only |
| DAG entry | None |
| Key rotation | None — leaver still holds the group key |
| Coordination | None — cannot fail |
| Reversibility | Trivial — JoinContextRequest deletes the marker as a side-effect of joining and writes a fresh ContextIdentity row |
| Other members observe | No |
| Forward secrecy | Not provided — leaver retains the key, can decrypt anything they sync later if they rejoin |
What this is not
- Not a kick. Leaver still appears in any "members of this context" query.
- Not a cryptographic leave. If the user wants forward secrecy on a context, they must leave the group containing it (which rotates the group key).
leave_group — self-removal, no cascade
Self-removal from a single group via GroupOp::MemberLeft. The leaver publishes the
leave op; every receiver deletes the leaver's (member, group) row.
As-built caveat — no key rotation on self-leave. Unlike owner-initiated
MemberRemoved, MemberLeft deliberately does not trigger the
key-rotation pipeline. The publisher of a self-leave is the leaver, and the leaver cannot generate
the fresh group key without also retaining it — which would defeat forward secrecy. So
MemberLeft is the governance-level departure (the membership row is removed, peers
observe the leave, the leaver is deny-listed) without a re-key. The path to a
cryptographically-complete leave is an admin/owner-initiated MemberRemoved, which
rotates. A two-phase self-leave rotation (a remaining admin's apply hook publishes the new key) is
a tracked follow-up, not yet shipped. See the inline note in
crates/governance-store/src/ops/group/member_left.rs.
New op
// existing variants...
MemberLeft { member: PublicKey },
}
Distinct from MemberRemoved for audit clarity — "this member chose to leave" vs
"this member was removed" are semantically different events even when they have similar effects.
Preconditions
- Signer equals the member being removed (
SelfLeaveOnly) and has a direct row in the group; reject withMemberNotDirectif the signer is only an inherited member — they must leave the anchor instead. - Signer is not the group's
owner_identity(OwnerCannotSelfLeave; mustTransferOwnershipfirst). - Removing the signer does not leave the group with zero admins
(
LastAdminviaensure_not_last_admin_removal).
Flow
- Leaver's client calls
leave_group(group_id). - Apply enforces the preconditions above (self-equality, direct-row, owner, last-admin).
- Leaver publishes
MemberLeft { member: signer }via the governance DAG. Signed by the leaver. Targets this group. - Op broadcasts to all members of the group. Each receiver applies it:
cascade_remove_member_from_group_treedrops the leaver'sContextIdentityrows,remove_memberdeletes the(leaver, group_id)membership row, and the leaver is added to the group deny-list so their state-delta traffic is dropped until they re-join. - The apply emits
OpEvent::MemberRemoved; if the leaver's role wasReadOnlyTeeit additionally emitsOpEvent::TeeMemberRemoved, which drives the leaver's own node to self-purge the group's local key material. - No automatic re-key. As noted above,
MemberLeftdoes not rotate the group key — the leaver still holds the last key it received and can decrypt writes made under it. Forward secrecy on future writes requires the owner-initiatedMemberRemovedpath (or the deferred two-phase self-leave rotation).
Cascade behavior
| Direct rows in subgroups under this group | Untouched. Leaver keeps those memberships. (You can be invited to a child while explicitly leaving the parent.) |
| Inherited memberships in Open subgroups | Resolve themselves. The walker stops finding the leaver once their parent row is gone — no extra work needed. |
Failure modes
- No forward secrecy on self-leave (by design, not a failure). Because
MemberLeftnever rotates, the leaver retains the ability to decrypt writes made under the key they last held. To revoke that, an admin/owner must issueMemberRemoved(which rotates) or the deferred two-phase self-leave rotation must land. - Inherited-only membership. A signer whose membership is purely inherited (no direct row)
is rejected with
MemberNotDirect; they must leave the anchoring ancestor instead.
Rejoin
- Restricted group: requires fresh
add_group_membersfrom a remaining admin. New keys delivered via the existing membership-add path. - Open group (if leaver had a redundant direct row): same. If leaver's path to the group was via inheritance from a parent, that inheritance is automatically restored as soon as they rejoin the parent.
leave_namespace — cascading, heaviest
A namespace is the root group. Leaving it is a single MemberLeft at the namespace root
whose apply cascades through every descendant group where the leaver has a direct row. This is the
only leave operation that fans out across multiple scopes. As with leave_group, the
self-leave op does not rotate keys (same publisher-holds-the-key constraint); row removal +
deny-listing only.
Preconditions
- Signer has a direct row at the namespace root (
MemberNotDirectotherwise). - Signer is not the
owner_identityof the namespace root (OwnerCannotSelfLeave) nor of any subgroup beneath it where they have a direct row (OwnerOwnsSubgroup, reporting the offending scope). - Removing the signer doesn't leave any scope (namespace or descendant) admin-less. If it
would →
LastAdminreports the offending scope, forcing a successor promotion before retry. All these checks run upfront, before any mutation.
Flow
- Client calls
leave_namespace(namespace_id). - Apply detects the no-parent (namespace-root) case and walks the subtree:
- Compute the descendant groups where the signer has a direct row.
- Run owner check + last-admin check across all of them upfront. If any fails
(
OwnerOwnsSubgroup,OwnerCannotSelfLeave,LastAdmin) it bails before mutating anything — no half-applied cleanup.
- Signer publishes a single
MemberLeft { member: signer }at the namespace root. - The apply cascades: for each direct-row descendant it calls
cascade_remove_member_from_group_tree, removes the membership row, deny-lists the leaver, and emitsOpEvent::MemberRemoved. For each descendant where the leaver's role wasReadOnlyTeeit also emits a per-subgroupOpEvent::TeeMemberRemoved. The same removal then runs for the namespace root, emitting a rootMemberRemoved(and a rootTeeMemberRemovedif the root role wasReadOnlyTee). - No per-scope key rotation. The cascade is row-removal only — there is no
KeyDeliveryfan-out. Forward secrecy on each scope's future writes is provided by the owner-initiatedMemberRemovedrotation pipeline, not by self-leave. - Local cleanup on the leaver's node is role-scoped, not a soft/hard toggle:
- Non-TEE roles: soft leave. Local rows (identity, signing keys, contexts) are kept so rejoin / keyshare / inheritance-rejoin flows can reuse them.
ReadOnlyTee: the emittedTeeMemberRemovedevents drive the self-purge listener to hard-delete local key material across the whole subtree and unsubscribe from the namespace gossipsub topic.
Edge case — last member of the namespace
If the leaver is the last remaining member, MemberLeft writes locally but has no
peers to broadcast to. The namespace effectively dies on the leaver's node.
If the leaver is also the owner AND last member, they're stuck in the cycle — they must transfer
ownership but there's no one to transfer to. Use a dedicated DeleteNamespace op
(analogous to DeleteGroup but at root) that bypasses the owner-cannot-leave rule.
Failure modes
- No forward secrecy on self-leave across any scope (same constraint as
leave_group): the cascade removes rows but does not re-key. Use owner-initiatedMemberRemovedfor a cryptographically-complete eviction. - Interrupted TEE self-purge. If the leaver was a
ReadOnlyTeenode and its local cascade-purge is interrupted (crash, or a signing-key delete error), a durable pending-self-purge marker survives and the startupreconcile_sweepcompletes the purge on next restart. See TEE self-purge.
Role-scoped TEE self-purge
Removal alone (the MemberRemoved rotation) already provides forward secrecy on a
namespace's future writes — the evicted node never receives the rotated key. But a
ReadOnlyTee fleet node still holds, on its own disk, the old key material from before
the rotation: signing keys, the group encryption keys it last received, its
NamespaceIdentity, and the governance op-log. A regular member's local rows are
deliberately kept (the soft-leave invariant lets kick-and-readd /
rejoin-via-keyshare / inheritance-rejoin reuse them). A ReadOnlyTee node has
no rejoin path — re-admission via MemberJoinedViaTeeAttestation derives a fresh
attestation pubkey — so that material buys nothing and is pure hygiene debt. The self-purge handler
hard-deletes it.
Design record: docs/adr/0002-fleet-tee-leave-protocol.md. Implementation:
crates/context/src/self_purge.rs. Fleet-side framing:
Fleet HA.
Role-scoped trigger
The apply path emits a generic OpEvent::MemberRemoved for every removal, and — only when
the removed member's stored role was ReadOnlyTee — an additional
OpEvent::TeeMemberRemoved (see ops/group/member_removed.rs and
member_left.rs). The self-purge listener subscribes to the op-apply event channel and
reacts only to TeeMemberRemoved:
- Non-TEE removals (
Admin/Member/ReadOnly) keep soft-leave semantics — no purge. Hardening to hard-purge every removal would regress theapps/scaffolding-e2e/workflows/group-{kick,leave}-*workflows that depend on the retained rows. ReadOnlyTeeremovals trigger a hard purge. The self-detection ("did this op evict me?") is a handler concern, not part of the node-agnostic apply contract — it reads this node's stored namespace identity (per-node state). This mirrors the auto-follow architectural split.
Subgroup vs namespace-root
decide_purge_action (pure store reads) resolves the namespace owning the evicted group,
confirms the evicted member is this node's own identity, and then picks one of two actions:
PurgeAction::Subgroup | Removed from one subgroup while still a
member of others under the same namespace. Purge only that group's local rows
(delete_group_local_rows) plus its context-index and tree-edge rows. Do not
unsubscribe from the namespace gossipsub topic — other memberships still need it. |
PurgeAction::Namespace | Removed from the namespace root. Cascade the whole subtree, drop namespace-level state, then unsubscribe from the namespace topic. |
PurgeAction::None | Event is for another member, or for a namespace this node has no identity in. No action. |
What gets hard-deleted
The cascade (cascade_namespace_state) iterates the root plus every descendant group and
calls delete_group_local_rows per group. That helper
(crates/governance-store/src/local_state.rs) deletes the membership rows, capabilities,
metadata, upgrade records, the group op-log + head, the deny-list, and — load-bearing —
the private key material:
- Signing keys.
SigningKeysRepository::delete_all_for_group— the 32-byte private signing-key material. Leaking these is the actual forward-secrecy hazard, so this is the step the failure-class gating treats as security-critical. - Group encryption keys.
GroupKeyring::delete_all_for_group— the AES group keys the node received while admitted (added in PR #2776). Without this the evicted node would retain its copy of past group keys on disk even though the rotation already orphaned them for future writes.
Forward secrecy is therefore split cleanly: future writes are protected by key rotation (re-key excluding the removed member, done by the publisher pipeline at removal); the evicted TEE node's own copies of past keys are deleted here by self-purge. The purge does not — and need not — trigger any rotation itself.
Namespace-level teardown (delete_namespace_local_state) then drops the
NamespaceIdentity, the namespace governance head, and the namespace op-log. Dropping the
identity and unsubscribing is gated on the signing-key purge succeeding (a best-effort
dead-pointer cleanup failure does not block the security-critical finalize).
Durable marker + startup reconcile
TeeMemberRemoved fires once per eviction and an already-evicted identity receives no
further removal events, so a missed or partially-failed purge has no event-driven retry. Recovery is
a startup sweep, made role-safe by a durable marker (ADR 0002 / #2721):
- Before the namespace cascade,
handle_member_removedwrites a durable pending-self-purge marker for the namespace. This is the one site that knows — node-aware and role-aware — that this is a confirmed TEE self-eviction of our identity. - On startup,
reconcile_sweepenumerates markers only and completes a purge iff (marker present) AND (still no surviving namespace-root membership). If the identity is already gone, or we have been re-admitted, it clears the stale marker without purging; a read error skips (never purge on uncertainty). - The marker is essential because, post-eviction, the role row is erased: a role-blind scan of
NamespaceIdentityrows could not distinguish evicted-TEE residue from a pending join or non-TEE soft-leave residue, and would false-purge both. The marker is the role/intent gate; the still-evicted check is the safety gate; both must hold.
Owner (stored identity, not a role)
Owner is not a GroupMemberRole variant. It is a single identity stored on group
metadata: GroupMetaValue.owner_identity (crates/store/src/key/group/mod.rs).
Every group has exactly one Owner, and that member is otherwise an ordinary Admin — the
"Owner" status is whatever the apply handlers grant by comparing the signer's public key against
owner_identity. The distinction is a small set of exclusive privileges plus immunity
from involuntary removal; it is enforced by identity comparison, not by a role bit.
Storage
The struct carries both fields: the legacy admin_identity (now a
fallback creator-admin marker for pre-existing groups) and the shipped owner_identity.
Newly-created groups initialize owner_identity == admin_identity. Field shape and
serialization are stable.
New op
// existing variants + MemberLeft...
TransferOwnership { new_owner: PublicKey },
}
- Signer must equal current
owner_identity. new_ownermust already be a member of the group.- Apply: update
owner_identitytonew_owner. Old owner becomes a regular admin (retains admin powers; doesn't lose them as a side-effect of transfer). - No key rotation (membership unchanged).
Privilege matrix
| Action | Member | Admin | Owner |
|---|---|---|---|
Invite members (with CAN_INVITE_MEMBERS) | ✓ | ✓ | ✓ |
Create context (with CAN_CREATE_CONTEXT) | ✓ | ✓ | ✓ |
Remove non-owner members (MANAGE_MEMBERS) | — | ✓ | ✓ |
| Promote member → admin | — | ✓ | ✓ |
| Demote admin → member | — | ✓1 | ✓ |
TransferOwnership | — | — | ✓ |
DeleteGroup / DeleteContext / DeleteNamespace | — | — | ✓ |
Be involuntarily removed (MemberRemoved) | ✓ | ✓ | ✗ — OwnerImmuneFromRemoval |
Self-leave via leave_group / leave_namespace | ✓ | ✓ (last-admin permitting) | ✗ — OwnerCannotSelfLeave / OwnerOwnsSubgroup |
1 Open question. The existing code allows admins to demote each other
(mod.rs:823); restricting to Owner-only would tighten centralization. This proposal
preserves the current behavior (admins can demote each other), but the doc author may want to
revisit.
Owner per context
Same model. CreateContext records its creator as owner_identity on
the context's metadata. The context owner can TransferContextOwnership and
DeleteContext. Other than that, regular context membership is governed by the
parent group as before.
Interactions with leave operations
leave_context: no owner constraint. Local opt-out doesn't touch governance — even an owner can mute a context locally.leave_group: rejected for owner. Error fast-paths toTransferOwnership.leave_namespace: rejected if signer owns the namespace OR any subgroup beneath it. Error lists every owned scope. UI may bundle these into a single multi-transfer flow.
Single-Page Reference
| Op | Scope | DAG? | Key rotation? | Cascade? | Reversible? |
|---|---|---|---|---|---|
leave_context |
Local only | None | None | n/a | Trivial — flip flag |
leave_group |
Single group | MemberLeft |
No (self-leave; deferred) | None | Re-invite (Restricted) / re-inherit (Open) |
leave_namespace |
Whole subtree | MemberLeft at root |
No (self-leave; deferred) | All direct-row scopes | Re-invite at root + re-inherit |
MemberRemoved (owner/admin) |
Single group (cascades contexts) | MemberRemoved |
Yes — re-key excludes removed | Context identities | Re-invite (re-key delivered on add) |
ReadOnlyTee removal + self-purge |
Subgroup or whole namespace | MemberRemoved + TeeMemberRemoved event |
Yes (via removal) + local key delete | Subtree on namespace-root | None under same identity (fresh attestation) |
TransferOwnership |
Single group/context | Op | None (membership unchanged) | None | Transfer back |
DeleteGroup (Owner) |
Single group + descendants | Op | n/a (everyone gets removed) | Yes | None — destructive |
DeleteNamespace (Owner) |
Whole subtree | Op | n/a | Yes | None — destructive |
Implementation Status
This page documents shipped behavior. Each row points at the code that implements it.
| Section | Status | Code |
|---|---|---|
| Membership Model Recap | Documents existing code | — |
leave_context | Implemented | crates/context/src/handlers/leave_context.rs, auto_follow.rs gate, POST /admin-api/contexts/:id/leave |
leave_group | Implemented | GroupOp::MemberLeft, crates/governance-store/src/ops/group/member_left.rs, POST /admin-api/groups/:id/leave — apply enforces self-leave + direct-row + owner + last-admin. No key rotation on self-leave (deferred two-phase rotation). |
leave_namespace | Implemented | Same member_left.rs apply detects the no-parent case and cascades through descendant groups where the leaver has direct rows; multi-scope owner + last-admin checked upfront. No per-scope key rotation. |
Key rotation on MemberRemoved | Implemented | crates/governance-store/src/group_governance_publisher.rs (sign_apply_and_publish_inner) + GroupKeyring in group_keys.rs — fresh key wrapped for remaining members, excluding the removed one. |
| Role-scoped TEE self-purge | Implemented (ADR 0002; #2680/#2724/#2725) | crates/context/src/self_purge.rs listener + cascade + marker-gated reconcile_sweep; delete_group_local_rows in governance-store/src/local_state.rs deletes signing keys (and, as of #2776, GroupKeyring encryption keys); OpEvent::TeeMemberRemoved emitted from member_removed.rs / member_left.rs. |
Startup curative sweep (redrive_stranded_ops_sweep) | Implemented (#2848 Part C) | crates/context/src/self_purge.rs — one-shot sweep on startup after op_events subscribe; iterates namespaces via NamespaceRepository::iter_identities, finds decryptable-but-unapplied groups via NamespaceRetryService::groups_with_held_key_buffered_ops, re-drives via redrive_encrypted_ops_for_group_counted; converges up to MAX_PASSES=64 with pass-narrowing. auto_follow::spawn called before self_purge::spawn in ContextManager::started (load-bearing ordering — see note above). |
Born-Open atomic subgroup create (RootOp::GroupCreated restricted field) | Implemented (#2771) — WIRE BREAK | crates/primitives/src/context.rs (RootOp::GroupCreated), crates/governance-store/src/ops/group/group_created.rs (group_created::apply) — restricted: bool field added; apply writes subgroup visibility via CapabilitiesRepository::set_subgroup_visibility before queuing OpEvent::SubgroupCreated; gated on has_subgroup_visibility absence. CreateGroupRequest and REST visibility field updated. Op-adapter projection reads restricted from live op (no longer hardcodes true). |
TEE subgroup admission diagnostics (handle_new_subgroup) | Implemented | crates/context/src/tee_subgroup_admit.rs — handle_new_subgroup now emits debug! on entry (subgroup + namespace IDs) and at each early-return path (Open subgroup skip, non-key-holder skip, no-root-TEE-members skip); warn! when a TEE member has no admission verdict in the root op-log; info! just before admitting a TEE member into a Restricted subgroup. Set RUST_LOG=calimero_context=debug (or debug on the context crate) to surface per-subgroup TEE admission decisions in logs. |
ScopeProjections::namespace_current_heads | Implemented | crates/context/src/scope_projections.rs — resolves a ContextGroupId to its namespace, reads the namespace DAG head, and returns parent hashes as the current-state cut argument. Returns None on unresolvable group or unreadable head; callers fall back to check_group_membership_path. |
ScopeProjections::member_now_ephemeral | Implemented | crates/context/src/scope_projections.rs — builds a fresh projection per call: resolves the namespace, folds the DAG via collect_namespace_ops, reads heads after the fold, and returns member_at_cut. Returns None (never decides against the member) on unresolvable namespace, store fault, or incomplete fold. |
ScopeProjections::member_now_checked | Implemented | crates/context/src/scope_projections.rs — unified membership gate for all four context-layer handlers. Calls member_now_ephemeral first; falls back to MembershipRepository::is_member when projection is undecidable; emits tracing::warn with marker=unified_projection_divergence / plane=membership-query on definite disagreement. Direct MembershipRepository imports removed from get_cascade_status.rs, get_group_upgrade_status.rs, list_all_groups.rs, and list_group_contexts.rs. |
ScopeProjections::scope_root_for + group_scope_root_ephemeral | Implemented | crates/context/src/scope_projections.rs — scope_root_for computes SHA-256(entities_root ‖ acl_hash ‖ governance_hash) for a folded scope; returns None when the scope is unfolded (callers must treat as "cannot compare", not divergence). group_scope_root_ephemeral builds a fresh ephemeral projection from the raw store, applies an ancestry-completeness guard, and delegates to scope_root_for; returns None mid-backfill. ACL component is currently empty pending C2 SetWriters routing. Pinned by scope_root_for_moves_on_hash_neutral_membership_change_and_is_none_when_unfolded. |
ScopeProjections::role_at_cut_for_group | Implemented | crates/context/src/scope_projections.rs — resolves the effective GroupMemberRole at a governance cut via auth_cut_context + member_path_at_cut. Handles direct, inherited-via-admin (Admin), and inherited-via-anchor (anchor role row) cases. Returns None on incomplete ancestry, absent member, unresolvable namespace, or missing anchor row. Prevents spurious membership-role divergence reports by abstaining rather than guessing on missing anchor rows. |
UnifiedApplier + pub mod unified_applier | Implemented | crates/context/src/unified_applier.rs (exported via crates/context/src/lib.rs) — UnifiedApplier wraps Arc<std::sync::RwLock<ScopeProjections>> and implements DeltaApplier<Op>; apply holds the write lock only across the synchronous ScopeProjections::ingest_op call, never across an .await; lock poison is recovered. new() for standalone/test use; with_projection(arc) as the C2.1 dual-write seam; projection() accessor exposes the Arc. Convergence pinned by dag_releases_out_of_order_ops_and_the_projection_converges (four-op causal chain, four delivery orders, all converge to the same scope_root). |
unified_op_store — persist_op + load_scope_ops | Implemented (no production call sites yet) | crates/context/src/unified_op_store.rs (exported as pub mod unified_op_store from crates/context/src/lib.rs) — persist_op Borsh-serializes an Op and writes it under ScopeUnifiedOpKey in Column::UnifiedOp; idempotent because the key is content-addressed by op id. load_scope_ops performs a seek-then-entries range scan over the scope prefix and deserializes all rows back into Vec<Op>. Rows are returned in key order, not causal order — callers must replay the loaded ops through DagStore<Op> + UnifiedApplier to recover causal ordering and rebuild projections; heads are deliberately not persisted. No production apply or sync path calls these functions yet. Behavior is pinned by persisted_op_log_reloads_and_replays_to_the_same_projection: persists a three-op causal chain (AdminChanged → MemberAdded → SetWriters), reloads from a fresh store handle, replays through DagStore<Op> + UnifiedApplier, and asserts the reconstructed scope_root matches an independent in-memory fold; also asserts loading an empty scope returns zero rows. |
ScopeProjections::check_op_store_completeness | Implemented (diagnostic-only observe gate; retires at C3 Stage 4) | crates/context/src/scope_projections.rs — accepts an already-computed gov-DAG fold (dag_ops) and calls load_scope_ops to compare op-id sets. Emits op_store_incomplete warn on gaps; emits op_store_gate_unavailable and returns None when the store is unreadable. Returns None (unverifiable), Some(vec![]) (verified complete), or Some(ids) (missing ids). Pinned by completeness_gate_flags_governance_ops_missing_from_the_op_store in crates/context/tests/op_store_reconstruction.rs. |
ScopeProjections::scope_root_from_op_store + ScopeProjections::shadow_compare_op_store | Implemented (observe-only; C2.2b gate) | crates/context/src/scope_projections.rs — scope_root_from_op_store loads all persisted ops for a given ScopeId via load_scope_ops, folds them into a fresh ScopeProjections instance, and returns the governance scope_root using a fixed zero entities_root. Returns Ok(None) when no ops exist for the scope; Err on store faults. This is the op-store's independent view of a scope's governance root, intended for completeness verification before C2.2b. shadow_compare_op_store is an observe-only method that compares the maintained (governance-DAG) projection's scope_root against the op-store reconstruction for a given namespace. On mismatch it logs a tracing::warn with marker=unified_op_store_divergence and truncated hex of both roots — this log marker is the e2e gate signal for C2.2b readiness. Silently skips (no divergence reported) when the scope is unfolded, the op-store is empty for it, or the store read fails. calimero_op/ScopeId types remain entirely inside the context crate and are not leaked into the node crate. Behavior is pinned by two additional assertions in the existing persisted_op_log_reloads_and_replays_to_the_same_projection test: one verifying scope_root_from_op_store equals the expected in-memory fold result, and one verifying it returns None for a scope that has no persisted ops. |
AtCutAuthorizer trait + LiveFallbackAuthorizer + EphemeralProjectionAuthorizer | Implemented | crates/context/src/governance_dag.rs — AtCutAuthorizer declares four required methods: is_admin_at_cut, is_admin_or_capability_at_cut, is_last_admin_at_cut, and membership_path_at_cut. All must return None when parents is empty (trait-level empty-cut contract). LiveFallbackAuthorizer implements all four returning None. EphemeralProjectionAuthorizer holds a Mutex<Option<(ContextGroupId, Arc<FoldedProjection>)>> fold cache; the private folded helper reuses a cached fold for the same group or calls ScopeProjections::ephemeral_projection on a miss. All four at-cut methods delegate through folded; each opens with an empty-parents guard. is_last_admin_at_cut delegates to ScopeProjections::is_last_admin_at_cut (in crates/context/src/scope_projections.rs), which mirrors live last-admin logic at the parent cut. membership_path_at_cut delegates to ScopeProjections::membership_path_at_cut, which calls member_path_at_cut on the folded projection via auth_cut_context and maps the result to AtCutMembershipPath. The standalone GroupGovernanceApplier::apply is unchanged (live-only); see the GroupGovernanceApplier note in the at-cut accessors section above. |
Owner identity + TransferOwnership | Implemented | GroupMetaValue.owner_identity, GroupOp::TransferOwnership (ops/group/transfer_ownership.rs), OwnerImmuneFromRemoval / OwnerCannotSelfLeave gates in MemberRemoved / MemberLeft. |
Namespace genesis op (RootOp::NamespaceCreated) | Implemented | crates/context/src/handlers/create_group.rs (parent_group_id == None branch) — signs, applies, and publishes RootOp::NamespaceCreated { founder: admin_identity } via sign_apply_and_publish_namespace_op; NoPeersSubscribed from publish treated as success; any other error triggers rollback of all locally-written root rows (meta, founder Admin member row, default caps, group key, signing key, optional name metadata) via GroupKeyring::delete_key_by_id, SigningKeysRepository::delete_key, MetaRepository::delete, MembershipRepository::remove_member, CapabilitiesRepository::delete_default, MetadataRepository::delete_group; namespace identity row deliberately excluded from rollback so retries reuse the same identity/founder; DAG head is head-atomic and does not require rollback. |
Startup curative sweep — redrive_stranded_ops_sweep
At node startup, a one-shot curative sweep re-drives any encrypted ops that are buffered but
decryptable — that is, the node holds the group key but the op was never applied, typically
because the GroupCreated event that would have triggered delivery arrived late or
was dropped. Without this sweep such ops remain stranded forever.
Where it runs
redrive_stranded_ops_sweep is called from self_purge::run, after
subscribing to op_events and before entering the main event loop. It blocks the
event loop until the sweep finishes, but errors are logged and swallowed — the sweep never
prevents startup.
Algorithm
- Call
known_namespace_identities(viaNamespaceRepository::iter_identities) to enumerate every namespace this node has an identity for. - For each namespace, call
NamespaceRetryService::groups_with_held_key_buffered_ops(the inverse ofgroups_awaiting_key) to find groups whose key is already held but which still have buffered ops. - Re-drive each group via
redrive_encrypted_ops_for_group_counted, which applies as many buffered ops as possible and returns a count of newly applied ops. - Loop until a full pass applies nothing new (cross-signer convergence), bounded by
MAX_PASSES = 64. Pass-narrowing means only groups that made progress in the previous pass are revisited, avoiding redundant work.
Spawn ordering — load-bearing
ContextManager::started calls auto_follow::spawn before
self_purge::spawn. This ordering is intentional and load-bearing:
auto_follow::spawn subscribes to op_events synchronously on the caller
thread before spawning its async task, and passes the receiver into auto_follow::run().
Because the curative sweep (inside self_purge::run) may emit
OpEvent::ContextRegistered for previously stranded contexts, the auto-follow receiver
must exist before those events are broadcast — otherwise they are silently dropped
by the broadcast channel. Reversing the spawn order would silently break auto-follow for any
context recovered by the startup sweep.
Reviewers: both call sites in ContextManager::started
carry inline comments documenting the ordering constraint. Any refactor that reorders
auto_follow::spawn and self_purge::spawn must preserve the
subscribe-before-sweep invariant.
Relevant symbols
redrive_stranded_ops_sweep— orchestrator; lives incrates/context/src/self_purge.rsredrive_buffered_ops_for_group— inner per-group apply loopredrive_encrypted_ops_for_group_counted— wraps the above and returns a count of newly applied ops for pass-narrowingnamespace_groups_with_held_key_buffered_ops— store query (inverse ofgroups_awaiting_key)NamespaceRetryService::groups_with_held_key_buffered_opsNamespaceRepository::iter_identities
Born-Open atomic subgroup create — RootOp::GroupCreated wire change
⚠ WIRE BREAK. RootOp::GroupCreated has a new Borsh
field (restricted: bool). Nodes running this version cannot exchange governance ops
with nodes running the previous wire format. Nodes must reset their governance
op-log before upgrading.
Problem
Previously, the GroupCreated op carried no visibility information. The
tee_subgroup_admit subscriber fired on every OpEvent::SubgroupCreated and
wrote a transient direct ReadOnlyTee membership row for Open subgroups — a row that
would need to be retracted once a SubgroupVisibilitySet(Open) op arrived. This
transient row could cause spurious TEE admission flows.
Fix — restricted field on GroupCreated
RootOp::GroupCreated gains a restricted: bool field:
restricted: true(default) — subgroup is born Restricted. Legacy behavior is preserved; no wire difference for groups created this way.restricted: false— subgroup is born Open. The apply handler writes the subgroup's visibility key viaCapabilitiesRepository::set_subgroup_visibilityduring apply, beforeOpEvent::SubgroupCreatedis queued.tee_subgroup_admitsees the group as already Open when it reacts, skips it, and no transient direct row is written.
The write is gated on CapabilitiesRepository::has_subgroup_visibility absence so
later SubgroupVisibilitySet flips are never clobbered.
API surface changes
CreateGroupRequestgains arestricted: boolfield (defaulttrue).- The
create-group-in-namespaceREST endpoint accepts an optionalvisibilityfield ("open"|"restricted", default"restricted"). - The op-adapter projection reads
restrictedfrom the live op instead of hardcodingtrue.
Relevant symbols
RootOp::GroupCreated—crates/primitives/src/context.rsgroup_created::apply—crates/governance-store/src/ops/group/group_created.rsCapabilitiesRepository::has_subgroup_visibility,CapabilitiesRepository::set_subgroup_visibilityCreateGroupRequest,payload_from_root_op
Open Questions & Follow-ups
- Two-phase self-leave rotation.
MemberLeftcurrently does not re-key (the leaver can't generate a key without retaining it). A deferred design has a remaining admin's apply hook publish the new key after the leave. Until it lands, owner-initiatedMemberRemovedis the only rotating leave. - Local cleanup is role-scoped, not a toggle. The earlier soft-vs-hard default question is
resolved by role (ADR 0002): non-TEE removals stay soft;
ReadOnlyTeeremovals hard-purge. No global default knob. - Admin-mutual demote. Admins can demote each other. Whether to restrict this to Owner-only is unresolved; current behavior is preserved.
- Old owner after
TransferOwnership. Becomes a regular admin (current behavior) or just a member (alternative)? - Subgroup-level self-purge reconcile. The startup reconcile is namespace-level only; a subgroup-only purge has no reconcile retry surface yet (tracked in #2726).
- Periodic reconcile. A purely-lagged-drop of
TeeMemberRemovedwrites no marker and is not recovered until a future eviction; a periodic sweep is an open follow-up.
EphemeralProjectionAuthorizer — empty-parents early return
Both is_admin_at_cut and is_admin_or_capability_at_cut on
EphemeralProjectionAuthorizer now return None immediately when the
supplied parents slice is empty, deferring unconditionally to the live fallback
authorizer. Previously, an empty parents slice was passed through to the fold
machinery, which evaluated the query against a genesis-state view. That view contains no granted
roles or capabilities, so any capability holder whose grant was established after genesis would be
falsely rejected by the projection even though the live store correctly recognises them.
When parents is empty
Two cases produce an empty parents slice at the authorizer call site:
- Genesis op. The very first op in a namespace's governance log has no predecessor hashes. The causal cut is genuinely empty because no prior ops exist.
- PermissionChecker constructed without apply-auth context. Some code paths
build a
PermissionCheckeroutside of a normal apply flow and pass an empty parents field as a zero value. These callers have no meaningful causal position to resolve against.
In both cases, judging against an empty fold is semantically wrong: the fold has not processed any
MemberAdded, MemberRoleSet, or DefaultCapabilitiesSet ops,
so the resulting AclView is empty. A query against it would return
Some(false) for every capability check, silently blocking operations that the live
store would correctly permit.
New behavior
- If
parents.is_empty(), both methods returnNoneimmediately, before touching the fold cache or callingScopeProjections::ephemeral_projection. - The
Nonereturn propagates to the caller as "projection is undecidable," which causes the standard live-fallback path to take effect. No warning is emitted — empty parents is a legitimate undecidable case, not a divergence. - When
parentsis non-empty, behavior is unchanged: the fold cache is consulted (or populated on a miss) and the at-cut check proceeds normally.
Contract clarification
The AtCutAuthorizer contract already specifies that None means
"undecidable — defer to the live gate." This change makes the empty-parents case an explicit
instance of that contract rather than a silent fall-through to a semantically incorrect verdict.
Callers must not treat None as a denial; the live authorizer is always consulted when
the projection abstains.
The None-on-incomplete-ancestry guard (from cut_ancestry_complete) is
distinct and remains in place: it fires when parents is non-empty but the fold has
not yet processed all ops reachable from those parents. The empty-parents guard fires earlier,
before the fold is attempted at all.
Affected code
crates/context/src/governance_dag.rs—EphemeralProjectionAuthorizer::is_admin_at_cutandEphemeralProjectionAuthorizer::is_admin_or_capability_at_cut: both now open withif parents.is_empty() { return None; }before thefolded(group_id)call.- No changes to
ScopeProjectionsor any call sites — the early return is entirely internal to the authorizer methods.LiveFallbackAuthorizeris unaffected (it already returnsNoneunconditionally for all methods, including the newis_last_admin_at_cut).
Deferred op-store read-flip — ops_for_namespace indirection layer
Status: C2.2b deferred. The unified op-store (load_scope_ops) is
not yet the source of truth for projection rebuilds. All projection-rebuild call sites
now route through a new stable indirection method, ScopeProjections::ops_for_namespace,
which currently delegates to collect_namespace_ops (the governance-DAG walk). The
flip to load_scope_ops is explicitly deferred until the late-decrypted membership
bug described below is resolved.
ops_for_namespace — the stable indirection point
ScopeProjections::ops_for_namespace(namespace_id, store) is the single dispatch
point used by every projection-rebuild site:
backfill_namespaceinscope_projections.rsmember_now_ephemeral/ephemeral_foldinscope_projections.rs- The
scope_rootreconstruction path inscope_projections.rs refresh_projection_for_cutin the node crate
All four previously called collect_namespace_ops directly. They now call
ops_for_namespace instead. collect_namespace_ops remains as the
underlying implementation and is unchanged; ops_for_namespace is a thin wrapper
that will be flipped to call load_scope_ops at C2.2b without requiring changes at
any call site.
Why the flip was blocked — late-decrypted membership (resolved by C2.2c)
An encrypted MemberAdded op can arrive at a node before that node holds the group
key. At apply time the op is processed as Noop and written to the op-store with a
Noop payload. When the group key later arrives — via a keyshare or a
KeyDelivery event — the governance-DAG walk (collect_namespace_ops)
re-decrypts the op at read time and correctly surfaces it as MemberAdded.
Previously, the op-store entry was not updated; it still recorded Noop.
Consequence: if ops_for_namespace were flipped to load_scope_ops before
this was fixed, any member whose MemberAdded op was frozen as Noop at
write time would silently disappear from the projection after a rebuild. This was confirmed by e2e
timeouts on the joiner's post-write sync in the group-3node,
scaffolding-e2e, and shared-storage rotation workflows.
Fix: ScopeProjections::repersist_namespace_ops. After any group key
is delivered, the key-delivery path calls
ScopeProjections::repersist_namespace_ops(namespace_id, store), which re-walks the
namespace through collect_namespace_ops (the canonical read-time decryption path) and
calls persist_op for every op it finds. Because ops are content-addressed by the same
op id, a now-decryptable op (e.g. MemberAdded) overwrites the stale Noop
that was frozen at apply time when the key was absent. The method is best-effort (errors are logged
but not propagated), idempotent, and bounded by the same MAX_BACKFILL_OPS walk cap
used elsewhere.
ScopeProjections::check_op_store_completeness — diagnostic completeness gate
ScopeProjections::check_op_store_completeness(dag_ops, store) is a diagnostic-only
observe gate that compares the op-ids from an already-computed gov-DAG fold (dag_ops)
against the op-ids loaded from the unified op-store via load_scope_ops. For every op
present in the gov-DAG fold but absent from the op-store it emits an op_store_incomplete
tracing::warn event carrying missing_count, dag_count,
op_store_count, and a hex-encoded sample_missing id. If the op-store
cannot be loaded at all it emits a distinct op_store_gate_unavailable marker and
returns None so a store fault cannot masquerade as a verified-clean result.
Return semantics:
None— op-store was unreadable; completeness is unverified. Callers must not treat this as a clean result.Some(vec![])— verified complete; every gov-DAG op-id is present in the op-store.Some(ids)— those op-ids are present in the gov-DAG fold but missing from the op-store.
This method is diagnostic-only and has no effect on whether ops are accepted or rejected.
It retires at C3 Stage 4 when the authoritative read path flips onto the op-store. The companion
regression test
completeness_gate_flags_governance_ops_missing_from_the_op_store in
crates/context/tests/op_store_reconstruction.rs builds three governance ops, persists
only two to the op-store, calls check_op_store_completeness with all three as the
dag-fold, and asserts the return is Some(vec![op3.id]); after persisting the third it
asserts the return becomes Some(vec![]), confirming the None vs
empty-vec distinction works correctly.
Removed: shadow_compare_op_store and scope_root_from_governance_dag
The observe-only shadow_compare_op_store method on ScopeProjections
and its companion scope_root_from_governance_dag have been removed, along with the
call site in refresh_projection_for_cut that triggered a full governance-DAG fold
on every backfill.
The comparison these methods performed was structurally flawed: the maintained projection holds
live ingest_op-fed ops that may not yet be written to the op-store. A gap in the
op-store could therefore be masked by the live apply path, making the divergence invisible to the
cross-check. The marker=unified_op_store_divergence log signal described in earlier
sections of this page is no longer emitted; the C2.2b readiness gate is now the
op_store_reconstruction_matches_governance_dag_for_late_decrypted_membership
regression test described below.
Regression test — op_store_reconstruction.rs
The integration test file
crates/context/tests/op_store_reconstruction.rs is the CI regression gate for this
fix. The test is named
op_store_reconstruction_recovers_late_decrypted_membership_after_key_delivery and is
structured as an explicit before/after:
- Injects an encrypted
MemberAddedop into the governance DAG with the group key absent at write time, persisting the op-store entry asNoop. - Before re-persist: asserts that
load_scope_opsreconstruction seesSome(false)for the member (the frozenNoopstate). - Calls
repersist_namespace_opsto simulate key delivery. - After re-persist: asserts that the op-store reconstruction matches the
governance-DAG fold — the member is now
Some(true).
The test runs in CI without #[ignore] and serves as the C2.2c readiness gate. The
module-level doc comment in the test file explains the full bug chain and the fix.
Summary of changes
ScopeProjections::ops_for_namespaceintroduced; all four projection-rebuild call sites updated fromcollect_namespace_opstoops_for_namespace.collect_namespace_opsunchanged — still the implementation behindops_for_namespace.shadow_compare_op_storeandscope_root_from_governance_dagremoved fromScopeProjections;marker=unified_op_store_divergenceis no longer emitted.refresh_projection_for_cutcall site that triggered a shadow fold on every backfill deleted.ScopeProjections::repersist_namespace_opsadded; called by the key-delivery path after a group key arrives to overwrite staleNoopop-store entries with their now-decryptable payloads. Best-effort, idempotent, bounded byMAX_BACKFILL_OPS.crates/context/tests/op_store_reconstruction.rsupdated: test renamed toop_store_reconstruction_recovers_late_decrypted_membership_after_key_delivery,#[ignore]removed, body restructured into an explicit before/after assertion around arepersist_namespace_opscall.
ScopeProjections::persist_namespace_head_ops — mirroring authored ops into the op-store
After a local authoring step publishes a governance op (GroupCreated,
GroupDeleted, MemberJoinedOpen), the unified op-store must be updated so
that any subsequent load_scope_ops-based projection rebuild sees the authored op
without relying on a peer retransmission. The method
ScopeProjections::persist_namespace_head_ops(store, namespace_id) is the single
call site for this update.
Why not a heads-only walk
A naive implementation would read the namespace's current DAG heads immediately after publish and
persist only those ops. This races: if a concurrent peer op is applied between the local
publish call and the heads read, the head record advances past the just-authored op.
The authored op is then unreachable from a heads-only walk and is silently omitted from the
op-store. The member or group it describes disappears from any rebuild that sources ops from
load_scope_ops.
Implementation — full gov-DAG fold via repersist_namespace_ops
persist_namespace_head_ops delegates unconditionally to
repersist_namespace_ops, which performs a full
collect_namespace_ops-backed fold bounded by MAX_BACKFILL_OPS. This
walk is read-time decryption-aware, so an authored op that was encrypted at publish time is
decoded here and written to the op-store with its actual payload (not as Noop). The
walk captures the authored op regardless of where it landed in the DAG relative to concurrent
peer ops, because collect_namespace_ops traverses the full ancestry from the current
head rather than just the head itself.
The call is:
- Idempotent.
persist_opis content-addressed by op id; calling it twice for the same op is a no-op. - Best-effort. Errors are logged by
repersist_namespace_ops(using the sametracing::warnpaths as the key-delivery re-persist) and are never propagated. A failure to update the op-store never causes the authoring step to return an error to the caller. - Bounded. The walk is capped at
MAX_BACKFILL_OPS, the same limit used by every othercollect_namespace_ops-backed path. An unusually large namespace cannot cause an unbounded fold at authoring time.
Unified authoring wrapper — sign_apply_and_publish_group_op
Rather than calling persist_namespace_head_ops individually in each handler,
all group-op authoring is routed through a single crate-internal async wrapper:
crate::sign_apply_and_publish_group_op. It calls
calimero_governance_store::sign_apply_and_publish and, on success, resolves the
namespace from the group ID via NamespaceRepository::resolve and calls
ScopeProjections::persist_namespace_head_ops. A namespace-resolve failure logs a
tracing::warn identifying the group ID but does not fail the authoring operation.
Callers must not call calimero_governance_store::sign_apply_and_publish
directly for group ops. Using the wrapper ensures the op-store is always updated at the
single chokepoint and that namespace-resolve failures are observable via tracing warnings.
Wired call sites
The method is called after the governance op is successfully published or applied, never before. A failure at the authoring step is not affected:
create_group— called afterGroupCreatedis successfully reported to the governance pipeline.delete_group— called afterGroupDeletedis applied.join_context— called after theMemberJoinedOpenop is published. The call is skipped (effectively a no-op path) if the publish or apply step errored.join_subgroup_inheritance— called afterMemberJoinedOpenis published, following the same skip-on-error pattern.- All other group-op handlers —
add_group_members,admit_tee_node,create_context(two call sites),delete_context,detach_context_from_group,leave_group,leave_namespace,set_member_auto_follow,set_member_capabilities,set_subgroup_visibility,set_tee_admission_policy,update_member_role,upgrade_group(four call sites),dispatch_cascade, and the generic governance handler inlib.rsall callcrate::sign_apply_and_publish_group_opinstead of the underlying store function.
Exception — remove_group_members. The key-rotation removal path uses
sign_apply_and_publish_removal (not the standard publish function) and cannot use the
wrapper. After each removal it inlines the same namespace-resolve +
persist_namespace_head_ops logic directly. A namespace-resolve failure logs a
tracing::warn identifying the group ID.
Regression test
The test
persist_namespace_head_ops_lands_locally_authored_op_in_the_op_store in
crates/context/tests/op_store_reconstruction.rs verifies the full round-trip:
- An encrypted
MemberAddedop is injected into the governance DAG (op-log + DAG head) but not into the op-store, simulating the local-author gap. - The test asserts that
load_scope_opsreturns no entry for the op before the call — confirming the gap exists. persist_namespace_head_opsis called.- The test asserts the op now appears in the op-store.
- A fresh projection is folded from the op-store contents and
member_at_cutis called. The test asserts it returnsSome(true)— proving the encrypted payload was decoded correctly, not merely that the op id was recorded.
The test runs without #[ignore] in CI and is the readiness gate for any future flip
of ops_for_namespace to source from load_scope_ops at authoring call
sites.
Relationship to repersist_namespace_ops
repersist_namespace_ops was introduced to recover late-decrypted
MemberAdded ops after key delivery (see the deferred op-store read-flip section
above). persist_namespace_head_ops reuses that same mechanism for a different trigger
— local authoring — ensuring both the key-delivery path and the authoring path produce a
consistent op-store state. The two call sites share the same underlying fold, the same error
handling, and the same MAX_BACKFILL_OPS bound.
Atomic op-store write — removal of per-site dual-write scaffolding
Status: shipped. store_signed_operation now persists every
governance op atomically at apply time, making all per-site op-store write calls redundant. This
section documents what was removed and why, and supersedes the
Deferred op-store read-flip and
persist_namespace_head_ops sections above where they describe the old
per-site scaffolding.
What was removed
- Receive-handler dual-write. The
persist_opcall insideapply_signed_namespace_opthat was labelled C2.1b dual-write has been deleted. Theingest_opfold into the live projection is deliberately kept; only the now-redundant durable-store write is gone. sign_apply_and_publish_group_opwrapper. The crate-private function inlib.rsthat wrappedcalimero_governance_store::sign_apply_and_publishand appended apersist_namespace_head_opscall is deleted entirely. All ~15 call sites (add_group_members,admit_tee_node,create_context,delete_context,detach_context_from_group,join_context,leave_group,leave_namespace,set_member_auto_follow,set_member_capabilities,set_subgroup_visibility,set_tee_admission_policy,update_member_role,upgrade_group,dispatch_cascade, and others) now callcalimero_governance_store::sign_apply_and_publishdirectly.ScopeProjections::persist_namespace_head_ops. The thin alias forrepersist_namespace_opsthat documented the C3 Stage 1/2/3 per-site persist pattern is deleted fromscope_projections.rs. All call sites are also removed:create_group,delete_group,join_context,join_subgroup_inheritance,remove_group_members,join_namespace(node crate),reparent_group(admin API), andcreate_group_in_namespace(admin API).
Why the per-site scaffolding was safe to delete
store_signed_operation is called on every apply path — both local authoring and
peer-receive — and now writes the op into the unified op-store as part of the same atomic
commit. Because the persist is guaranteed at the apply chokepoint, no handler needs to issue a
separate post-publish persist call. The two problems the scaffolding addressed are resolved at
source:
- Local-author gap. A locally authored op is written by
store_signed_operationat apply time, before the handler returns. There is no window in which the op could be absent from the op-store after a successful publish. - Late-decrypted membership.
repersist_namespace_ops(called by the key-delivery path) still overwrites staleNoopentries with their now-decryptable payloads. The atomic write path records the best payload available at apply time; the key-delivery re-persist corrects any entry frozen asNoopbefore the group key arrived. Both paths remain in place; only the redundant per-site calls are removed.
Race that the old heads-only walk would have introduced
The deleted persist_namespace_head_ops performed a full
collect_namespace_ops-backed fold after publish to avoid a heads-only-walk race
(a concurrent peer op advancing the head past the just-authored op before the walk ran). The
atomic write in store_signed_operation sidesteps this entirely: the authored op is
written at apply time under its content-addressed op id, so its presence in the op-store is
independent of where the DAG head points afterward.
Surviving use of repersist_namespace_ops
repersist_namespace_ops itself is not deleted. It continues to
serve the key-delivery path: when a group key arrives late, the key-delivery handler calls
repersist_namespace_ops to re-walk the namespace via
collect_namespace_ops and overwrite any op-store entries that were frozen as
Noop at apply time because the key was absent. This is the only remaining call site.
Updated test — locally_authored_op_lands_in_the_op_store_atomically
The test previously named
persist_namespace_head_ops_lands_locally_authored_op_in_the_op_store is renamed
locally_authored_op_lands_in_the_op_store_atomically. Its structure is updated to
reflect the atomic-write path:
- An encrypted
MemberAddedop is passed throughstore_signed_operation(the atomic path). - The test asserts the op appears in the op-store immediately after that call, with
no intervening
persist_namespace_head_opscall required — authoring handlers need only callcalimero_governance_store::sign_apply_and_publishdirectly; no post-author mirror step is needed. - The test then calls
ScopeProjections::repersist_namespace_opsdirectly (replacing the now-deletedpersist_namespace_head_opsalias) and asserts the op-store state is unchanged — confirming idempotency over an already-present entry.
The test runs without #[ignore] in CI and is the readiness gate confirming that the
atomic-write path is sufficient and that the per-site scaffolding carried no load that
store_signed_operation does not now cover.