Skip to content

Call an external service

Task: call a service outside the cluster (a captcha check, a payment API, a model, any HTTPS API) and write what it answers into state.

A call may ask its node for one HTTPS request. The node makes it only when every gate below holds and answers with its reason otherwise. Outbound requests are off by default: an app that needs them runs on a cluster whose operator turned them on, for its hosts or for any host an app declares.

Gate Who sets it
[outbound] enabled = true, and the request’s host is in hosts (or hosts holds "*": any host an installed app declares) The cluster’s operator
The request’s URL is under a prefix in the app’s signed release manifest (outbound): the same host, and the prefix’s path or one below it at a /, with any query string The app’s publisher
A prefix covering the URL is approved for the app’s publisher Whoever holds MANAGE_APPLICATION org-wide in the app’s organization (its owner and admins; a member granted it), its personal organization included; the operator where the cluster was deployed with --outbound-approvals operator
The run is a call that writes: not a view, an init, a migration, or a call by a read-only member The node
One request per run; GET, POST, PUT, PATCH or DELETE; https on port 443 to a DNS name that resolves only to public addresses; no fragment; no redirect followed; no proxy The node
Request body ≤ 64 KiB, URL ≤ 2 KiB, ≤ 8 headers of ≤ 1 KiB, response body ≤ 1 MiB (a longer one is refused, not cut), 10 s for the whole request The node (calimero_primitives::outbound)
Host, Cookie, Connection, Content-Length, Transfer-Encoding, Proxy-* and X-Forwarded-* headers are refused The node

A refusal is an AppError with code Fetch and a message field. A refusal for a missing approval starts with a stable word an app can match on:

  • NotApproved: nobody approved the prefix the URL is under. Nothing was ever approved for this publisher, or the owner saw the prefix and left it out or revoked it. Tell a member to ask the owner.
  • AwaitingApproval: a later release of the app declares the prefix and the owner has not been asked about it yet. The prefixes approved before keep working. Show the owner what to approve.

A run with no outbound client at all (outbound is off, the app is not a signed release declaring a prefix, or the run is not a writing call) and a second request in one run trap the run.

A release declares the secrets its requests name in its signed manifest, with the hosts each may go to, so whoever deploys it sees what to supply:

[package.metadata.calimero.secrets.NEAR_AI_API_KEY]
hosts = ["cloud-api.near.ai"] # each the host of one of `outbound`'s prefixes
description = "NEAR AI Cloud API key"
required = true

meroctl app pack writes it as secrets and refuses a host no outbound prefix covers, as a node does at install (400 BAD_SECRETS). A node then fills only a declared name, and only into a request to one of its declared hosts (and one the operator bound it to). A release that declares none is filled from the operator’s bindings alone, as before. The release’s capability card (GET /packages/{package}/{version}/capabilities) lists them, and an upgrade’s consent page names a secret it adds or sends elsewhere. A release declaring secrets installs only once every node runs 1.0.0-rc.31 (declaredSecrets).

Whoever deploys an instance may supply a declared secret’s value for that instance alone, instead of the operator:

  • PUT /apps/{app}/secrets/{name} { value } (the owner, or whoever manages the organization’s apps, with a passkey step-up over the name and the value’s fingerprint, never the value): kept in the app’s log on its replicas, sealed, out of the app’s own reach (no host function reads it), and filled only into a request to that secret’s declared hosts. It wins over the operator’s value for that name.
  • GET /apps/{app}/secrets lists each declared secret with source: app (its fingerprint, who set it and when) or cluster, never a value.
  • DELETE /apps/{app}/secrets/{name} removes it; the deployment’s or the operator’s applies again.

A deployment’s managers may supply one value for every instance created from it (PUT /deployments/{id}/secrets/{name} { value }, a passkey step-up over deployment:{id} / __set_secret / { name, fingerprint }; GET answers fingerprints, DELETE clears it; meroctl deployment secret set|list|clear), for a name a release the deployment offers declares, or a release one of those references in its uses declares (for the apps an instance creates with it). It is kept in the system group’s state and read by the outbound gate on the node that runs the request. A request is filled with the instance’s own value, else its deployment’s, else the operator’s, each only for a name the release declares and only to its declared hosts, and the deployment’s only for a release of a package the deployment offers; an instance’s GET /apps/{app}/secrets then says source: deployment, nothing more of it. See Deployments.

filling {{secret:NAME}} in one run's requesta run asks for NAMEan instance or aunit memberits release must declare it1 · the instance'sPUT /apps/{app}/secrets/{name}in the app's own logelse2 · its deployment'sPUT /deployments/{id}/…only if the run's package isone the deployment coverselse3 · the operator's[outbound] env_secretsor secrets serviceelse: nothingthe outbound gate, on the node that runs itan approved host, and only one the release declares for NAMEany other host: the request is refused, no value leavesthe response, each copy of the value → [redacted]

Read it left to right: the first value found wins, each source only for a name your release declares, and whichever wins goes out only to that secret’s declared hosts and never comes back in a response.

The app never holds a secret. A request names one as {{secret:NAME}} in a header value or the body; the node fills it in from its operator’s secrets, for this app only (the package and signer of the installed release, never anything the app says), and only into a request to a host the operator bound that secret to. Any copy of it in the response is replaced with [redacted] before the app sees the response. So a method cannot write the secret into state, where it would replicate to every member.

  1. Declare the URL prefixes in the release’s manifest.json, under outbound (calimero_bundle::BundleManifest::outbound). They are signed with the rest of the manifest and shown to whoever approves them, so declare the narrowest that works. Each is https:// with a host and no credentials, query or fragment; the node refuses a bundle with any other:

    "outbound": [
    "https://challenges.cloudflare.com/turnstile/v0/siteverify",
    "https://api.stripe.com/v1/"
    ]

    https://api.stripe.com/v1/ covers every path below /v1/.

    meroctl app pack writes it from --outbound, once per prefix, and refuses a prefix the node would (not https://, credentials, a query or a fragment):

    Terminal window
    meroctl app pack --wasm app.wasm --package com.example.shop \
    --app-version 1.2.0 --key my-signing-key.json \
    --outbound https://challenges.cloudflare.com/turnstile/v0/siteverify \
    --outbound https://api.stripe.com/v1/ --out dist/com.example.shop-1.2.0.mpk
  2. Make the request from a method that writes, and store the decision:

    use calimero_sdk::action::Request;
    /// Whether Cloudflare Turnstile accepts `token` (`apps/fixture-captcha`).
    pub fn verify_turnstile(&mut self, token: String) -> app::Result<bool> {
    let body = format!(
    "secret={{{{secret:TURNSTILE_SECRET}}}}&response={}",
    form_escape(&token)
    );
    let request = Request::post(TURNSTILE_VERIFY_URL, body)
    .header("Content-Type", "application/x-www-form-urlencoded");
    let Ok(response) = env::fetch(&request) else {
    return Ok(false); // an unreachable verifier is never a pass
    };
    Ok(response.status == 200
    && response.json::<Siteverify>().is_ok_and(|answer| answer.success))
    }
  3. Have the owner approve the prefixes (see approving), and the operator allow the host and supply the secret (see the operator’s side).

The organization’s data is what a call can send, so the organization agrees to the hosts too: the operator allowing a host and the publisher declaring it are not enough. Someone holding MANAGE_APPLICATION org-wide approves with a passkey step-up over the exact change:

POST /orgs/{org}/outbound
{ "package": "com.acme.forms", "prefixes": "manifest" }

"manifest" (the default) approves every prefix the package’s latest release declares; a list approves those. An app that declares no outbound needs and asks for nothing.

  • Per publisher. The approval is keyed by the package’s pinned signer, so a release signed by someone else starts with nothing approved.
  • Per organization. It holds for every app of the package in the organization. An app created without naming an organization is in its owner’s only organization, or one made with it, approved the same way on /orgs/{org}/outbound (there are no per-account approvals since 1.0.0-rc.20).
  • Releases only add. A prefix a later release declares that the owner was never shown answers AwaitingApproval until approved; the rest keep working.
  • Revoke by setting the list. "replace": true sets the publisher’s list exactly; an empty list with it revokes everything. It holds from the next call.
  • Operator-governed clusters. A cluster deployed with --outbound-approvals operator takes approvals from its operator instead, for every app: POST /ops/outbound with the same body, over the Ops API. The owner’s route then answers that the operator holds them.

With meroctl: org outbound <org> --package <pkg> [--prefix <p>]… [--replace], and account outbound --package <pkg> …, which does the same on the account’s only organization. On a development cluster add --dev-passkey <file> to approve the step-up.

An app’s frontend can also ask for the approval on the node’s consent page (POST /consent/requests with kind: outbound), where the person approves with their passkey.

The [outbound] table of merod’s configuration, or of the file --settings names (which replaces it). Give every node of a cluster the same table: the call runs on the app’s leader, whichever node that is.

[outbound]
enabled = true
hosts = ["challenges.cloudflare.com", "api.stripe.com"] # or ["*"]
# Secrets from environment variables, read once at start: a missing one
# fails the start.
[outbound.env_secrets.TURNSTILE_SECRET]
env = "MERO_TURNSTILE_SECRET"
packages = ["com.calimero.mero-forms"]
signers = ["did:key:z6Mk…"] # the package's signer
hosts = ["challenges.cloudflare.com"]

hosts = ["*"] lets every installed app reach the hosts its own signed manifest declares and its owner approved, so a new app needs no config change. Every other gate still holds, and a secret still goes only to the hosts bound to it.

Instead of env_secrets, a node can ask a secrets service beside it (exclusive with env_secrets):

[outbound.sidecar]
url = "http://127.0.0.1:7070/v1/secret"
token_env = "MERO_SECRETS_TOKEN"

The node POSTs {"package", "signer", "name"} with the token as a bearer and expects 200 {"value", "hosts"} or 404. The URL must be http on a loopback address. A managed deployment answers there with the secrets a deployer set for their app, so the secret lives on the nodes that serve the app and nowhere else.

The node logs every request it makes or refuses, with the app and the URL, never the secret, and counts the requests it sends (calimero_app_outbound_requests_total).

A secret is matched on both the package and the release’s signer: anyone can name a package, only its publisher can sign it.

env::fetch needs a component: on the host target it answers an error. In TestApp, mock_http(..) answers it in every run, so a test sets the response it wants and checks what the app decided (Testing).

End to end, use a node built with --features calimero-node/outbound-testing: the scenario runner’s in-process nodes are, and so is cargo build -p merod --features calimero-node/outbound-testing. Its [outbound.testing] table, which a release build refuses, answers .test hosts in-process and takes literal secrets. A scenario gives its nodes that TOML as cluster.settings:

[outbound]
enabled = true
hosts = ["echo.outbound.test"]
[outbound.testing]
echo = true # .test hosts are answered in-process
[outbound.testing.secrets.ECHO_KEY]
value = "s3cr3t-e2e"
packages = ["com.example.my-app"]
signers = ["did:key:z6Mk…"]
hosts = ["echo.outbound.test"]

The echo answers 200 with the request the node sent, as JSON: method, URL, path, query, headers, body and the number of secrets filled in. It answers only after every gate has passed, and its answer is redacted like any other, so a scenario sees exactly what left the node. The node’s cluster tests use it this way: actions_run_on_the_leader_and_reach_only_declared_approved_hosts (declared and approved prefixes, secrets, redaction) and a_call_checks_a_turnstile_token_through_a_captcha_service (a call’s env::fetch through apps/fixture-captcha), both in crates/node/tests/component_cluster.rs:

Terminal window
cargo test -p calimero-node --test component_cluster -- reach_only_declared_approved_hosts

env::fetch works from a call, a timer or a delivered xcall. The request goes out once, from the leader, and the call’s log entry records it with the response, so a retry with the same idempotency key replays the answer instead of fetching again (what a single-use token like Turnstile’s needs). A timeout or a refusal comes back as an error the call handles. Example: apps/fixture-captcha (verifyTurnstile), a captcha service other apps call synchronously (Synchronous xcalls). Protocol detail: Outbound HTTP from component calls.

For work that needs several requests, or a request that should not hold up the call, an #[app::action] runs outside the log on the leader: it fetches with ctx.fetch, reads the app with ctx.view and writes through ctx.call to an #[app::allow(action)] method. A call or timer starts one with env::start_action(method, &args) (Timers and actions).

  • A request never runs in a view. Read the answer back from state.
  • Redaction removes the secret as sent. A service that answers with it encoded (base64, URL-escaped) defeats it, so bind a secret only to hosts that do not echo it.
  • The response is not authenticated beyond TLS: a service you call can answer anything, so call only services you trust for the decision you store.