Release Pipeline

Release taxonomy, CI/CD setup, pipeline flows, policy migration, and post-release verification

Release Classes

Every release falls into one of three classes. Each class has distinct operational rules, tagging conventions, and minimum release-note content.

Stable

Tag: mero-kms-vX.Y.Z or mero-tee-vX.Y.Z

Rules:

Must pass all CI checks including E2E
Must include full release notes
Must include compatibility catalog update
Attestation policy must be finalized (not policy_not_ready)
Requires release-auditor approval

Release notes must include: changelog, breaking changes, upgrade path, MRTD/RTMR changes, compatible counterpart versions.

Release Candidate (RC)

Tag: mero-kms-vX.Y.Z-rc.N or mero-tee-vX.Y.Z-rc.N

Rules:

Must pass CI checks; E2E may be advisory
Policy may be policy_not_ready
Not for production deployment
Compatibility catalog update is optional
Release notes can be abbreviated

Purpose: Pre-production testing, measurement collection, integration validation.

Hotfix

Tag: mero-kms-vX.Y.(Z+1) (patch bump)

Rules:

Branches from the stable tag being patched
Minimal diff — security/critical fixes only
Must pass all CI checks including E2E
Must explain why a patch release was necessary
If MRTD changes, must update policy and catalog

Release notes must include: CVE or incident reference, exact change description, rollback instructions.

CI/CD Setup

The release pipelines require specific GitHub repository configuration. Variables are set at the repository or environment level; secrets are stored in GitHub Secrets.

Required GitHub Variables

Environment Variables Table

MERO_KMS_VERSION

Current KMS release version (e.g., 1.2.0). Used by policy generation and compatibility checks.

MERO_KMS_PROFILE

Target profile for this release: debug, debug-read-only, or locked-read-only.

MERO_TEE_VERSION

Current node-image release version. Used for cross-family compatibility validation.

GCP_PROJECT_ID

Google Cloud project for VM image storage and TDX instance creation.

GCP_ZONE

Target zone for TDX-capable GCP instances (e.g., us-central1-a).

GCP_IMAGE_FAMILY

GCP image family prefix for node images.

GHCR_REGISTRY

GitHub Container Registry path for KMS Docker images.

PHALA_DSTACK_URL

URL for the Phala dstack instance used in E2E tests.

ITA_API_URL

Intel Trust Authority API endpoint for attestation verification in tests.

COMPATIBILITY_CATALOG_PATH

Path in the repo to the compatibility catalog JSON file.

Required Secrets

GCP_SERVICE_ACCOUNT_KEY

Service account JSON key for GCP operations (image creation, instance management). Must have compute.images.create, compute.instances.* permissions.

GHCR_TOKEN

GitHub token with packages:write scope for pushing Docker images to GHCR.

ITA_API_KEY

Intel Trust Authority API key for quote verification during E2E tests.

COSIGN_PASSWORD

Password for Sigstore keyless signing (used by cosign in OIDC mode).

Base Image Notes

Node images are built on top of a GCP-provided TDX-capable base image. The base image version determines RTMR0 (firmware) and RTMR1 (kernel). When Google updates the base image, RTMR0/RTMR1 values change and a new release is needed to publish updated measurements.

// Base image selection in Packer config:
source_image_family = "ubuntu-2404-tdx"
source_image_project = "confidential-vm-images"

// Pin to a specific image for reproducibility:
source_image = "ubuntu-2404-tdx-v20250101"

Triggers & Guards

Multiple guards prevent broken or inconsistent releases from being published. Guards run as required status checks or as early steps in the release workflows.

PR Doc Guard

Runs on every PR that modifies release-related files. Ensures that changes to attestation policy, compatibility catalog, or measurement files include corresponding documentation updates. Blocks merge if docs are missing.

# Triggered by changes to:
paths:
  - 'attestation-policy-*.json'
  - 'published-mrtds.json'
  - 'compatibility-catalog.json'
  - 'scripts/release-*.sh'

Release Version Sync Guard

Runs at the start of every release workflow. Validates that the tag version matches MERO_KMS_VERSION or MERO_TEE_VERSION in the repository variables. Prevents releasing with stale version references.

# Fails if:
# tag = mero-kms-v1.3.0 but MERO_KMS_VERSION = 1.2.0
# tag = mero-tee-v2.0.0 but MERO_TEE_VERSION = 1.9.0

KMS ↔ Node Dependency Guard

Validates cross-family compatibility. When a KMS release is triggered, verifies the referenced node-image version exists and is compatible. When a node-image release is triggered, verifies the KMS version compatibility.

# The guard checks:
# 1. The counterpart release tag exists
# 2. The compatibility catalog lists the pair
# 3. The attestation policy references valid MRTDs for the counterpart

ALLOWED_* Forbidden in Release

Release workflows must not set ALLOWED_MRTD, ALLOWED_RTMR0, etc. as environment variables. These are generated from the release process, not hardcoded. The guard scans workflow env for any ALLOWED_* variable and fails if found.

KMS Release Pipeline

Workflow: release-kms-phala.yaml. Triggered by pushing a tag matching mero-kms-v*.

Node-Image Release Pipeline

Workflow: release-node-image-gcp.yaml. Triggered by pushing a tag matching mero-tee-v*.

Key Difference: Measurement Collection

Unlike the KMS pipeline (where measurements come from Docker image inspection), the node-image pipeline must boot an actual TDX VM to collect measurements. Step 3 creates an ephemeral GCP Confidential VM instance from the newly built image, extracts MRTD/RTMR values from a TDX quote, then terminates the instance.

Release Auditor

The release-auditor is a post-publish workflow step that validates the completeness and integrity of a published release. It runs automatically after assets are uploaded.

Auditor Checks

Asset Completeness

Verifies all expected files are present in the GitHub release. Missing files fail the audit. Expected file list varies by release family (KMS vs node-image).

Signature Validity

Re-verifies every Sigstore bundle in the release. Confirms the signing identity matches the expected workflow and repository.

Checksum Consistency

Downloads each asset and verifies its SHA-256 hash against checksums.sha256. Catches upload corruption or partial uploads.

Policy Validation

For KMS releases: parses each attestation-policy-{profile}.json and validates the JSON schema, ensures all MRTD/RTMR fields are present, and checks that allowed_tcb_status is not empty.

Post-Release E2E

End-to-end tests verify the full attestation and key-release flow using the actual released artifacts. Two test suites run independently.

mero-tee-node E2E

Deploy Node

Create a GCP Confidential VM using the newly released image.

Boot Verification

Wait for the node to boot and verify it can produce a TDX quote.

KMS Attestation

Node verifies the KMS (Plane 1) using POST /attest.

Key Release

Node completes the challenge/get-key flow and verifies it receives a valid encryption key.

Cleanup

Terminate the ephemeral VM instance.

KMS-Node E2E Guardrails

Cross-Version Test

If the release includes a compatibility catalog update, test the new KMS version against all listed compatible node-image versions (and vice versa).

Profile Isolation Test

Deploy a debug-profile node and a locked-read-only KMS policy. Verify the key release is rejected with 403 PolicyViolation.

Stale Policy Test

Deploy the new node-image with an old KMS policy. Verify it fails (MRTD mismatch). Then update the policy and verify it succeeds.

Measurement Consistency

Compare the MRTD/RTMR values observed during E2E against published-mrtds.json from the release. Any mismatch is a release blocker.

Compatibility Catalog

The compatibility catalog is a version-controlled JSON file that records which KMS versions work with which node-image versions. It is the source of truth for deployment compatibility.

Structure

{
  "schema_version": "1",
  "entries": [
    {
      "kms": "1.2.0",
      "node_image": "1.2.0",
      "profiles": ["debug", "debug-read-only", "locked-read-only"],
      "tested": true,
      "notes": "Initial paired release"
    },
    {
      "kms": "1.2.0",
      "node_image": "1.1.0",
      "profiles": ["locked-read-only"],
      "tested": true,
      "notes": "Backward compat — node 1.1.0 MRTDs in KMS 1.2.0 policy"
    }
  ]
}

Update Process

New Release Triggers Update

When a new KMS or node-image release passes E2E, the pipeline adds entries to the catalog for all tested version pairs.

PR with Catalog Change

The pipeline opens a PR updating compatibility-catalog.json. The PR doc guard ensures release notes mention the new compatibility entries.

Manual Review

A maintainer reviews the compatibility entries, verifies E2E results, and merges.

KMS Policy Operations

Attestation policy files control which nodes can obtain keys from the KMS. They are profile-scoped and follow a strict promotion path.

Profile-Scoped Policy Files

// Policy files in a KMS release:
attestation-policy-debug.json
attestation-policy-debug-read-only.json
attestation-policy-locked-read-only.json

// Each contains allowed measurements for ONE profile only.
// A KMS instance loads exactly one policy file at startup.

Promotion Path

RC Release

Policy files are generated but may be marked policy_not_ready. This is expected — measurements may not be finalized.

Measurement Stabilization

Once E2E tests confirm stable MRTD/RTMR values, the policy is updated with actual measurements. The policy_not_ready flag is removed.

Stable Release

The finalized policy is included in the stable release tag. It contains only verified measurements from successful E2E runs.

Production Deployment

Operators deploy the stable KMS with the locked-read-only policy. The policy URL and optional SHA-256 pin are set in the KMS environment.

Release Artifacts

Every stable release includes the following artifacts, produced by modular scripts within the pipeline.

SBOM

Software Bill of Materials in SPDX JSON format. Generated by syft from the Docker image (KMS) or Packer manifest (node-image). Lists all packages, libraries, and their versions.

Auto-Generated Release Notes

Git changelog since the last release tag. Includes commit messages, PR references, and contributor attributions. Supplemented with manually written breaking changes and upgrade instructions.

Checksums

checksums.sha256 — SHA-256 hashes of every release asset. The checksum file itself is signed by Sigstore, creating a single file that anchors integrity of all assets.

Sigstore Signatures

Keyless signing via Fulcio + Rekor. Each .sigstore.json bundle contains the certificate (with embedded OIDC claims), signature, and Rekor transparency log entry. Verification uses cosign verify-blob.

Modular Scripts

Release workflows delegate work to focused bash scripts under scripts/. Each script has a single responsibility and follows shared conventions.

Script Organization

KMS Scripts

build-kms-image.sh

Build and tag the KMS Docker image. Accepts version and registry args.

generate-attestation-policy.sh

Generate per-profile attestation policy JSON from MRTD inputs.

compute-compose-hash.sh

Normalize and hash the Docker Compose file.

publish-kms-release.sh

Upload all assets to a GitHub release tag.

verify-release-assets.sh

Post-publish verification of signatures and checksums.

Node-Image Scripts

build-node-image.sh

Run Packer with Ansible provisioner for the target profile.

collect-measurements.sh

Boot ephemeral TDX VM, extract quote, parse MRTD/RTMRs.

publish-node-release.sh

Upload measurements and SBOM to a GitHub release tag.

generate-merod-kms-phala-attestation-config.sh

Generate client-side attestation config for merod.

Logging Conventions

# All scripts use structured log output:
[INFO] Informational messages (progress, status)
[WARN] Non-fatal issues (missing optional files, retries)
[ERROR] Fatal errors (script exits with non-zero)
[PASS] Verification passed
[FAIL] Verification failed

# Scripts exit with meaningful codes:
0 = success
1 = general error
2 = usage error (bad args)
10 = verification failure
11 = missing required asset

KMS Policy Migration

When upgrading the KMS or changing profiles, the attestation policy must be updated. This section covers the required environment variables and migration constraints.

Required Environment Variables

MERO_KMS_VERSION

The KMS version this policy targets. Must match the release tag exactly (e.g., 1.2.0, not v1.2.0).

MERO_KMS_PROFILE

The profile this KMS instance serves: debug, debug-read-only, or locked-read-only. Determines which policy file is loaded.

MERO_KMS_POLICY_URL

URL to the attestation policy JSON. Typically a GitHub release asset URL. The KMS fetches this on startup.

MERO_KMS_POLICY_SHA256

Optional. SHA-256 hash of the policy file for pinning. If set, the KMS rejects any policy file that doesn't match this hash.

Forbidden: ALLOWED_* in Release Workflows

Release workflows must never set ALLOWED_MRTD, ALLOWED_RTMR0, ALLOWED_RTMR1, ALLOWED_RTMR2, ALLOWED_RTMR3, or ALLOWED_TCB_STATUS as environment variables. These are legacy overrides that bypass the structured policy file.

# FORBIDDEN in any release workflow:
ALLOWED_MRTD=...         // Use attestation-policy-{profile}.json instead
ALLOWED_RTMR0=...        // Measurements come from the release process
ALLOWED_TCB_STATUS=...    // TCB allowlist is in the policy file

# The release guard scans for these and fails the workflow if found.

policy_not_ready Behavior

During RC releases, the policy file may contain a "status": "policy_not_ready" flag. When the KMS loads a policy with this flag:

RC Behavior

The KMS starts but logs a warning on every key request. Attestation verification is relaxed — all MRTD/RTMR values are accepted. This allows testing the key-release flow before measurements are finalized.

Stable Must Not Have This

A stable release with policy_not_ready is a release blocker. The release auditor checks for this flag and fails if present in any stable release's policy file.

Migration Steps

Identify Target Versions

Determine the KMS version and node-image version pair. Verify they appear in the compatibility catalog.

Download New Policy

Fetch the attestation-policy-{profile}.json from the target KMS release. Verify its Sigstore signature.

Update KMS Config

Set MERO_KMS_VERSION, MERO_KMS_PROFILE, and MERO_KMS_POLICY_URL to point to the new policy. Optionally set MERO_KMS_POLICY_SHA256.

Rolling Restart

Restart the KMS. On startup it fetches the new policy and applies it. Existing node connections are unaffected until they re-attest.

Verify Post-Migration

Run POST /attest against the restarted KMS and verify it returns a valid quote. Then test a key release from a node running the compatible image.