Chain-of-trust for Nox - Part 2 : Runtime Attestation UI

About the Author

Aghiles Ait Messaoud is a PhD, Research Software Engineer working at iExec. He specializes in distributed systems and confidential computing. He currently works at improving the trustworthiness of iExec platform by leveraging TEE capabilities. You can contact him at aghiles.ait-messaoud@iex.ec for any question regarding the article.

Introduction

Part 1 of this series established the foundation of the Nox chain of trust: the boot-time verification that prevents a Confidential Virtual Machine (CVM) from obtaining any secret unless it has booted a known operating-system image and a known application stack on attested Intel TDX hardware. That guarantee, however, is enforced internally by the platform at boot time; on its own, it gives an end user or an external auditor no direct way to check, at a later moment, that a live Nox service is still the exact workload that was attested.

This document is the second part of the series and addresses that gap: runtime attestation, the ability for any party to verify, at any point during a CVM's lifetime, that a given Nox component is genuinely running the expected code inside a genuine Intel TDX Trust Domain (TD). We describe (i) how a fresh TDX quote is requested from a running CVM and what it contains; (ii) the components deployed around the TDX fleet to discover the running CVMs and expose their quotes; (iii) the end-to-end sequence and the user-facing interface that surface this evidence; and (iv) the step-by-step protocol by which a quote is verified, from its signature down to the docker-compose actually executed.

It builds directly on the TEE foundation described in Part 1 and deliberately limits its scope to runtime attestation as surfaced through the Runtime Attestation UI. The remaining links of the chain of trust, source-code binding and on-chain governance of the authorized measurements, are treated in later parts.

The remainder of this document is organised as follows. In Background, we recall what runtime attestation is, how a TDX quote is structured, and the three quote fields Nox relies on. In Architectural deployment, we describe the components deployed around the TDX servers and the data they exchange. In Reference environment, we pin the versions of the components this article is based on. In Sequence Diagram, we trace the end-to-end flow by which the UI discovers the running CVMs and attests them. In User Interface Presentation, we present the interface and its three levels of attestation. In Attestation steps of a Nox component, we detail the verification protocol applied to a single CVM. Finally, in Future Work, we outline the planned improvements: image provenance, user-supplied challenges, and additional quote verifiers.

Background

In this section, we review the concepts this document builds on: what runtime attestation is, how a TDX quote is structured, and the three quote fields Nox relies on, namely report_data, the RTMRs, and their associated event logs.

Runtime Attestation

Runtime attestation is the mechanism that lets any remote party verify, on demand and at any point during a CVM's lifetime, that a Nox component is genuinely running inside a real Intel TDX Trust Domain (TD) with the expected code and configuration. Where boot-time attestation (see Part 1) establishes the chain of trust from the hardware up to dstack-OS, runtime attestation exposes that trust to external verifiers through a simple challenge-response protocol.

TDX quote

A TDX quote is a cryptographically signed structure produced by the platform's Quoting Enclave. It carries the evidence a verifier needs to trust a TD and is signed with an Intel-provisioned attestation key, so that anyone can validate its authenticity by walking the certificate chain back to Intel's root of trust. The three fields Nox relies on for runtime attestation (report_data, the RTMRs, and their associated event logs) are detailed below.

report_data

report_data is a 64-byte field whose content is entirely chosen by the workload. TDX copies it verbatim into the signed quote, which lets an application bind arbitrary data to the hardware attestation. Common uses are:

  • Freshness / challenge-response: embedding a verifier-supplied nonce, as the Runtime Attestation UI does, to prove the quote was generated on demand.
  • Identity binding: embedding the hash of a public key or TLS certificate (the basis of RA-TLS, see Part 1), so that a secure channel is provably terminated inside the TD.
  • Other proofs, such as blockchain wallet addresses or application-state hashes.

In Nox, the UI generates a random challenge and expects to find exactly that value in the quote's report_data.

RTMRs

Runtime Measurement Registers (RTMRs) are the TDX equivalent of TPM PCRs: append-only registers that cannot be written directly, only extended through a hash chain. A TD exposes four of them, and dstack assigns each a well-defined role:

  • RTMR0: the virtual hardware / firmware environment.
  • RTMR1: the Linux kernel.
  • RTMR2: the kernel command line and initrd.
  • RTMR3: application-specific measurements: app-id, os_image_hash, compose-hash, instance-id, and key-provider.

The initial memory contents and configuration of the TD are captured separately in MRTD. During verification, RTMR0–RTMR2 (together with MRTD) attest the integrity of the boot chain, while RTMR3 attests that the expected application code and configuration are running. In Nox we focus on RTMR3, since it ties the quote to the specific Nox component and its docker-compose.

event logs

An RTMR value is an opaque hash: it attests whether the expected measurements were folded in, but not what they were. The event log is the human-readable record of the individual measurement events that were extended into an RTMR (in practice, RTMR3). Each event is a key-value measurement such as app-id, compose-hash, instance-id, or key-provider, and every event is folded into the register with the extension formula:

RTMR3_new = SHA384(RTMR3_old || SHA384(event_log))

A verifier replays the event log through this formula and compares the recomputed register to the rt_mr3 value inside the signed quote. If they match, the individual event values, notably os_image_hash and compose_hash, can be trusted as faithful representations of the OS image and docker-compose actually executed by the CVM.

Architectural deployment

In this section, we describe the components deployed around the TDX servers to make the Runtime Attestation UI work, and we illustrate the shape of the data they exchange.

Runtime Attestation UI architectural deployment for Nox

To make the Runtime Attestation UI work, five main components collaborate:

  • Runtime Attestation UI portal: The interface accessed by the user to visualize the Runtime Attestation process. It is already deployed on the iExec side at NOX · Chain of Trust, but can also be rebuilt and deployed on personal computers using its source code GitHub - iExec-Nox/nox-attestation-portal. The UI requests nox-cvms-exporter-aggregator to get the list of active CVMs together with, for each one, its freshly fetched quote and docker-compose. The aggregator itself contacts each CVM dstack-quote-service, so the UI never talks to the CVMs directly.
  • nox-cvms-exporter: a host service that connects to the dstack hypervisor (dstack-vmm) to collect the list of active CVMs (their dstack-quote-service url) on the local TDX machine and sends it to the nox-cvms-exporter-aggregator.
  • nox-cvms-exporter-aggregator: a service deployed on Azure Kubernetes Service that aggregates the list of active CVMs received from each individual nox-cvms-exporter. For every listed CVM, it then contacts the CVM dstack-quote-service to fetch its fresh quote (bound to the UI challenge) and its docker-compose, and returns the enriched list to the Runtime Attestation UI portal.
  • Proof of Cloud Trust Server: iExec server for Proof of Cloud (GitHub - proofofcloud/trust-server ) to check whether a quote is issued from a whitelisted TDX machine and not revoked. Details of the Proof of Cloud whitelisting process are documented in Part I of our article series.
  • Phala PCCS: Phala local cache to store collaterals (TCB Infos, Intel PCK certificates and certificates revocation list) to verify a quote.

Reference environment

In this section, we pin down the exact versions of the components involved in the Runtime Attestation UI, so that the behaviour described in the rest of this article can be reproduced and, as these components evolve, compared against a known baseline.

Sequence Diagram

In this section, we trace the end-to-end sequence by which the Runtime Attestation UI discovers the running CVMs and attests them, from opening the portal to displaying the result.

This diagram describes how the Runtime Attestation UI discovers the Confidential VMs (CVMs) running across the TDX fleet and attests them.

  1. Open portal: The user opens the portal, either directly through the iExec-hosted instance NOX · Chain of Trust or by self-building and running the source repository GitHub - iExec-Nox/nox-attestation-portal. On load, the UI generates a single random challenge (32 bytes) that is expected to be bound on the gathered quotes.
  2. Discovery request: The UI calls the aggregator GET /cvms endpoint, passing its challenge as a query parameter (?challenge=...). This parameter is mandatory since the challenge must be relayed to the CVMs to bind the returned quotes.
  3. Fan-out to exporters: For each TDX machine in its configured exporter list, the aggregator calls GET {base_url}/cvms in parallel. The challenge is not forwarded at this stage; it is only used later, when fetching the quotes.
  4. Local CVM enumeration: Each exporter queries its local dstack-vmm (POST /prpc/Status?json) to list the VMs running on that machine.
  5. VMM response: dstack-vmm returns the raw VM list. The exporter filters out VMs whose status is stopped/removed and excludes the kms and dstack-gateway system CVMs.
  6. Exporter response: The exporter builds each CVM quote-service base URL and returns the CVMs grouped by app_id , each instance carrying { instance_id, url, machine_id }.
  7. Quote request (enrichment): The aggregator enriches each instance with its fresh quote: for every instance, the aggregator calls the CVM quote-service GET {url}/quote?data={challenge}, relaying the UI challenge so the returned quote is bound to it.
  8. Quote response: The quote-service returns { quote, event_log } since the UI needs the quote for the signature check and the event log for the RTMR3 replay.
  9. Manifest request: Concurrently with the quote request, the aggregator calls the same CVM quote-service GET {url}/info to retrieve the deployment manifest.
  10. Manifest response: The quote-service returns its /info payload, from which the aggregator extracts the docker-compose manifest (tcb_info.app_compose).
  11. Merge & respond: The aggregator regroups the enriched instances by app_id and returns the merged list to the UI, each instance now carrying { instance_id, machine_id, quote: { quote, event_log }, app_compose }. The url field is kept internal to the aggregator and is never exposed to the UI, so the browser has no way to reach the CVMs directly. The JSON file below illustrates an example of an aggregated entry sent by the nox-cvms-exporter-aggregator to the attestation UI.
  1. Verify & display: For each CVM returned, the UI verifies the attestation locally and displays the CVMs together with the result. This verification protocol is detailed in the Attestation steps of a Nox component section.

User Interface Presentation

In this section, we present the Runtime Attestation UI and the three levels of attestation it exposes to the user.

Runtime Attestation UI as displayed in NOX · Chain of Trust

The screenshot above shows the Runtime Attestation UI. On load, the left-hand panel lists the Nox components running on the testnet TDX machines, each annotated with its number of replicas (e.g., 6 for nox-kms). The interface offers three levels of Nox component attestation:

  1. Full verification: Once clicked on the Verify all button, all the CVM instances of all Nox components will be verified.
  2. Component-level verification: This level becomes available once a Nox component is chosen on the left-side panel. Once clicked on Verify all, all the CVM instances of the chosen Nox component will be verified regardless of their underlying machine.
  3. Instance-level verification: This level also becomes available once a Nox component is chosen on the left-side panel. Once clicked on Verify, only the specific instance of the chosen Nox component will be verified.

Attestation steps of a Nox component

In this section, we detail the step-by-step verification protocol applied to a single Nox component CVM, and we describe what the UI displays once every check has passed.

Runtime Attestation workflow for a Nox component

The verification protocol applied to a Nox component CVM proceeds as follows:

  1. Get fresh quote: the UI generates a random challenge and obtains, through the aggregator, a fresh quote bound to that challenge (see Sequence Diagram). The challenge is expected to be embedded in the quote’s report_data.
  2. Check quote signature and certificate chain: the quote signature and its certificate chain up to Intel are validated by a DCAP verifier. In the current implementation this runs locally in the browser through an embedded Phala dcap-qvl (quote verification library: GitHub - Phala-Network/dcap-qvl). The qvl fetches the verification collateral (the TCB information and the Intel-rooted certificate chain) from the dstack PCCS (Phala-hosted Provisioning Certificate Caching Service) and performs the check itself. In parallel, the quote is submitted to the Proof of Cloud trust server, which reports whether the attested machine belongs to a whitelisted (certified) cloud fleet. This proof-of-cloud result is informative and non-blocking: if the trust server is unavailable or times out, the attestation still proceeds.
  3. Check quote freshness: the quote’s freshness is confirmed by verifying that the random UI-generated challenge is embedded in the quote’s report_data.
  4. Display RTMR values: an informative step that extracts and displays the RTMR values of the quote.
  5. Replay RTMR3: the event logs returned alongside the quote (in the aggregator response) are used to replay RTMR3 following Intel’s formula: RTMR3_new = SHA384(RTMR3_old || SHA384(event_log)). If the replayed RTMR3 matches the attested one (in the quote), the event-log values os_image_hash and compose_hash can be trusted as representative of the executed OS and docker-compose.
  6. Check OS image: the os_image_hash RTMR3 event log is extracted, and a download link is provided so that the image and its hash can be checked manually if needed. At iExec, this image is dstackOS.
  7. Check compose-hash: the docker-compose (app_compose, provided in the aggregator response) is hashed, and the result is compared to the compose_hash embedded as an RTMR3 event log.

Once these verification steps are performed, the docker-compose corresponding to the compose-hash of the attested CVM is displayed as shown in the image below.

Part of nox-kms docker-compose 

The docker-compose of the CVM encompasses three services:

  • nox-component: the business service executed in the CVM (e.g., nox-kms)
  • quote-service: the service used to obtain the quote and the event logs required to attest the CVM
  • fluent-bit: iExec log exporter service for internal observability

Future Work

In this section, we outline the planned improvements to the attestation flow: image provenance, user-supplied challenges, and additional quote verifiers.

Image provenance

As of now, the attestation stops when we display the docker compose that lists the images executed in the attested CVM. A next step is to leverage Sigstore infrastructure to sign the GitHub Actions workflow that generated each image, during its build, and publish the SLSA (Supply-chain Levels for Software Artifacts) attestation identified by the sha256 checksum of the image (subject-name) in GitHub and the docker registry of the image. A Rekor entry will also naturally be pushed. By doing so, we will be able to extend the verification steps mentioned in this article by attesting the provenance of each Nox image (i.e., build workflow and GitHub repo commit).

User-input challenge

Currently, to test the freshness of a quote, we use an auto-generated challenge that is transmitted from the UI to the aggregator as a query parameter, which then relays it to each quote-service. We will strengthen this process by allowing UI users to enter their own challenge to get an additional level of trust in the freshness of the quote.

Multiple verifiers

Currently, we use Phala qvl as the sole verification method to check the Intel signature and certificate chain of a quote. We will add additional verification methods, either as backups or as mandatory checks if available, to improve trust toward the verification result. The potential verifiers to add are iExec-hosted dstack-verifier and Intel Trust Authority.