treewide: add support for building on aarch64-darwin by sespiros · Pull Request #2260 · edgelesssys/contrast

sespiros · 2026-03-17T18:47:57Z

contrast can currently be built from a machine on the x86_64-linux architecture because the only confidential computing runtime architecture it supports is x86_64-linux. Several of the client-side tools do not have any strict requirement to run on x86_64-linux though, which means that their building process can be decoupled from x86_64-linux dependencies. That will enable building contrast from other architectures such as aarch64-darwin.

We achieve this by splitting the package building logic to differentiate between runtimePkgs (packages that build the contrast runtime) and native pkgs (packages that build client-side tooling).

Replaces #2132

Cross-compilation approach and alternatives considered

Why not cross-compilation ?

Cross-compilation has two possible directions, both problematic:

a) Cross-compiling x86_64-linux packages on aarch64-darwin: Produces different store paths than native x86_64-linux builds, different stdenv, different toolchain inputs, different derivation hashes. For runtime components with measured boot and attestation, store paths must match between developer machines and CI. Cross-compilation breaks this.

b) Cross-compiling aarch64-darwin packages on x86_64-linux: nixpkgs does not support cross-compiling to darwin from non-darwin hosts. The nix.dev cross-compilation tutorial states: "macOS/Darwin is a special case, as not the whole OS is open-source. It's only possible to cross compile between aarch64-darwin and x86_64-darwin." The nixpkgs manual platform support table confirms: "Cross compiling is only supported on Darwin hosts" for darwin targets. The release-cross.nix CI matrix reflects this, no linux→darwin targets are tested. Go with CGO_ENABLED=0 is a special case since it has a self-contained compiler and linker that can produce Mach-O binaries without any darwin toolchain.

What I did instead

runtimePkgs in the pkgs scope: Instead of passing runtimePkgs through the overlay chain, I inject it as a pkgs attribute via overlay, following the upstream pattern (pkgs.pkgsCross.*, pkgs.pkgsStatic). The contrast overlay has a standard final: prev: signature and consumers access it as pkgs.runtimePkgs without explicit parameter threading.

Native remote building, not cross-compilation: runtimePkgs is import nixpkgs { system = "x86_64-linux"; }. The Nix daemon delegates builds to a remote x86_64-linux builder. Store paths are identical to CI, binary cache works, attestation hashes match.

Platform overrides: On non-linux systems, overrideScope pulls linux-only packages (kata, kernel, qemu, etc.) from runtimePkgs while rebuilding developer tools (CLI, resourcegen, e2e) natively.

Sets compatibility: All sets (base, debug, badaml-sandbox, etc.) get runtimePkgs via defaultOverlays automatically.

Tests

On an aarch64-darwin laptop with nix-darwin and a x86_64-linux nix remote builder configured:

nix flake check
just

msanft

While skimming this PR, I took the liberty to leave a few comments. Feel free to ignore if they don't apply, since I don't have the full context of this PR. Generally, the runtimePkgs approach sounds good to me.

dev-docs/macos.md

overlays/contrast.nix

treefmt.nix

sespiros · 2026-03-24T21:41:42Z

While skimming this PR, I took the liberty to leave a few comments. Feel free to ignore if they don't apply, since I don't have the full context of this PR. Generally, the runtimePkgs approach sounds good to me.

Thanks for the review and glad you liked the approach (I hit a lot of walls with 3-4 other approaches I've tried)! I will address all the feedback. I've only left one of your comments unaddressed because this is one of the pieces I've added very early 1-2 months ago and I need to remember what was my reasoning.

charludo

Thanks for yourefforts! I have some additional thoughts and questions about the structure of this, let's talk about it at the meeting later. No need to address the comments below before that.

.github/workflows/static.yml

dev-docs/macos.md

overlays/contrast.nix

flake.nix

overlays/contrast.nix

charludo

So after tinkering for a bit too long yesterday, I ultimately came back round to your current approach, with some modifications. I still do not really like having to have the overlay, but the alternatives I've tried out are either brittle or convoluted, so let's go with the current implementation and maybe come back to this in the future.

However, there's some changes I'd like to request, after which IMO this will again be a bit cleaner, and more importantly, easier to handle.

flake.nix

overlays/contrast.nix

packages/containers.nix

packages/scripts.nix

charludo · 2026-04-02T06:20:33Z

overlays/runtimepkgs.nix

+        contrastPkgsStatic = final.runtimePkgs.contrastPkgsStatic.overrideScope (
+          _: _: {
+            kata = final.runtimePkgs.contrastPkgsStatic.kata.overrideScope (
+              _: _: { inherit (cPrev.kata) genpolicy; }
+            );
+          }
+        );


Why is genpolicy here instead of in the kata override below?

First of all in the kata override we specify which packages need to build on linux. genpolicy needs to stay on darwin as it is a binary embedded in the cli and invoked by it.

But actually there is a different issue here. I am doing an override which is a bit hacky so I will push a fixup. The intention was to stop building the static genpolicy binary for darwin because this doesn't work on darwin. For the darwin client we need to be doing a native non-static build for genpolicy.

The mdsh-fmt script only uses mdsh and jq. busybox was an unnecessary dependency that causes build failures on darwin since nixpkgs' darwin busybox is broken. Signed-off-by: Spyros Seimenis <sse@edgeless.systems>

Separate push scripts into container-scripts.nix so container images (linux-only) can be overridden independently from push scripts (which need to run natively on the developer's machine). Signed-off-by: Spyros Seimenis <sse@edgeless.systems>

The let/in binding was needed when containers.nix merged container images with push scripts. Now that push scripts are in a separate file, the wrapper is unnecessary. Signed-off-by: Spyros Seimenis <sse@edgeless.systems>

The contrast runtime (coordinator, initializer, kata, containers) only runs on x86_64-linux, but client-side tools (CLI, resourcegen, e2e) can run on any platform. Separate these by introducing a runtimePkgs overlay that replaces linux-only packages with their x86_64-linux equivalents on non-linux hosts (e.g. aarch64-darwin). This enables building and using the CLI natively on macOS while runtime packages are transparently sourced from x86_64-linux. Signed-off-by: Spyros Seimenis <sse@edgeless.systems>

On linux, the CLI embeds a statically linked genpolicy (musl) for portability across distros. On darwin, static linking of system libraries is not supported (Apple TN2064), and the native dynamically linked build works since macOS system libraries are always present. Select the genpolicy variant at the package level based on stdenv.hostPlatform.isDarwin, removing the need for a contrastPkgsStatic override in the darwin overlay. Signed-off-by: Spyros Seimenis <sse@edgeless.systems>

The renovate package fails to build on darwin due to better-sqlite3 requiring libtool which is missing from the nixpkgs build environment. Since renovate-config-validator only validates a bot config file and the linux CI still checks it, skip it on non-linux systems. Signed-off-by: Spyros Seimenis <sse@edgeless.systems>

Signed-off-by: Spyros Seimenis <sse@edgeless.systems>

The Rust openssl-sys crate resolves OpenSSL on macOS by first checking hardcoded homebrew paths (/opt/homebrew/opt/openssl@3), then falling back to pkg-config. GitHub Actions macOS runners have homebrew pre-installed, so the path exists and openssl-sys commits to it, but the actual headers are inaccessible inside the nix sandbox — causing the build to fail before pkg-config is ever tried. On developer machines without homebrew the path doesn't exist, so openssl-sys falls through to pkg-config which finds nix's OpenSSL. Setting OPENSSL_DIR explicitly bypasses the search heuristic entirely, making the build work consistently regardless of what's installed on the host. Signed-off-by: Spyros Seimenis <sse@edgeless.systems>

Build the darwin CLI and formatter on a macOS runner to catch darwin-specific build failures (broken nixpkgs deps, missing platform guards, etc.). The job depends on cli-build so linux dependencies are available in cachix — the macOS runner only builds the darwin-native parts. Make the setup_nix action cross-platform by gating linux-specific steps (apparmor, btrfs) on runner.os == 'Linux'. Signed-off-by: Spyros Seimenis <sse@edgeless.systems>

sespiros force-pushed the sse/darwin-support branch from db2a688 to b6a5f93 Compare March 17, 2026 18:49

sespiros added the no changelog PRs not listed in the release notes label Mar 17, 2026

sespiros force-pushed the sse/darwin-support branch 9 times, most recently from 285a613 to 37d36da Compare March 24, 2026 17:19

sespiros requested review from charludo and katexochen March 24, 2026 17:31

sespiros marked this pull request as ready for review March 24, 2026 17:31

sespiros mentioned this pull request Mar 24, 2026

release: add cli for aarch64-darwin #2278

Draft

msanft reviewed Mar 24, 2026

View reviewed changes

charludo self-assigned this Mar 25, 2026

charludo requested changes Mar 25, 2026

View reviewed changes

sespiros force-pushed the sse/darwin-support branch 2 times, most recently from aaa2293 to 1bc5f16 Compare March 25, 2026 15:04

charludo requested changes Apr 1, 2026

View reviewed changes

flake.nix Outdated Show resolved Hide resolved

flake.nix Outdated Show resolved Hide resolved

overlays/contrast.nix Outdated Show resolved Hide resolved

packages/containers.nix Outdated Show resolved Hide resolved

packages/scripts.nix Outdated Show resolved Hide resolved

sespiros force-pushed the sse/darwin-support branch from 1bc5f16 to 97aacf0 Compare April 1, 2026 13:16

sespiros requested a review from charludo April 1, 2026 14:23

charludo reviewed Apr 2, 2026

View reviewed changes

sespiros added 5 commits April 2, 2026 12:39

scripts: remove unused busybox from mdsh-fmt runtimeInputs

ed0c65b

The mdsh-fmt script only uses mdsh and jq. busybox was an unnecessary dependency that causes build failures on darwin since nixpkgs' darwin busybox is broken. Signed-off-by: Spyros Seimenis <sse@edgeless.systems>

containers: remove unnecessary let/in wrapper

51280f5

The let/in binding was needed when containers.nix merged container images with push scripts. Now that push scripts are in a separate file, the wrapper is unnecessary. Signed-off-by: Spyros Seimenis <sse@edgeless.systems>

sespiros added 4 commits April 2, 2026 12:43

dev-docs: add instructions for development on macos

b2ed01c

Signed-off-by: Spyros Seimenis <sse@edgeless.systems>

sespiros force-pushed the sse/darwin-support branch from 6d087fc to a0adb80 Compare April 2, 2026 10:04

sespiros requested a review from charludo April 2, 2026 10:05

Conversation

sespiros commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Cross-compilation approach and alternatives considered

Why not cross-compilation ?

What I did instead

Tests

Uh oh!

msanft left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sespiros commented Mar 24, 2026

Uh oh!

charludo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

charludo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

charludo Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

sespiros Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sespiros commented Mar 17, 2026 •

edited

Loading