Skip to content

Add GitHub workflow for cross-compilation and fix build issues for macOS and FreeBSD#636

Merged
slp merged 11 commits intocontainers:mainfrom
nohajc:main
Apr 24, 2026
Merged

Add GitHub workflow for cross-compilation and fix build issues for macOS and FreeBSD#636
slp merged 11 commits intocontainers:mainfrom
nohajc:main

Conversation

@nohajc
Copy link
Copy Markdown
Contributor

@nohajc nohajc commented Apr 14, 2026

I figured it would be a good idea to run automated builds for our cross-compilation scenarios. So the new pull request workflow tests the following combinations:

  1. macOS host, Linux init
  2. macOS host, FreeBSD init
  3. Linux host, FreeBSD init

Of course I immediately found out my newly merged FreeBSD init port didn't compile, so the fix is included (it was just a broken Makefile rule).

I also tried to fix various issues with building unit tests on macOS which popped up in my VS Code. Most of the changes were suggested by AI with the following explanation:

test_get_supported_cpuid — x86_64-specific, removed entirely
Vm::supported_cpuid() is a KVM concept (fetching the CPUID leaf table the hypervisor supports). HVF on ARM has no such thing. The method doesn't exist on the macOS Vm struct.

test_setup_irqchip — KVM concept, removed entirely
The KVM Vmm had an explicit setup_irqchip() call to configure the in-kernel interrupt controller before vCPUs could be created. HVF doesn't have this: the GIC is set up differently and is not part of Vm. The method simply doesn't exist here.

test_kvm_context — KVM concept, removed entirely
KvmContext is a wrapper around the /dev/kvm file descriptor. HVF has no equivalent — Vm::new(nested_enabled) talks to Hypervisor.framework directly via dlopen. There's nothing to test here.

test_vcpu_kick + test_vcpu_rtsig_offset — signal-based mechanism, removed entirely
On Linux/KVM, to interrupt a vCPU that's running inside KVM_RUN, a real-time signal (SIGRTMIN + offset) is sent to the vCPU thread, and the KVM signal handler sets kvm_run.immediate_exit = 1. The test verified this entire signal delivery path by:

Spawning a thread, starting it in a KVM inner loop
Sending the RTSIG from the outer thread
Checking that kvm_run.immediate_exit was set
On macOS/HVF the entire mechanism is stubbed out — register_kick_signal_handler() is an empty function, KvmRunWrapper doesn't exist on HVF, and vcpu.fd doesn't exist. Interrupting an HVF vCPU is done through hvf::vcpu_request_exit() instead.

test_vm_save_restore_state / test_vcpu_save_restore_state — x86_64 only, gated already
These were already #[cfg(target_arch = "x86_64")] and test KVM save/restore state (PIT, IOAPIC, MSRs, CPUID). None of that exists on the ARM64/HVF path so they just couldn't compile.

What changed inside test_vcpu_tls
The original test had three extra assertions using a run_on_thread_local() method that doesn't exist in this codebase's Vcpu:

    assert!(Vcpu::run_on_thread_local(|_| ()).is_err());
run_on_thread_local was a KVM-era helper that accepted a closure and ran it against the TLS-stored Vcpu pointer. It was never ported to the HVF Vcpu. The simplified version keeps the meaningful assertions — that you can't double-init TLS, that you can reset it, and that you can't reset it twice — while using the methods that actually exist (init_thread_local_data / reset_thread_local_data).

Comment thread .cargo/config.toml
Copy link
Copy Markdown
Contributor Author

@nohajc nohajc Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The runner configuration lets me run unit tests from VS Code editor.

@nohajc nohajc force-pushed the main branch 2 times, most recently from cfae6d8 to b5dc66c Compare April 14, 2026 22:45
Comment thread .cargo/macos-test-runner.sh Outdated
name: Cross-compilation (macOS)
runs-on: macos-latest
env:
DYLD_LIBRARY_PATH: /Library/Developer/CommandLineTools/usr/lib/
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@slp I had issues building the macOS version which looked like this:
slp/homebrew-krun#14

Setting DYLD_LIBRARY_PATH fixed it for the GitHub action. Maybe the krun_display patch for homebrew is not needed.

Comment thread .cargo/macos-test-runner.sh Outdated
Comment thread .cargo/macos-test-runner.sh
@nohajc nohajc force-pushed the main branch 2 times, most recently from 02a1a18 to 40f295f Compare April 15, 2026 14:03
@mtjhrc
Copy link
Copy Markdown
Collaborator

mtjhrc commented Apr 15, 2026

Please try to make sure the commit messages do what the title and commit msg text says.

Like github actions: add cross-compilation workflow commit also introduces Linux -> FreeBSD cross-compilation.
I would expect 2 commits:
Makefile: Add Linux -> FreeBSD init cross-compilation support
CI: Add init cross-compilation workflow

build fixes for macOS is very vague and there is no commit message text besides titile...

Comment thread src/vmm/build.rs
@nohajc
Copy link
Copy Markdown
Contributor Author

nohajc commented Apr 15, 2026

@mtjhrc Split commits and improved messages.

Comment thread src/libkrun/build.rs
@mtjhrc
Copy link
Copy Markdown
Collaborator

mtjhrc commented Apr 16, 2026

There are still cargo test failures. Also I need to specify CC_LINUX env var otherwise cargo test doesn't even run.
Do you have that, or how are you using it?

You should probably also disable these failing tests:

test legacy::i8042::tests::test_i8042_kbd ... FAILED
test legacy::i8042::tests::test_i8042_read_write_and_event ... FAILED
$ CC_LINUX=... cargo test
$ CC_LINUX="/usr/bin/clang -target aarch64-linux-gnu -fuse-ld=lld --sysroot $(pwd)/linux-sysroot -B$(pwd)/linux-sysroot/usr/lib/gcc/aarch64-linux-gnu/12 -L$(pwd)/linux-sysroot/usr/lib/gcc/aarch64-linux-gnu/12 -Wno-c23-extensions" cargo test
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.04s
     Running unittests src/lib.rs (target/debug/deps/arch-7e7fa7be3e6801ce)
/Users/mhrica/Dev/libkrun/c/main/target/debug/deps/arch-7e7fa7be3e6801ce: replacing existing signature

running 2 tests
test aarch64::tests::test_regions_gt_1024gb ... ok
test aarch64::tests::test_regions_lt_1024gb ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/arch_gen-f9e1e70fc02b4a7c)
/Users/mhrica/Dev/libkrun/c/main/target/debug/deps/arch_gen-f9e1e70fc02b4a7c: replacing existing signature

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/aws_nitro-642edc16ad499ce2)
/Users/mhrica/Dev/libkrun/c/main/target/debug/deps/aws_nitro-642edc16ad499ce2: replacing existing signature

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/cpuid-7d75ebe5abbc11cd)
/Users/mhrica/Dev/libkrun/c/main/target/debug/deps/cpuid-7d75ebe5abbc11cd: replacing existing signature

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/devices-ad2d2385014d2cb8)
/Users/mhrica/Dev/libkrun/c/main/target/debug/deps/devices-ad2d2385014d2cb8: replacing existing signature

running 51 tests
test legacy::i8042::tests::test_i8042_commands ... ok
test legacy::rtc_pl031::tests::test_rtc_read_write_and_event ... ok
test bus::tests::busrange_cmp_and_clone ... ok
test bus::tests::bus_read_write ... ok
test bus::tests::bus_read_write_values ... ok
test bus::tests::test_display_error ... ok
test legacy::i8042::tests::test_i8042_buffer ... ok
test bus::tests::bus_insert ... ok
test virtio::console::port_queue_mapping::test::test_queue_idx_to_port_id_ok ... ok
test virtio::console::port_queue_mapping::test::test_port_id_to_queue_idx ... ok
test virtio::console::port_queue_mapping::test::test_queue_idx_to_port_id_panic_rx_control - should panic ... ok
test legacy::i8042::tests::test_i8042_kbd ... FAILED
test legacy::i8042::tests::test_i8042_read_write_and_event ... FAILED
test virtio::console::port_queue_mapping::test::test_queue_idx_to_port_id_panic_tx_control - should panic ... ok
test virtio::descriptor_utils::tests::reader_test_simple_chain ... ok
test virtio::descriptor_utils::tests::reader_test_incompatible_chain ... ok
test virtio::descriptor_utils::tests::read_full ... ok
test virtio::descriptor_utils::tests::reader_writer_shared_chain ... ok
test virtio::descriptor_utils::tests::reader_writer_shattered_object ... ok
test virtio::descriptor_utils::tests::split_beginning ... ok
test virtio::descriptor_utils::tests::reader_unexpected_eof ... ok
test virtio::descriptor_utils::tests::split_end ... ok
test virtio::descriptor_utils::tests::split_border ... ok
test virtio::descriptor_utils::tests::split_outofbounds ... ok
test virtio::descriptor_utils::tests::split_middle ... ok
test virtio::descriptor_utils::tests::writer_test_incompatible_chain ... ok
test virtio::descriptor_utils::tests::write_full ... ok
test virtio::fs::multikey::test::get ... ok
test virtio::descriptor_utils::tests::writer_test_simple_chain ... ok
test virtio::fs::multikey::test::remove ... ok
test virtio::fs::multikey::test::update_alt_key ... ok
test virtio::fs::multikey::test::update_both_keys_alt ... ok
test virtio::fs::multikey::test::update_both_keys_main ... ok
test virtio::fs::multikey::test::update_main_key ... ok
test virtio::fs::multikey::test::update_value ... ok
test virtio::fs::read_only::tests::read_only_open_flags_allow_append ... ok
test virtio::fs::read_only::tests::read_only_open_flags_reject_truncate ... ok
test virtio::fs::read_only::tests::read_only_open_flags_reject_write_access ... ok
test virtio::mmio::tests::test_ack_features_by_page ... ok
test virtio::mmio::tests::test_bus_device_activate ... ok
test virtio::mmio::tests::test_bus_device_read ... ok
test virtio::mmio::tests::test_get_acked_features ... ok
test virtio::mmio::tests::test_bus_device_reset ... ok
test virtio::mmio::tests::test_get_avail_features ... ok
test virtio::mmio::tests::test_new ... ok
test virtio::mmio::tests::test_set_acked_features ... ok
test virtio::queue::tests::test_add_used ... ok
test virtio::queue::tests::test_checked_new_descriptor_chain ... ok
test virtio::queue::tests::test_queue_processing ... ok
test virtio::queue::tests::test_queue_validation ... ok
test virtio::mmio::tests::test_bus_device_write ... ok

failures:

---- legacy::i8042::tests::test_i8042_kbd stdout ----

thread 'legacy::i8042::tests::test_i8042_kbd' (4502008) panicked at src/devices/src/legacy/i8042.rs:416:13:
assertion failed: i8042.kbd_interrupt_evt.read().unwrap() > 1
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- legacy::i8042::tests::test_i8042_read_write_and_event stdout ----

thread 'legacy::i8042::tests::test_i8042_read_write_and_event' (4502009) panicked at src/devices/src/legacy/i8042.rs:333:9:
assertion `left == right` failed
  left: 1
 right: 2


failures:
    legacy::i8042::tests::test_i8042_kbd
    legacy::i8042::tests::test_i8042_read_write_and_event

test result: FAILED. 49 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.13s

error: test failed, to rerun pass `-p devices --lib`
# Last command returned 101

@nohajc
Copy link
Copy Markdown
Contributor Author

nohajc commented Apr 16, 2026

@mtjhrc Yeah, currently I have CC_LINUX in my vscode settings. I suppose this could be improved overall but that's a task for another day.

Sorry, I think I actually only tried to run mmio and vstate tests. At first, I was mostly focused on build failures.

@nohajc
Copy link
Copy Markdown
Contributor Author

nohajc commented Apr 16, 2026

I'd suggest we also add cargo test to CI in another PR.

@nohajc
Copy link
Copy Markdown
Contributor Author

nohajc commented Apr 16, 2026

As for those two particular legacy::i8042 tests, not sure if I should just disable them. They don't seem macOS specific and I have no idea why they're failing. Maybe it would deserve more investigation?

Claude suggests some assertions in those tests are correct on Linux but not macOS.

IMHO if the primary goal here was to fix the build, we could address this properly in another PR. I already feel I should have made my PR more focused in the first place.

@mtjhrc
Copy link
Copy Markdown
Collaborator

mtjhrc commented Apr 17, 2026

/gemini review

Comment thread Makefile Outdated
@mtjhrc
Copy link
Copy Markdown
Collaborator

mtjhrc commented Apr 20, 2026

This is turning into a bit of a rabbit hole. :) And not really what I set out to solve in the beginning.

Huge thanks though! The macOS unit tests have been completely unusable, this gives us an opportunity to properly start maintaining them.

Actually managed to fix everything but please review the event_manager tests because the suggested fix was AI generated.

Looks fine for now - the sematics of the emulated EventFd are kind of broken but the actual non-test code is smart enough to avoid the problems, I plan to look into refactoring EventFd stuff in libkrun 2.0 so there's a separate explicit read and write end.

The last issue I have running cargo test on macOS is that I need to raise ulimit -n to about 1024. The default 256 is too low.

Yes this fine and generally recommended when using libkrun - we consume a lot of file descriptors. Btw by default macOS gives you a lower limit over SSH (256 fds) than e.g. if you open a terminal in the GUI...

@nohajc
Copy link
Copy Markdown
Contributor Author

nohajc commented Apr 20, 2026

Huge thanks though!

No problem! This project is fun. :)

@nohajc
Copy link
Copy Markdown
Contributor Author

nohajc commented Apr 20, 2026

Example of successful cross-compilation workflow runs:
https://github.com/nohajc/libkrun/actions/runs/24682731130/job/72183942282

@nohajc
Copy link
Copy Markdown
Contributor Author

nohajc commented Apr 21, 2026

Ok, I think it's done. Any other thoughts @mtjhrc? :)

@nohajc nohajc requested a review from mtjhrc April 21, 2026 18:28
nohajc added 10 commits April 21, 2026 23:50
Signed-off-by: Jan Noha <nohajc@gmail.com>
Signed-off-by: Jan Noha <nohajc@gmail.com>
- fixed broken init rule
- fixed FreeBSD base image URL for x86_64 and arm64

Signed-off-by: Jan Noha <nohajc@gmail.com>
- removed parts of tests which prevented `cargo test` from compiling
- added `config.toml` which codesigns unit tests when executing `cargo test`
- `tests/run.sh` and `.cargo/macos-test-runner.sh` share the same entitlements plist

Signed-off-by: Jan Noha <nohajc@gmail.com>
- linkage should be established in the vmm create instead of libkrun

Signed-off-by: Jan Noha <nohajc@gmail.com>
- cargo tests for this crate were incorrectly enabled on macOS

Signed-off-by: Jan Noha <nohajc@gmail.com>
Signed-off-by: Jan Noha <nohajc@gmail.com>
- linker flag moved to the end of the command

Signed-off-by: Jan Noha <nohajc@gmail.com>
The macOS EventFd is pipe-backed, and as_raw_fd() returns the read end. EventSet::OUT (EVFILT_WRITE) on a read-only pipe fd never fires on kqueue.

Changes made to event_manager.rs:

- Switched DummySubscriber from EventSet::OUT to EventSet::IN throughout (interest_list, register_ev2, event handling)
- Pre-write data to eventfds before run() so IN events fire immediately
- Replaced processed_ev1_out/processed_ev2_out with processed_ev1_in/processed_ev2_in
- test_modify still exercises the EventManager::modify() code path and error case, but no longer tests OUT→IN transition (which requires Linux-only eventfd semantics)
- The test_unregister test takes ~3s due to a separate issue: the macOS Epoll::wait() ignores the timeout parameter and hardcodes a 3-second timeout.

Signed-off-by: Jan Noha <nohajc@gmail.com>
Signed-off-by: Jan Noha <nohajc@gmail.com>
- the implementation is currently Linux-specific

Signed-off-by: Jan Noha <nohajc@gmail.com>
Copy link
Copy Markdown
Collaborator

@mtjhrc mtjhrc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, this is great, fixing the unit tests on macOS has been long overdue, thanks!

@nohajc
Copy link
Copy Markdown
Contributor Author

nohajc commented Apr 22, 2026

Some of them which don't require starting a VM could be even run in CI.

@slp slp added 2.0 and removed 1.x labels Apr 22, 2026
@slp slp merged commit 23f58f8 into containers:main Apr 24, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants