ABI: Land the ABI 43 batch on current main - #1264
Open
brandonpayton wants to merge 110 commits into
Open
Conversation
Serialize supported tagged catch state per activation and reject non-reconstructible reference and table state. Require the ABI 43 safety capability across build and host launch boundaries.
Give Node and browser process Workers the same versioned reference graph, external-reference authority, imported state, table replicas, exceptions, and replay-gate owners. Recreate main and side-module Store-local identities before continuation rewind. Release temporary roots on completion, abort, exec, and teardown. Stage dynamic loading as host-only prepare/next operations followed by ordinary Wasm initializers. Serialize pthread loader ownership, replay exact module layouts, and defer signal delivery until libc regains the guest boundary.
Require ABI 43 activation-state-safe capability, exact role metadata, zero native start sections, and staged-loader inventory across package receipts, indexes, sidecars, Homebrew validation, and shell guards. Build valid compiler-EH fixtures normally instead of routing them to an unsupported bucket. Select the declared LLVM archiver on Darwin so final-key source rebuilds remain reproducible. Treat executable kernel.kernel_fork and side-module env.fork as fork-entry evidence in the common binary inventory. Keep the current Homebrew program projection and standalone resolver unchanged; regenerate both once from the coherent batch tip.
Move host-to-kernel structured transfers into bounded, ABI-described scratch regions owned by the kernel. Replace guest-pointer-dependent marshalling for process, IPC, socket, spawn, and signal state with checked wire formats and exact pointer-width layouts. Generate matching Rust, C, and TypeScript metadata, enforce allocation and copy-back bounds, and cover native, wasm32, wasm64, Node, and browser paths. This is the foundational ABI 43 change; later commits close additional ownership and publication seams.
Package cached toolchain sysroots without embedding the source worktree path. Reconstruct the expected layout after cache restore so the action works in a different checkout and on a different runner.
Declare and stage the wasm64 examples needed by browser tests instead of assuming that a previous local build left them in place. Teach CI scope detection, workspace packing, and browser fetch tests to enforce the fixture manifest.
Finish moving blocked retries, VFS operations, IPC, sockets, process state, and scalar results behind kernel-owned scratch and entry gates. Remove remaining host dependence on mutable guest pointer layouts. Apply the same ownership rules in Node and browser workers. Add focused tests for append and offset behavior, environment transactions, worker lifecycle, and fail-stop scratch violations, then refresh the ABI 43 snapshot and generated bindings.
Publish each kernel scratch region and its generated contract as one atomic ABI state. Fail closed when the host observes a missing, stale, or partially initialized region rather than continuing with mixed metadata. Refresh the ABI projection and exercise publication, reusable kernel stacks, and fail-stop validation in the shared Node and browser paths.
Honor the Node bridge endpoint-ownership flags after FIN or cleanup. A close may synchronously reclaim the global pipe slot, so later pumps must neither query nor notify its reusable numeric index.
Give Node conformance runs an explicit, session-owned VFS root and mount lifecycle. Prevent tests from depending on or mutating ambient host storage, and make teardown retire the same mount state that was admitted. Route the libc, POSIX, and Sortix runners through that production host path and cover mount creation, export, and cleanup.
Reject VFS seed entries whose normalized paths shadow an existing session mount or another seed. This keeps the admitted image authoritative and prevents path ordering from silently changing machine state.
Snapshot executable bytes when the resolver admits a program instead of rereading a mutable host file during worker launch. The process now runs the exact image that passed ABI and artifact validation. Cover package resolution and rootfs export while preserving the same behavior in the shared Node and browser launch machinery.
Apply retired-memory debt admission synchronously before ordinary fork creates or copies a child WebAssembly.Memory. Return EAGAIN when the bounded debt limit is saturated, without yielding across the parent snapshot or allocating another complete address space. Keep ordinary fork semantics independent from vfork and prove that the rejected path performs no child allocation or copy.
Add explicit ownership tokens for host workers that intentionally share one process-memory backing. Count the backing once, retire it only after the final alias crosses its quiescence fence, and propagate forced termination until final release. This ownership mechanism does not reinterpret ordinary fork: only callers with an explicit shared-memory transaction may retain an alias.
Drive ordinary-fork admission through the production kernel entry gate instead of constructing a partial worker object. Verify that a retired-memory EAGAIN removes the provisional child and completes the parent mailbox with the truthful error.
Allow a separately launched child worker to borrow the parent process continuation while keeping worker-local replay control state independent. Rebuild main-module and active side-module references from admitted bytes without handing the child authority over parent continuation storage. Gate completion through an explicit lifetime coordinator and cover main, dynamic-link, and active-side replay in Node, Chromium, Firefox, and WebKit fixtures.
Document the fork memory root cause, the shared-memory vfork design, ordinary-fork admission, rejected alternatives, ABI implications, and the evidence still required for release. Add a component resident-set-size harness that separates worker, module, shared-memory, full-clone, and sparse-clone costs. Its results are scoped to those components and do not claim Homebrew application performance.
Binaryen --fpcast-emu rewrites indirectly called functions to a uniform i64 trampoline signature. Host pthread entry calls were still passing a plain JavaScript number, which traps when the wrapper expects BigInt arguments. Build the call arguments from the WebAssembly function parameter count: preserve the plain pointer ABI for ordinary entries and use a BigInt pointer plus zero-filled i64 slots for fpcast trampolines. ABI 43 forward-port: fork-from-thread children now enter through the exported wpk_fork_resume_thread helper. Retain that plain replay ABI and apply fpcast-expanded arguments only where the host calls the table entry directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit a2b70da)
Character-special mode is broader than terminal identity. Kandelo encoded host terminal stdio and virtual devices such as /dev/null, framebuffer, audio, and DRM as CharDevice, so terminal probes could mistake any virtual character device for an interactive terminal. Derive terminal identity from the open file description. Dedicated PTY master and slave types are terminals; legacy host stdio is a terminal only when its stable canonical path and host handle agree. Reuse that result for isatty, termios, ioctl namespace gating, and fpathconf. The ABI 43 forward-port retains the current exact 60-byte termios layout. This changes no syscall number, marshalling rule, exported signature, structure layout, generated binding, or VFS ABI metadata. Validation: - native kernel unit suite: 1,491 passed - ABI snapshot and generated bindings check - git diff --check (cherry picked from commit 3b470d3)
Windows does not expose a POSIX permission model through Node. Native entries carry no owner/group/other split or directory search bit, so a guest that drops privileges cannot traverse or write the host-backed sandbox it was given. On Windows only, represent writable host directories as 0777, read-only directories as 0555, writable files as 0666, and read-only files as 0444. Preserve native type bits and let guest chmod and chown metadata override the synthesized permissions. POSIX hosts keep their native modes unchanged. The ABI 43 forward-port performs synthesis only after exact conversion of the native BigInt mode. Existing EOVERFLOW checks for modes, links, sizes, and timestamps remain intact. Validation: - host declaration generation and typechecking - native-metadata and Node uid/gid suites: 15 passed - Windows host execution was not available on this macOS worktree - git diff --check Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit aa12186)
A new git worktree (and a fresh checkout) does not inherit git submodules, a musl sysroot, node_modules, or fetched test binaries, so Vitest and the conformance/browser suites cannot run until those are built or fetched. Agents were treating this as "cannot validate" rather than a setup step. Document the exact sequence in validation.md and build-docs-and-prs.md: submodule init (with the stray-libc/musl recovery), build-musl.sh (sysroot), build.sh (kernel wasm → local-binaries/kernel.wasm + rootfs), root + host `npm ci` (root provides tsx for the conformance runners), and fetch-binaries.sh. Add the explicit instruction not to report "I can't run Vitest/conformance/browser" because a fresh worktree lacks artifacts — build or fetch them and report the real result, or name the exact step that failed. Verified by running the sequence end-to-end in a fresh worktree: kernel wasm + sysroot + rootfs build, `vitest` (778 pass; the only failures were a missing fetched `programs/wasm64/hello64.wasm`), and Sortix conformance (10/10) all ran. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit a3739a2) This conceptual commit combines recovery commits cda80003f and 050a7875e.
Make Area: Purpose the repository convention for PR titles and commit subjects. Add contract-oriented examples and guidance for choosing one primary prefix for cross-cutting changes. Validation: - VitePress documentation build - git diff --check (cherry picked from commit 60fb395)
Chromium-based user agents also contain AppleWebKit. Treating that token as sufficient enabled WebKit-only post-destroy memory handling in Chrome, Edge, Opera, and Chromium on iOS. Forward-port the predicate into the current shared kernel-owned boot helper and keep a pure user-agent classifier for regression coverage. Exclude Chromium, Chrome on iOS, Edge, Opera, Firefox, and Firefox on iOS while retaining Safari and WebKit. Validation: - browser engine detection test: 1 passed - git diff --check (cherry picked from commit 4cbaf2b)
Mutate the sorted mapping vector in place instead of draining and rebuilding it for every munmap call. Preserve the current ABI 43 page-rounding behavior and retain the vector allocation when entries are removed. Keep the original kernel-trap syscall-ring diagnostic while adapting it to the current fatal-entry and blocking-retry cleanup paths. Validated with: scripts/dev-shell.sh -- scripts/ci-run-test-suite.sh cargo-kernel scripts/dev-shell.sh -- npm --prefix host run typecheck End-to-end performance has not been measured yet; that remains part of the batched Node and browser validation. (cherry picked from commit 18a5aa3)
(cherry picked from commit 076362b)
POSIX loopback is machine-scoped. Kandelo previously searched only the sender process socket arena, so an unconnected sendto could report success while dropping the datagram and a connected socket could retain a false ECONNREFUSED. Capture an owned route after the process-local send, release the direct process-table borrows, and then deliver to one accepting foreign endpoint. This preserves sender metadata, avoids duplicate delivery across inherited or SO_REUSEADDR bindings, and clears the provisional connected-socket error only after successful foreign delivery. This changes no ABI exports, imports, or repr(C) layouts. Validated with: scripts/dev-shell.sh -- scripts/ci-run-test-suite.sh cargo-kernel scripts/dev-shell.sh -- bash scripts/check-abi-version.sh scripts/dev-shell.sh -- npm run docs:build (cherry picked from commit c413bd8) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Following stat and fstatat calls on /dev/fd/N and the standard-stream aliases now return the referenced descriptor’s authoritative metadata. No-follow calls report stable devfs symlink metadata, readlink exposes the alias target, and O_NOFOLLOW rejects the final symlink. Keep devfs directory, getdents, and lstat device/inode metadata coherent. Add a guest regression exercised through NodeKernelHost and BrowserKernel. The browser test uses the current minimal runner so it does not activate unrelated shell-package generations. This fixes GNU coreutils mistaking /dev/null for stdin without changing the ABI. (cherry picked from commit 35f0455)
Place Kandelo syscall glue and the C runtime startup object before user link inputs, preserve the caller source, archive, and library order, and keep the final musl archive last. This prevents an explicit -lc from resolving syscall definitions before the platform overrides and avoids scanning dependency libraries before their consumers. Adapt the original change to the current prepared-linker and reproducible path logic. The Kandelo-native driver already uses the same order. (cherry picked from commit 915a84b)
Open the replacement iterator before retiring the active stream, and keep the old state authoritative if opening or closing the replacement sequence fails. Add kernel failure coverage and shared host-bridge coverage for throwing close calls and numeric handle reuse. Adapt the host tests to the current Rust-lent destination capability instead of bypassing private state. (cherry picked from commit e96c381)
WebKit can compile valid ABI 43 fork artifacts while failing to return their import descriptors. That prevents otherwise portable programs from reaching the normal host path. Bind Kandelo-created modules to descriptors parsed from the exact bytes that passed artifact admission, and keep native engine reflection for externally supplied modules. Cover the fallback with host tests and an exact Chromium, Firefox, and WebKit artifact test. Validation: - scripts/dev-shell.sh -- npm --prefix host run typecheck - scripts/dev-shell.sh -- npm run docs:build - 114 focused host tests - Playwright reflection test on Chromium, Firefox, and WebKit
brandonpayton
added
the
batched-changes
The cargo-xtask lane validates repository-owned inputs declared by VFS product manifests. ABI 43 adds the musl license to the SDK product, so the lane must materialize that exact gitlink before running. Keep the shell composition fixture aligned with the ABI 43 catalog and cover both staging and synthesized prepare-merge routes.
Member
Author
|
Fresh final-tree local validation for the Formula-authority cutover is green. |
Contributor
Phase B-1 matrix build status —
|
| Package | Arch | Status | Sha |
|---|---|---|---|
| icu | wasm32 | failed | — |
| libcurl | wasm32 | failed | — |
| libcxx | wasm32 | failed | — |
| libcxx | wasm64 | failed | — |
| libiconv | wasm32 | failed | — |
| libpng | wasm32 | failed | — |
| libxml2 | wasm32 | failed | — |
| libzip | wasm32 | failed | — |
| openssl | wasm32 | failed | — |
| openssl | wasm64 | failed | — |
| sdl2 | wasm32 | failed | — |
| sdl3 | wasm32 | failed | — |
| sqlite | wasm32 | failed | — |
| sqlite | wasm64 | failed | — |
| zlib | wasm32 | failed | — |
| zlib | wasm64 | failed | — |
| bc | wasm32 | failed | — |
| bzip2 | wasm32 | failed | — |
| coreutils | wasm32 | failed | — |
| cpython | wasm32 | failed | — |
| curl | wasm32 | failed | — |
| dash | wasm32 | failed | — |
| diffutils | wasm32 | failed | — |
| dinit | wasm32 | failed | — |
| erlang | wasm32 | failed | — |
| fbdoom | wasm32 | failed | — |
| file | wasm32 | failed | — |
| findutils | wasm32 | failed | — |
| gawk | wasm32 | failed | — |
| git | wasm32 | failed | — |
| grep | wasm32 | failed | — |
| gzip | wasm32 | failed | — |
| homebrew-bootstrap | wasm32 | failed | — |
| kandelo-sdk | wasm32 | failed | — |
| kernel | wasm32 | failed | — |
| less | wasm32 | failed | — |
| lsof | wasm32 | failed | — |
| m4 | wasm32 | failed | — |
| make | wasm32 | failed | — |
| mariadb | wasm32 | failed | — |
| mariadb | wasm64 | failed | — |
| modeset | wasm32 | failed | — |
| msmtpd | wasm32 | failed | — |
| nano | wasm32 | failed | — |
| ncurses | wasm32 | failed | — |
| netcat | wasm32 | failed | — |
| nginx | wasm32 | failed | — |
| php | wasm32 | failed | — |
| posix-utils-lite | wasm32 | failed | — |
| ruby | wasm32 | failed | — |
| sdl-dsp-test | wasm32 | failed | — |
| sdl2-mixer-playwave | wasm32 | failed | — |
| sed | wasm32 | failed | — |
| spidermonkey | wasm32 | failed | — |
| tar | wasm32 | failed | — |
| tcl | wasm32 | failed | — |
| unzip | wasm32 | failed | — |
| userspace | wasm32 | failed | — |
| vim | wasm32 | failed | — |
| wget | wasm32 | failed | — |
| xz | wasm32 | failed | — |
| zip | wasm32 | failed | — |
| zstd | wasm32 | failed | — |
| bash | wasm32 | failed | — |
| mariadb-test | wasm32 | failed | — |
| mariadb-vfs | wasm32 | failed | — |
| mariadb-vfs | wasm64 | failed | — |
| nethack | wasm32 | failed | — |
| node | wasm32 | failed | — |
| redis-vfs | wasm32 | failed | — |
| spidermonkey-node | wasm32 | failed | — |
| vim-browser-bundle | wasm32 | failed | — |
| nethack-browser-bundle | wasm32 | failed | — |
| rootfs | wasm32 | failed | — |
Auto-generated; replaced on each push. Raw data in the publish-status workflow artifact.
Candidate bottles publish under one Formula repository, while dependency provenance spans the ABI candidate parent. Pass that parent and ABI to both provenance phases, and model native unlink behavior in the publication fixture.
Candidate jobs intentionally run without the production Formula identity. Make their exact tapped checkout read-only during Homebrew evaluation, then restore owner permissions only during cleanup.
Record the exact Linux compatibility lock generated by the pinned Homebrew workflow after Homebrew Core advanced libgit2 to 1.9.7.
Staging candidates must generate fresh bottle JSON for their ABI-specific registry instead of retaining canonical-root metadata with --keep-old. PR status collection now carries one read-only token only to GitHub API metadata discovery. This avoids anonymous rate exhaustion while release assets and GHCR objects remain public and anonymous. The generated request-policy digest binds that protected workflow change.
…ormula-test-checker # Conflicts: # abi/staging/request-policy.generated.json
Regenerate the ABI-bound program package index so staged Formula tests use the exact ABI 43 package graph. Resolve Netcat Automake support data from its projected keg without executing the relocated native wrapper.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Kandelo needs the ABI 43 platform batch to land against the current
mainwithout discarding either the batch's attributed history or the browser CORS
and lazy-shell contracts that landed while the batch was in progress.
PR #1240 no longer merges cleanly. Force-pushing its history would make the
reviewed source harder to recover and would violate its non-destructive merge
expectation. This replacement rebases the same attributed commits onto current
main, then adds purpose-scoped integration fixes.The first hosted bottle rehearsal also exposed a migration blocker: the
temporary Kandelo build-helper bridge still treated legacy package registry
metadata as source authority. That made a tap Formula fail whenever its reviewed
source identity differed from the soon-to-be-retired registry copy, even though
the bottle builder was supposed to build the Formula's source.
Merge method
REBASE MERGE ONLY. DO NOT SQUASH. DO NOT CREATE A MERGE COMMIT.
The 88 non-merge commits preserve intentional authorship and bisection
boundaries. This PR supersedes #1240 once its exact checks and hosted candidate
rehearsal are green.
What changed
main, including PR Restore lightweight lazy shell and browser npm installs #1253.browser worker protocols.
privileged login, sudo, and Ruby product.
/home/userandUser <user@wasm.local>compatibility contract.builds, while retaining fail-closed candidate and publication namespaces.
authority. The temporary bridge now binds only the exact Kandelo build helper;
legacy
package.tomlandbuild.tomlmetadata cannot veto the Formula.Validation
Fresh checks used
scripts/dev-shell.shon the final tree:cargo test -p xtask ... homebrew_tier2_preflight::tests: 23 passedbash scripts/test-homebrew-formula-runtime-closure.shbash scripts/test-homebrew-rootfs-publication-selection.shbash scripts/test-homebrew-patched-launcher.sh: 93 Python tests passed,two intentional skips, plus launcher integration
python3 scripts/test-homebrew-staging-candidate-roots.py: 2 passedruby scripts/check-homebrew-publish-workflow-trust.rbgit diff --checkEarlier validation on the integrated ABI 43 tree also covered:
bash scripts/check-abi-version.shcd host && npm run typecheckbash scripts/test-homebrew-tap-identity.shbash scripts/test-homebrew-staging-candidate-compose.shbash .github/scripts/test-merge-candidate-workflows.shThe first hosted candidate wave reached the real generic bottle builder and
exposed the source-authority mismatch fixed here. Fresh GitHub checks and a new
hosted candidate build/publication wave remain release gates; this PR does not
claim that bottles have already been promoted.
History and attribution
main.this replacement passes CI.