Support cross-compiling onnxruntime for Windows from a Linux host#29642
Draft
Penguinwizzard wants to merge 14 commits into
Draft
Support cross-compiling onnxruntime for Windows from a Linux host#29642Penguinwizzard wants to merge 14 commits into
Penguinwizzard wants to merge 14 commits into
Conversation
The MLAS x64 assembly kernels in onnxruntime/core/mlas/lib/amd64 previously
only assembled with the Microsoft Macro Assembler (ml64.exe), because they
relied on the Windows SDK's macamd64.inc and on several MASM constructs that
the LLVM MASM-compatible assembler (llvm-ml) does not accept. This blocked
cross-compiling ONNX Runtime for Windows-x64 from a non-Windows host (e.g.
building on Linux, as the Edge browser does).
This change makes all 37 amd64/*.asm files assemble under BOTH ml64.exe and
llvm-ml, and adds CI to keep them that way:
- Replace the Windows SDK macamd64.inc dependency with an ORT-owned,
dual-compatible prologue/entry/unwind macro layer in mlasi.inc (built on
MASM `proc frame` + SEH directives). The generated code, .pdata and .xdata
are unchanged for ml64.
- Fix every remaining MASM-vs-llvm-ml incompatibility class, including:
nested-struct default init (`<>` -> `{}`, which also unblocked llvm-ml's GT
parser), label collisions from llvm-ml lacking MASM's default OPTION SCOPED
(via LOCAL / anonymous @@ labels / token-pasted unique suffixes), the single
`::` public-in-proc label in SconvKernelAvx.asm (guarded by IFDEF LLVM_ML),
and an invalid VEX FMA size hint (DWORD PTR -> YMMWORD PTR) in
SconvKernelFma3.asm. See individual files for details.
- Add .github/workflows/windows_mlas_masm_cross_assembler.yml: on a Windows
runner it assembles every amd64/*.asm with both ml64.exe (no define) and
llvm-ml (/DLLVM_ML=1), failing if either assembler rejects any file.
llvm-ml must be invoked with /DLLVM_ML=1 so the alternate-entry public label
uses the llvm-ml-compatible single-colon + PUBLIC form; ml64 gets no define and
is byte-for-byte unchanged. Downstream cross-compile consumers (Edge) must add
that define to their llvm-ml invocation.
Verified locally: 37/37 amd64/*.asm assemble under llvm-ml. ml64 assembly and
runtime SEH-unwind correctness are validated by the new Windows CI job.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…m-ml) Extend the MLAS dual-assembler work into a real Linux cross-build of the whole onnxruntime_mlas library, and move CI off windows-latest onto a Linux runner. - cmake/win_x64_crosscompile_toolchain.cmake (new): a Windows-x64 cross toolchain using clang-cl + lld-link for C/C++, llvm-ml for the amd64 MASM kernels, and a Windows SDK/CRT staged by `xwin` (consumed via /imsvc + /libpath:). Targets x86_64-pc-windows-msvc, feeds llvm-ml -m64 /DLLVM_ML=1 and the amd64 include dir, and skips CMake's link-based compiler check (static-lib only). - cmake/onnxruntime_mlas.cmake: add clang-gated per-file arch flags (AVX2/AVX512/AVXVNNI/AVX512VNNI/AMX/SSE4.1) for the x64 intrinsic kernels. clang-cl enforces per-function target features where MSVC does not; the block is gated on Clang so the MSVC/ml64 build is byte-for-byte unchanged. - .github/workflows/linux_mlas_win_cross.yml (new): ubuntu-latest job that installs LLVM + xwin, cross-configures with the toolchain, builds onnxruntime_mlas, and asserts the library plus its amd64 .asm objects. This is a strict superset of the old assemble-only check and exercises the llvm-ml (/DLLVM_ML) branch end to end. - Remove .github/workflows/windows_mlas_masm_cross_assembler.yml: superseded by the Linux cross-build. The complementary ml64.exe path (no /DLLVM_ML) is still covered by the native Windows build/test pipelines. Tests are not run on Linux: the artifacts are Windows-x64 binaries and Wine is unreliable for the AVX-512/AMX kernels; runtime correctness stays with the native Windows pipelines. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
GitHub release assets (and their co-located .sha256 sidecars) are mutable, so a compromised release could silently swap the xwin binary. Pin the known-good SHA256 of the xwin tarball and verify the download against it before use. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The MSVC CRT headers staged by xwin enforce a minimum Clang version via a static_assert (STL1000: "expected Clang 19.0.0 or newer"). LLVM 18 fails that check; use LLVM 19. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extend the Linux->Windows-x64 cross-compile support beyond the MLAS static library to the entire onnxruntime.dll, using clang-cl + lld-link for C/C++, llvm-ml for the MLAS amd64 MASM kernels, llvm-rc for the version resource, and a Windows SDK/CRT staged by xwin. This is what downstream consumers such as the Edge browser actually build (they compile ORT from source, not just MLAS). Fixes the following real cross-portability issues that only surface under a case-sensitive filesystem and/or clang's stricter front end; the native MSVC build is unaffected: - ONNX ScopedHandle: INVALID_HANDLE_VALUE performs a reinterpret_cast that is not a valid non-type template parameter under Clang. Replace the ScopedResource alias with a concrete class holding the sentinel at runtime (onnx.patch). - <wil/Resource.h> / <Winver.h>: the real headers are lowercase; fix the include spellings so they resolve on case-sensitive filesystems. - spin_pause.cc: clang-cl requires the waitpkg target feature for the _tpause intrinsic (used behind a runtime HasTPAUSE() guard); add -mwaitpkg for the Windows/Clang/x86_64 cross build. - onnxruntime.rc VERSIONINFO: the octal-escaped copyright/registered glyphs in narrow strings are rejected by llvm-rc (no code page); use wide L"\x00A9" / L"\x00AE" escapes, which both rc.exe and llvm-rc accept with identical output. Also pass the Windows SDK include paths to llvm-rc, which does not honor /imsvc. - MLAS QgemmU8X8KernelAvx2.asm: ExitKernel is declared LOCAL in an outer macro but referenced from nested macros. ml64 resolves the nested reference to the outer LOCAL; llvm-ml emits it as an undefined external, breaking the DLL link. Thread the exit label through as an explicit macro parameter so the substitution happens in the outer macro body (where the LOCAL is visible) under both assemblers. Gate the per-target COMPILE_WARNING_AS_ERROR so an explicit -DCMAKE_COMPILE_WARNING_AS_ERROR=OFF is honored (default stays ON). clang-cl emits many warnings MSVC never does; treating them as errors would make the cross-build unbuildable for no portability benefit. The native Windows pipelines keep -WX on. Replace the MLAS-only cross-build workflow with linux_win_cross_build.yml, which cross-builds and link-checks the whole onnxruntime.dll on a Linux runner (LLVM 19 + SHA-pinned xwin, dynamic MSVC runtime pinned so protobuf and ORT agree), and verifies the output is a PE image embedding the llvm-ml-assembled MLAS kernels. Runtime correctness and the ml64 path remain covered by the native Windows build/test pipelines. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…copedHandle Follow-up to the Windows-x64 Linux cross-build, addressing review findings: - Pin and verify the SHA256 of apt.llvm.org/llvm.sh before running it as root, matching the existing xwin pin. The script is fetched over the network and executed with sudo; verifying its digest prevents a compromised or rotated script from running as root on the runner. Documented that the digest must be re-verified when the upstream script legitimately changes. - Quote the Windows SDK/CRT include paths in CMAKE_RC_FLAGS_INIT so resource compilation still works if the xwin root ever resolves to a path with spaces. - Give the ONNX ScopedHandle replacement a move constructor and move assignment (using release()), restoring the movability the original ScopedResource-based alias effectively lacked but that future ONNX code may expect. Copy remains deleted. Verified the patch still applies cleanly to onnx v1.22.0 and that the full onnxruntime.dll relinks with the updated header. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The CI full-DLL cross-build failed compiling ordinary TUs (capture.cc,
ostream_sink.cc): the xwin CRT <intrin.h> transitively includes clang's
<mmintrin.h>, and on the apt.llvm.org clang-19 snapshot the 64-bit MMX vector
typedefs (e.g. __v4hi) are only treated as vectors when 'mmx' is in the global
target features. Without it the vector_size attribute is dropped and the header
fails ("__v4hi (aka 'short')"). Enabling -mmmx globally in the cross toolchain
keeps those typedefs vectors. Cross-only (toolchain file); no effect on the
native MSVC build or ORT's own SIMD codegen (ORT uses no MMX).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…n.h) capture.h defined `__attribute__(x)` to nothing under _MSC_VER to disable it "for MSVC". But clang-cl also defines _MSC_VER, and clang's own intrinsic headers rely on __attribute__((__vector_size__)) for the 64-bit MMX vector typedefs (__v2si/__v4hi/__v8qi). Because capture.cc includes capture.h before logging.h, the poison strips those attributes from the later absl -> <intrin.h> -> <mmintrin.h> chain, collapsing e.g. __v2si to scalar int and failing to compile with 20 "__v4hi (aka 'short')" errors. Real cl.exe is unaffected (its <intrin.h> uses no __attribute__). Narrow the poison to real MSVC only (`#if !defined(__clang__)`); clang-cl keeps real __attribute__, which it supports. Also revert the earlier inert `-mmmx` cross-toolchain option (no-op on the MSVC target; did not fix this). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
lld-link on a case-sensitive filesystem could not open 'PathCch.lib' at the final onnxruntime.dll link: xwin splats case-variant symlinks (pathcch.lib, PATHCCH.lib) but not the mixed-case "PathCch.lib" that path_lib.cc requested via `#pragma comment(lib, "PathCch.lib")`. Native link.exe is case-insensitive, so lowercase "pathcch.lib" resolves on both Windows and the cross toolchain. Lowercase both PathCch.lib pragmas in path_lib.cc, and proactively lowercase Dbghelp.lib -> dbghelp.lib in debug_alloc.cc (a debug-only TU with the same latent hazard). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Penguinwizzard
changed the title
The dual-assembler port added multi-label LOCAL directives to macros invoked multiple times, because llvm-ml lacks ml64's default OPTION SCOPED per-PROC label scoping and would otherwise see duplicate global labels. Three of those LOCAL logical lines exceed ml64's ~512-char line limit, breaking the native Windows (MASM) build with A2039 plus a cascade of A2070 errors: QgemmU8S8KernelAvx2.asm MlasGemmCopyPackAAvx2 / MlasGemmCopyPackBAvx2 ConvSymKernelAvx2.asm ConvSymKernelFunction Replace LOCAL with token-paste unique-label suffixes (matching the port's existing ExitKernel&KernelType& pattern). A per-macro literal discriminator plus the macro parameter guarantees globally-unique labels across every invocation: MlasGemmCopyPackAAvx2(ASigned) -> <label>A&ASigned& MlasGemmCopyPackBAvx2(IsVnni,BSigned) -> <label>B&IsVnni&&BSigned& ConvSymKernelFunction(Isa) -> <label>K&Isa& ConvSymDepthwiseKernelFunction(Isa) -> <label>D&Isa& This removes the long lines entirely (no LOCAL directive) so ml64 accepts the files, while keeping labels unique for llvm-ml. Only label names change; every jump/branch target still resolves, so there is no behavioral change. Verified under llvm-ml both with and without -DLLVM_ML; the no-define mode makes all labels global and still resolves, a strict superset of ml64's per-PROC scoping. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous commit removed the over-long LOCAL directives but the generated token-paste labels kept a lone trailing '&' after the final substituted macro parameter (e.g. ProcessRemainingRowsA&ASigned& , ExitRoutineA&ASigned&:). ml64 rejected every macro invocation with A2070 (invalid instruction operands): a trailing '&' at a token boundary is not the MASM idiom and appears to make ml64 merge the label with the following source line. Upstream ml64-proven token-paste (origin/main SconvKernelAvx.asm:152,261) uses '&' only to separate a parameter from adjacent identifier characters, never a trailing '&' before ':' or end-of-line. Regenerate the four affected macros to match that idiom: MlasGemmCopyPackAAvx2(ASigned) -> <label>A&ASigned MlasGemmCopyPackBAvx2(IsVnni,BSigned) -> <label>B&IsVnni&&BSigned ConvSymKernelFunction(Isa) -> <label>K&Isa ConvSymDepthwiseKernelFunction(Isa) -> <label>D&Isa Labels remain globally unique (ExitRoutineA0/A1, ExitRoutineB0/B10/B11, ExitKernelKAvx2/KAvxVnni/DAvx2/DAvxVnni). Verified under llvm-ml with and without -DLLVM_ML. Label-only renames; every branch target still resolves, so there is no behavioral change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The token-paste unique-label approach (commits 42e071c, be65f5e) kept failing native ml64 with A2070 "invalid instruction operands" at every macro-invocation site, for a reason not reproducible with any local MASM-compatible assembler (llvm-ml and UASM both accept it). Pivot to the only mechanism proven in BOTH ml64 and llvm-ml: a single first-statement LOCAL directive. LOCAL was already the port's original approach; its sole ml64 failure was A2039 (line too long) on three over-long label lists, which then cascaded into A2070s. Rather than work around the length with token-paste, shorten the LOCAL label names so each logical line fits ml64's ~512-char limit: MlasGemmCopyPackAAvx2 LOCAL 564 -> 186 chars MlasGemmCopyPackBAvx2 LOCAL 615 -> 195 chars ConvSymKernelFunction LOCAL 662 -> 240 chars Labels are macro-local scratch names; renaming defs + branch targets is purely cosmetic (no jump target or behavior changes). LOCAL scopes each label per-expansion in both assemblers, so the two invocations of each macro no longer collide under llvm-ml (which treats plain labels as global) while ml64 keeps its per-proc scoping. Verified: each file assembles cleanly under llvm-ml with AND without -DLLVM_ML (no-define makes all labels global, a strict superset of ml64's requirement, proving global uniqueness + full ref resolution). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dual-assembler port changed local-frame struct fields from OWORD to
QWORD 2 DUP(?) because llvm-ml rejects OWORD/XMMWORD as struct field
types. That change is byte-size-identical (16 bytes) but alters the
field's DECLARED element type from 16-byte to 8-byte (QWORD). ml64
infers a memory operand's size from the field type, so the unqualified
16-byte xmm save/restore instructions
movaps Offset[rsp], Reg (save_xmm128 macro)
movaps xmm6, Frame.SavedXmm6[rsp] (inline restores)
became "movaps <xmm>, <8-byte mem>" under ml64 -> A2070 "invalid
instruction operands". (llvm-ml is lenient here and infers 16 bytes from
the xmm register, which is why the Linux cross-build stayed green and the
bug was masked on native Windows by an earlier A2039 abort.)
Fix: add an explicit XMMWORD PTR size override at every 16-byte frame
access, so both assemblers use 16 bytes regardless of the field's
declared type. Centralized the save path in the shared save_xmm128 macro
(mlasi.inc, covers all files) and added XMMWORD PTR to the 123 inline
movaps restore sites across the amd64 kernels. movaps of an xmm register
is always 16-byte, so the emitted bytes are unchanged; this is purely a
size-annotation fix for ml64's stricter operand-size checking.
Verified: all 37 amd64/*.asm assemble under llvm-ml with -DLLVM_ML=1
(system llvm-ml-19 and the Chromium-bundled llvm-ml).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…(fix A2009)
The dual-assembler port rewrote AVX-512 embedded-broadcast operands from
MASM's 'DWORD/QWORD BCST [mem]' keyword (which llvm-ml does not accept) to
the NASM-style 'DWORD/QWORD PTR [mem]{1toN}' decorator. ml64 rejects the
{1toN} decorator -> A2009 "syntax error in expression". This was masked on
native Windows until the earlier A2039/A2070 failures were fixed and ml64
reached these AVX-512 kernels (QgemmU8X8KernelAvx512Core.asm(702)).
Neither broadcast form is accepted by both assemblers, so select the right
one at assembly time via LLVM_ML (defined only for the llvm-ml build). Added
shared text equates in mlasi.inc:
IFDEF LLVM_ML : MlasBcstD=<DWORD PTR> MlasBcstDSuffix=<{1to16}> ...
ELSE : MlasBcstD=<DWORD BCST> MlasBcstDSuffix=<> ...
A broadcast operand is written MlasBcstD [mem]MlasBcstDSuffix and expands
to 'DWORD PTR [mem]{1to16}' for llvm-ml or 'DWORD BCST [mem]' for ml64 -- the
latter being byte-identical to origin/main's proven ml64 text. Applied to:
- QgemmU8X8KernelAvx512Core.asm (36 operands) and SconvKernelAvx512F.asm (6),
which used the raw inline {1to16} form;
- SgemmKernelCommon.inc / DgemmKernelCommon.inc, whose FgemmElementBcst /
FgemmElementBcstSuffix text equates (consumed by FgemmKernelAvx512FCommon.inc)
are now conditional (DWORD/QWORD BCST + empty suffix under ml64).
Verified: all 37 amd64/*.asm assemble under llvm-ml -DLLVM_ML=1 and the
emitted broadcast encodings are correct (vpmulld (%r11){1to16},...). The ml64
branch reproduces main's exact BCST text; empty-suffix TEXTEQU expansion
adjacent to ']' validated with a MASM-compatible assembler.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This changeset alters our MASM code to build in not only ml64 but also clang-ml,
allowing us to cross-compile for Windows targets from Linux. It also adds the related
build scripts and a GitHub action to validate that this continues to function.
Motivation and Context
Being able to cross-compile for Windows from Linux builders offers additional
flexibility in development and CI environments, which is desirable for many projects.