diff --git a/src/proposed/0240-x86_APICv.md b/src/proposed/0240-x86_APICv.md new file mode 100644 index 0000000..3c5d487 --- /dev/null +++ b/src/proposed/0240-x86_APICv.md @@ -0,0 +1,351 @@ + + +# Intel APICv support for x86 VCPU + + + +- Author: Bill Nguyen +- Proposed: 2026-04-15 + +## Summary + +This RFC proposes adding kernel support that allows userspace to use Intel’s hardware support for Advanced Programmable Interrupt Controller (APIC) virtualisation on x86, commonly referred to as [APICv](https://edc.intel.com/content/www/us/en/design/ipla/software-development-platforms/client/platforms/alder-lake-desktop/12th-generation-intel-core-CPUs-datasheet-volume-1-of-2/010/intel-apic-virtualization-technology-intel-apicv/). + +On x86 virtual machines, the guest OS expects to interact with a local APIC for interrupt masking, prioritisation, acknowledgement, and interrupt delivery. This functionality can be emulated entirely in software by a userspace Virtual Machine Monitor (VMM), but this can result in a high rate of VM exits and poor performance for some guests. + +Intel APICv is a collection of VT-x features that allow the host CPU to assist with local-APIC virtualisation. Depending on which APICv features are enabled and supported by the host CPU, the CPU can: + +- virtualise some guest APIC accesses without involving the VMM, +- track selected virtual APIC state in hardware, and +- evaluate and deliver some virtual interrupts without requiring a VM exit. + +This RFC does not propose that the kernel implement a virtual APIC. Instead, it proposes that the kernel provide the minimal support needed for userspace VMMs to supply the APICv-related pages and use the relevant Virtual Machine Control Structure (VMCS) controls safely through the existing VCPU interface. + +The initial scope of this RFC covers these APICv-related features: + +- Use TPR shadow +- Virtualise APIC accesses +- APIC-register virtualisation +- Virtual-interrupt delivery +- Virtualised End-of-Interrupt (EOI) + +Other related features, such as x2APIC virtualisation, virtual interrupt posting, and IPI virtualisation, are left for future work. + +This RFC does not break backward compatibility. + +## Motivation + +- Why are we doing this? + +An interrupt controller is a mandatory requirement for any commodity Operating Systems that you would run inside a Virtual Machine. While virtualising the APIC in pure software is sufficient for Linux guests, we have found that virtualising Windows without APICv resulted such poor performance that the guest was unusable. + +One of the specific issues we ran into was related to the Task Priority Register (TPR), also known as Control Register 8 (CR8). The TPR is used by the CPU to determine which classes of interrupts are currently allowed to be delivered. In other words, it acts as a simple interrupt priority threshold: interrupts at or below the threshold are masked, while interrupts above it may be delivered by the local APIC. + +We measured that the Windows NT kernel updates the TPR very frequently, approximately 50,000 times per second on our setup. In a software APIC model, each TPR access can require a VM exit so that the VMM can emulate the operation and update the virtual APIC state. This corresponds to approximately 50,000 VM exits per second, or roughly 200,000 context switches per second for just emulating the TPR (guest -> seL4 -> VMM -> seL4 -> guest)! + +We have not seen Linux use the TPR anywhere near as much as Windows, which is why the performance on Linux guests has been acceptable. + +When we switched to using Intel APICv, those TPR updates could be handled by the CPU without requiring a VM exit to the VMM. This eliminated a major source of VM exits. On our VMM, Windows 10 boot time dropped from roughly 15-20 minutes to approximately 1-2 minutes, and the resulting desktop became usable. + +TPR virtualisation is only part of the motivation. APICv also reduces exits on the interrupt delivery path more generally. + +With a purely software APIC, the typical flow for delivering a virtual interrupt looks roughly like this: + +1. An external interrupt is raised. +2. If the guest VCPU is currently running, the interrupt causes a VM exit so the kernel can schedule the VMM. +3. The VMM receives the notification and examines the virtual APIC state to determine whether the guest is currently able to take the interrupt. +4. If the guest has interrupts disabled, or if the interrupt priority rules do not currently permit delivery, the VMM must defer injection and resume the guest. +5. At some later point in time, another VM exit occurs when the VMM gets another opportunity to attempt delivery when the architectural rules are satisfied. +6. Once the interrupt is finally injected, the guest eventually signals completion by writing to the APIC End-Of-Interrupt (EOI) register, which itself triggers another VM exit so that the VMM can update interrupt-controller state. + +This means that a single virtual interrupt involves several VM exits and multiple round-trips between guest, kernel, and VMM. + +With APICv, most of this work can be offloaded to the CPU. The VMM can mark a virtual interrupt as pending via and resume the guest, while the CPU tracks the relevant virtual APIC state and performs interrupt evaluation and delivery directly in VMX non-root mode where possible. This reduces VM exits and lowers the overhead for interrupt-heavy guests. + +- What problem does it solve? What use cases does it support? + +This RFC allows more performant guests on x86, in particular, it allows for Windows guests to have acceptable performance. + +It is particularly useful for guests that update the virtual APIC state frequently or receive a large number of interrupts. + +- What is the expected outcome? + +The expected outcome is that userspace VMMs on seL4 can use Intel APICv support to offload parts of local-APIC virtualisation to the CPU, substantially reducing VM exits and improving guest performance without moving APIC policy or virtualisation logic into the kernel. + +## Guide-level explanation + + + + + + +Support for this feature is gated behind a new configuration flag: + +`INTEL_APICV` + +By default, this option is OFF. + +If the option is OFF, there is no change to kernel behaviour. + +If the option is ON, the kernel is built with support for APICv-related checking and exposes a new system call for configuring the APICv pages. The userspace VMM may then use whichever supported APICv controls are available on the current host CPU. If the host CPU does not support all of the APICv features implemented in this RFC, the kernel will refuses to boot. + +To use APICv, userspace must provide two 4 KiB frames to the kernel using a new system call: + +`seL4_X86_VCPU_SetAPICvPages()` + +These frames are used as: + +- the APIC-access page, and +- the virtual-APIC page. + +Conceptually: + +The APIC-access page is the page through which xAPIC-style memory-mapped APIC register accesses are virtualised. +The virtual-APIC page is a page used by VT-x to store selected virtual local-APIC state. + +The APIC-access page should be mapped into guest RAM through EPT at the location expected for guest APIC accesses. Conventionally `0xFEE0_0000`. + +The virtual-APIC page should be accessible to the VMM so that the VMM can observe and maintain the guest’s virtual APIC state as needed. + +Depending on which APICv features are enabled, guest APIC operations may either: + +- still cause VM exits, or +- be satisfied directly by the CPU using the virtual-APIC page and related VMCS state. + +### APICv features covered by this RFC + +Document referenced (will be referenced in this RFC as "the Intel manual"): +``` +Intel® 64 and IA-32 Architectures Software Developer’s Manual +Combined Volumes: 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D, and 4 +Order Number: 325462-088US +June 2025 +``` + +This RFC covers the following APICv-related controls. They can be turned on by setting the corresponding feature enable bits in the VMCS. If the host CPU does not support the requested feature bit, it will be cleared to zero automatically by the kernel, note that this is an existing kernel behaviour. + +The relevant bit in the "Primary Processor-Based VM-Execution Controls" field of the VMCS is: +- Bit 21: Use TPR shadow. Allow the CPU to service `mov` to and from CR8 automatically from TPR value inside the "Virtual APIC" page. This is particularly important for Windows guests, which update CR8 / TPR very frequently. See section 31.3 "VIRTUALIZING CR8-BASED TPR ACCESSES" of the Intel manual. + + +The relevant bits in the "Secondary Processor-Based VM-Execution Controls" field of the VMCS are: +- Bit 0: Virtualise APIC accesses. Accesses by the guest to the APIC-access page are recognised specially by VT-x. These accesses may still cause VM exits, but the exit reason is specifically identified as an APIC-access exit. On its own, this feature does not eliminate all APIC-related exits. Instead, it forms part of another APICv control below. See section 31.4 "VIRTUALIZING MEMORY-MAPPED APIC ACCESSES" of the Intel manual. + +- Bit 8: APIC-register virtualisation. An extension of the above, but automatically satisfy reads and writes of a vast range of APIC registers from the "virtual APIC" page instead of causing a VM Exit. In effect, this allows the CPU to emulate selected APIC register accesses in hardware, using the virtual APIC state supplied by userspace. This reduces the amount of APIC emulation work that must be done by the VMM. See section 31.4.2 "Virtualizing Reads from the APIC-Access Page" and 31.4.3 "Virtualizing Writes to the APIC-Access Page" of the Intel manual. + +- Bit 9: Virtual-interrupt delivery. Allow the CPU to automatically evaluate pending interrupts and deliver them whenever the VCPU is resumed and the architectural rules permit delivery. Without this feature, the VMM must typically decide whether an interrupt is currently deliverable and then inject it manually. With virtual-interrupt delivery, more of that decision-making and delivery can happen in hardware. See section 31.2 "EVALUATION AND DELIVERY OF VIRTUAL INTERRUPTS" of the Intel manual. + +As an extension to Virtual-interrupt delivery. To be notified when a guest has finished servicing a virtual interrupt, set the interrupt vector's bit in the VMCS's "EOI-exit bitmap". Once the guest APIC driver issues an EOI, a VM Exit will be triggered by the CPU with exit reason "Virtualised EOI". The interrupt vector will be in the fault qualification message register. This is useful when you need to update certain virtual hardware state or acknowledge a host interrupt line. + +### How users should think about this feature + +Users should think about this RFC as providing kernel mechanisms for safely using APICv, not a kernel-managed virtual APIC. + +The kernel does not take over local-APIC emulation. Userspace still owns the virtual interrupt-controller design and remains responsible for: + +- maintaining correct virtual APIC state, +- choosing which APICv controls to enable, +- falling back when features are unsupported, and +- ensuring that the overall VMCS configuration is consistent. + +This RFC simply gives userspace a way to attach the required APICv pages to a VCPU so that VMMs can take advantage of the relevant hardware support. + +As with existing VCPU interface, the kernel validates certain VMCS writes to ensure that kernel security properties guarantees cannot be violated. Such as disabling external interrupt exiting to allow a VCPU to monopolise the CPU time. However, the kernel does not attempt to check that the VCPU's full VMCS configuration is architecturally correct. As before, userspace remains responsible for constructing a valid VMCS setup, otherwise, resuming a VCPU with invalid values in the VMCS will trigger an immediate VM exit with reason "VM-entry failure due to invalid guest state". + +## Reference-level explanation + + + +Intel APICv is not a single feature bit. It is a family of related VT-x controls and VMCS fields that together provide hardware assistance for local-APIC virtualisation. + +This RFC covers only the subset that has been explored and tested in our VMM implementation work: + +- Use TPR shadow +- Virtualise APIC accesses +- APIC-register virtualisation +- Virtual-interrupt delivery +- Virtualised End-of-Interrupt (EOI) + +Other APICv-related functionality, including x2APIC virtualisation, IPI virtualisation, and posted interrupts, is left out of scope for the initial proposal. + +### Prototype + +We've already implemented a prototype of this RFC here: + +Prototype in-kernel implementation: https://github.com/seL4/seL4/compare/master...au-ts:seL4:x86_vmm_dev + +Example integration with Microkit: https://github.com/au-ts/microkit/tree/apicv_poc + +Example usage in our VMM: https://github.com/au-ts/libvmm/pull/198 + +### Overview + +Every x86 VCPU has an associated VMCS. Intel APICv requires we write the physical addresses of two 4 KiB pages into the corresponding VMCS fields: + +- the APIC-access address, and +- the virtual-APIC address + +The main design choice in this RFC is that these two pages are supplied by userspace rather than embedded inside the kernel’s VCPU object. Previously, the kernel allows userspace to turn on APICv features but does not provide any mechanisms to set up these pages. + +Accordingly, when the kernel is built with `INTEL_APICV`, it exposes a new VCPU invocation: + +`seL4_X86_VCPU_SetAPICvPages()` + +This call take two 4 KiB frame capabilities and writes their physical addresses into the `VMX_CONTROL_APIC_ACCESS_ADDRESS` and `VMX_CONTROL_VIRTUAL_APIC_ADDRESS` VMCS fields of the VCPU. + +The kernel’s role is intentionally limited. It does not implement a virtual APIC and does not make policy decisions about how a VMM should use APICv. Instead, it: + +- exposes the VMCS write mechanism needed for APICv, +- validates the supplied capabilities and frame sizes, +- ensures the addresses written into the VMCS are derived from valid objects, and +- allows userspace to combine this with the existing VMCS control configuration interface. + +### CPU support + +APICv support is fragmented across CPU generations and features. Some processors may support only a subset of APICv features. + +For that reason, this RFC does not treat APICv as an all-or-nothing capability. The proposed model is: + +- If `INTEL_APICV` is disabled, no APICv support is exposed and kernel behaviour is the same as before, preserving backward compatibility. +- If `INTEL_APICV` is enabled, the kernel checks that hardware supports at least 1 APICv feature, enable APICv pages configuration and ensure that only the supported APICv features can be turned on by userspace. + +In other words, the kernel provides the mechanism, while userspace remains responsible for feature selection and fallback behaviour. + +This matches the existing spirit of the x86 VCPU interface: the kernel provides access to VMX mechanisms without attempting to enforce high-level VMM policy. + +## Drawbacks + + + +This RFC covers only a subset of the broader APICv feature family. As additional APICv-related features such as x2APIC virtualisation or posted interrupts are considered in future, care will be needed to avoid introducing an API that is awkward to extend. + +## Rationale and alternatives + + + +The proposed design aims to expose the minimum kernel support needed for userspace VMMs to benefit from Intel APICv without moving interrupt-controller policy into the kernel. + +This is desirable for several reasons +- It preserves the current seL4 approach in which the kernel provides low-level virtualisation mechanisms, while userspace remains responsible for higher-level VMM design and device emulation. +- It keeps the feature opt-in. If the build configuration `INTEL_APICV` is disabled, there is no behavioural change. Existing users and VMMs remain unaffected. +- It avoids increasing the size of the VCPU object for all users. Only VMMs that actually want APICv need to allocate and manage the two additional 4 KiB pages. + +### Alternative 1: embed the APICv pages inside the VCPU object + +One alternative would be to treat the APIC-access page and virtual-APIC page as implicit parts of the VCPU object, much like the VMCS page in `struct vcpu`. + +Under this alternate design, the kernel would allocate storage for these pages automatically and would write the corresponding VMCS fields internally. Userspace would only need to enable the desired APICv control bits. + +The main drawback of this approach is object size. It would increase the size of every VCPU object by two additional 4 KiB pages, even on systems that never use APICv. In our case, this would increase the VCPU object size from size bit 14 to 15. + +Because APICv is an optional performance optimisation rather than a universal requirement, this proposal does not consider that space trade-off worthwhile. + +### Alternative 2: implement a higher-level kernel virtual APIC abstraction + +Another alternative would be for the kernel to provide a more abstract virtual interrupt-controller interface and hide APICv behind it. + +This RFC does not take that approach. Doing so would move substantial VMM policy into the kernel and require the kernel to understand and maintain significantly more local-APIC emulation logic. + +That would be a much larger design change than what is needed here. + +For this reason, we believe the proposed design is the best trade-off between performance benefit and complexity. + +## Prior art + + + +There is prior art for this RFC in Linux KVM, where APICv is used as an acceleration feature. KVM's documentation notes that on sufficiently recent Intel processors, [APICv features are enabled by default for nested virtualisation](https://docs.kernel.org/virt/kvm/x86/running-nested-guests.html#additional-nested-related-kernel-parameters-x86). + +Xen also adopted a similar policy to KVM, where [APICv will be enabled by default](https://xenbits.xen.org/docs/4.15-testing/misc/xen-command-line.html#apicv-intel). + + + +## Unresolved questions + + + +1. Is it more desirable if we follow KVM and Xen's approach to enable APICv by default rather than gating it behind a build time flag? Of course, userspace still get to decide which specific features to use. The kernel role is still providing the mechanisms needed.