From 84fdc5167e3ee3a712be60eb8ba715e5a8b5e88c Mon Sep 17 00:00:00 2001 From: Alex Ball Date: Thu, 30 Jul 2026 17:09:47 -0700 Subject: [PATCH 01/15] docs: add badges, adjust tags Signed-off-by: Alex Ball --- docs/configuration/component-manager-rms.md | 2 +- docs/configuration/dns.md | 2 +- docs/configuration/machine_identity.md | 20 +++-- docs/configuration/secrets-storage.md | 2 +- docs/hcl.md | 89 +++++++++---------- docs/manuals/vpc/flat_vpcs_zero_dpu.md | 4 +- docs/operations/firmware-updates.md | 2 +- docs/operations/monitoring-health.md | 2 +- docs/operations/nvos-password-rotation.md | 2 +- docs/operations/tenant-lifecycle-cleanup.md | 2 +- .../boot-interfaces-and-dpu-modes.md | 2 +- docs/provisioning/machine-validation.md | 2 +- fern/changelog/2026-01-23.mdx | 2 +- fern/changelog/2026-02-12.mdx | 4 +- fern/changelog/2026-05-15.mdx | 4 +- fern/changelog/2026-07-31.mdx | 8 +- 16 files changed, 77 insertions(+), 72 deletions(-) diff --git a/docs/configuration/component-manager-rms.md b/docs/configuration/component-manager-rms.md index c7ce9a647c..651a4d999e 100644 --- a/docs/configuration/component-manager-rms.md +++ b/docs/configuration/component-manager-rms.md @@ -1,4 +1,4 @@ -# Component Manager RMS Backends (Day 1) +# Component Manager RMS Backends (Day 1) v2.0 New Operator guide for configuring **Rack Manager Service (RMS)** backends in the `[component_manager]` section of `nico-api` site config, and the **rack profile** diff --git a/docs/configuration/dns.md b/docs/configuration/dns.md index aa0a0522c2..1319af3e0c 100644 --- a/docs/configuration/dns.md +++ b/docs/configuration/dns.md @@ -1,4 +1,4 @@ -# DNS +# DNS v2.0 New NICo answers DNS for everything it manages. Records are never authored by hand: they derive from the machine, BMC, and instance inventory in the `nico-api` database, appear when an interface or instance gains an address, and disappear when it loses one. This page covers the names NICo serves, how the site zone and per-segment subdomains are configured, and how reverse (PTR) resolution works. For the deployment side - the `nico-dns` service, the recursive resolver in front of it, and the fixed infrastructure service names - refer to [IP and Network Configuration](../provisioning/ip-and-network-configuration.md#3-dns-configuration). diff --git a/docs/configuration/machine_identity.md b/docs/configuration/machine_identity.md index 5855fae0d6..6f0507268e 100644 --- a/docs/configuration/machine_identity.md +++ b/docs/configuration/machine_identity.md @@ -1,4 +1,4 @@ -# Machine Identity (Day 1) +# Machine Identity (Day 1) v2.0 New Operator guide for per-organization **machine identity** configuration: JWT-SVID issuance for tenant workloads, optional RFC 8693 token delegation, discovery endpoints, verification, and signing-key rotation. @@ -118,12 +118,12 @@ When token delegation is configured, NICo issues a short-lived **intermediate** **Endpoint:** `PUT /v2/org/{org}/nico/site/{siteID}/tenant-identity/token-delegation` -> **Recommendation:** Token delegation causes `nico-api` to call the org-configured `tokenEndpoint` over HTTP(S). For external token exchange URLs, configure site-level egress controls in `[machine_identity]` during [Day 0](../getting-started/installation-options/day0-machine-identity.md): -> -> - `token_endpoint_http_proxy` — route outbound token-exchange HTTP through a controlled egress proxy -> - `token_endpoint_domain_allowlist` — restrict which hostnames tenants may register on `tokenEndpoint` -> -> Together these mitigate SSRF-style risk if a tenant admin supplies an endpoint the API should not reach. They are optional at install time but **strongly recommended** for production sites that delegate to external hosts. +**Recommendation:** Token delegation causes `nico-api` to call the org-configured `tokenEndpoint` over HTTP(S). For external token exchange URLs, configure site-level egress controls in `[machine_identity]` during [Day 0](../getting-started/installation-options/day0-machine-identity.md): + +- `token_endpoint_http_proxy` — route outbound token-exchange HTTP through a controlled egress proxy +- `token_endpoint_domain_allowlist` — restrict which hostnames tenants may register on `tokenEndpoint` + +Together these mitigate SSRF-style risk if a tenant admin supplies an endpoint the API should not reach. They are optional at install time but **strongly recommended** for production sites that delegate to external hosts. Example (adjust fields to match your STS): @@ -142,9 +142,11 @@ curl -sS -X PUT \ }' ``` -> **PUT is full replace:** omitting `clientSecretBasic` on an update clears stored credentials. Re-supply secrets on every update that should keep basic auth. +**PUT is full replace:** omitting `clientSecretBasic` on an update clears stored credentials. Re-supply secrets on every update that should keep basic auth. -> **Note:** `tokenEndpoint` may use `http://` with an IP address (for example a node-local sidecar). NICo allows this for in-instance STS; use allowlists and network policy in production where appropriate. + +`tokenEndpoint` may use `http://` with an IP address (for example a node-local sidecar). NICo allows this for in-instance STS; use allowlists and network policy in production where appropriate. + ```bash # Remove delegation (return to direct signing) diff --git a/docs/configuration/secrets-storage.md b/docs/configuration/secrets-storage.md index 34e489ffb1..1419af2ffd 100644 --- a/docs/configuration/secrets-storage.md +++ b/docs/configuration/secrets-storage.md @@ -1,4 +1,4 @@ -# Secrets Storage +# Secrets Storage v2.0 New NICo keeps the credentials it manages (BMC logins, switch and UFM accounts, factory defaults, and so on) in a credentials store. Vault is the historical and default backend. NICo can also store credentials in Postgres, encrypted per credential with envelope encryption, and can read both backends side by side; that combination is what makes a gradual, reversible migration off Vault possible. diff --git a/docs/hcl.md b/docs/hcl.md index f698a52b92..6749f069f5 100644 --- a/docs/hcl.md +++ b/docs/hcl.md @@ -8,59 +8,59 @@ inquiries or technical specifications, please contact the original hardware vend **Last Updated: 2/20/2026** -| Host Machine | BMC/Management Firmware Version | BIOS/UEFI Version | Misc. Firmware Version (FPGA, CPLD, LXPM, etc.) | -|-------------------------------------|-------------------------------------|---------------------|-------------------------------------------------| -| GB200 NVL - Wiwynn | 25.06-2_NV_WW_02 | 1.3.2GA | 1.3.2GA | -| NVSwitch Tray - Wiwynn | 1.3.2GA | 1.3.2GA | 1.3.2GA | -| GB200 Compute Tray (1RU) | 1.3.2GA | 1.3.2GA | 1.3.2GA | -| NVSwitch Tray DGX | 1.3.2GA | 1.3.2GA | 1.3.2GA | -| DGX H100 | 25.06.27 (DGXH100_H200_25.06.4 pkg) | 1.06.07 (DGXH100_H200_25.06.4 pkg) | | -| Lenovo ThinkSystem SR670 V2 | 6.10 | 3.30 | 3.31.01 | -| Lenovo ThinkSystem SR675 V3 | 14.10 | 8.30 | 4.20.03 | -| Lenovo ThinkSystem SR675 V3 OVX* | 14.10 | 8.30 | 4.20.03 | -| Lenovo ThinkSystem SR650 | 10.40 | 4.30 | 2.13 | -| Lenovo ThinkSystem SR650 V3 | 6.92 | 3.70 | 4.21.01 | -| Lenovo ThinkSystem SR650 V2 | 5.70 | 3.60 | 3.31.01 | -| Lenovo ThinkSystem SR650 V2 OVX* | 5.70 | 3.60 | 3.31.01 | -| Lenovo ThinkSystem SR655 V3 | 5.80 | 5.70 | 4.20.03 | -| Lenovo ThinkSystem SR655 V3 OVX* | 5.80 | 5.70 | 4.20.03 | -| Lenovo ThinkSystem SR665 V3 OVX* | 5.80 | 5.70 | 4.20.03 | -| Lenovo SR650 V4 | 1.90 | 1.30 | 5.03.00 | -| Lenovo HS350X V3 | 1.20 | 2.17.0 | | -| Dell PowerEdge XE9680 | iDRAC 7.20.60.50 | 2.7.4 | 1.6.0 | -| Dell PowerEdge R750 | iDRAC 7.20.60.50 | 1.18.1 | 1.1.1 | -| SYS-221H-TNR | 1.03.18 | 2.7 | SAA Ver = 1.3.0-p7 | -| Dell PowerEdge R760 | iDRAC 7.20.60.50 | 2.7.5 | 1.2.6 | -| ARS-121L-DNR | 01.08.02 / 01.03.16 (LCC) | 2.2a / 2.0 (LCC) | SAA Ver = 1.2.0-p6 / SUM = 2.14.0-p6 (LCC) | -| SYS-221H-TN24R | X1.05.10 | 2.7 | SAA Ver = 1.3.0-p5 | -| ARS-221GL-NR | 1.03.16 | 2.0 | | -| HPE ProLiant DL385 Gen10 Plus v2 | 3.15 | 3.80_09-05-2025 | | -| DL380 Gen12 | 1.20.00 | 1.62_02-06-2026 | | -| SSG-121E-NES24R | 01.04.19 | 2.7 | SAA Ver = 1.3.0-p1 | -| SYS-121H-TNR | X1.05.10 | 2.7 | SAA Ver = 1.3.0-p5 | -| SYS-821GE-TNHR | 1.03.18 | 2.7 | SAA Ver = 1.3.0-p7 | -| Dell R760xd2 | iDRAC 7.20.80.50 | 2.9.4 | 1.1.2 | -| Dell R670 | iDRAC 1.20.80.51 | 1.7.5 | | -| Dell R770 | iDRAC 1.20.80.51 | 1.7.5 | | -| SYS-421GE-TNRT | 1.03.19 | 2.6 | SAA Ver = 1.2.0-p8 | -| Dell PowerEdge R640 | iDRAC 7.00.00.182 | 2.24.0 | 1.0.6 | +| Host Machine | BMC/Management Firmware Version | BIOS/UEFI Version | Misc. Firmware Version (FPGA, CPLD, LXPM, etc.) | +| ------------ | ------------------------------- | ----------------- | ----------------------------------------------- | +| GB200 NVL - Wiwynn | 25.06-2_NV_WW_02 | 1.3.2GA | 1.3.2GA | +| NVSwitch Tray - Wiwynn | 1.3.2GA | 1.3.2GA | 1.3.2GA | +| GB200 Compute Tray (1RU) | 1.3.2GA | 1.3.2GA | 1.3.2GA | +| NVSwitch Tray DGX | 1.3.2GA | 1.3.2GA | 1.3.2GA | +| DGX H100 | 25.06.27 (DGXH100_H200_25.06.4 pkg) | 1.06.07 (DGXH100_H200_25.06.4 pkg) | | +| Lenovo ThinkSystem SR670 V2 | 6.10 | 3.30 | 3.31.01 | +| Lenovo ThinkSystem SR675 V3 | 14.10 | 8.30 | 4.20.03 | +| Lenovo ThinkSystem SR675 V3 OVX* | 14.10 | 8.30 | 4.20.03 | +| Lenovo ThinkSystem SR650 | 10.40 | 4.30 | 2.13 | +| Lenovo ThinkSystem SR650 V3 | 6.92 | 3.70 | 4.21.01 | +| Lenovo ThinkSystem SR650 V2 | 5.70 | 3.60 | 3.31.01 | +| Lenovo ThinkSystem SR650 V2 OVX* | 5.70 | 3.60 | 3.31.01 | +| Lenovo ThinkSystem SR655 V3 | 5.80 | 5.70 | 4.20.03 | +| Lenovo ThinkSystem SR655 V3 OVX* | 5.80 | 5.70 | 4.20.03 | +| Lenovo ThinkSystem SR665 V3 OVX* | 5.80 | 5.70 | 4.20.03 | +| Lenovo SR650 V4 | 1.90 | 1.30 | 5.03.00 | +| Lenovo HS350X V3 | 1.20 | 2.17.0 | | +| Dell PowerEdge XE9680 | iDRAC 7.20.60.50 | 2.7.4 | 1.6.0 | +| Dell PowerEdge R750 | iDRAC 7.20.60.50 | 1.18.1 | 1.1.1 | +| SYS-221H-TNR | 1.03.18 | 2.7 | SAA Ver = 1.3.0-p7 | +| Dell PowerEdge R760 | iDRAC 7.20.60.50 | 2.7.5 | 1.2.6 | +| ARS-121L-DNR | 01.08.02 / 01.03.16 (LCC) | 2.2a / 2.0 (LCC) | SAA Ver = 1.2.0-p6 / SUM = 2.14.0-p6 (LCC) | +| SYS-221H-TN24R | X1.05.10 | 2.7 | SAA Ver = 1.3.0-p5 | +| ARS-221GL-NR | 1.03.16 | 2.0 | | +| HPE ProLiant DL385 Gen10 Plus v2 | 3.15 | 3.80_09-05-2025 | | +| DL380 Gen12 | 1.20.00 | 1.62_02-06-2026 | | +| SSG-121E-NES24R | 01.04.19 | 2.7 | SAA Ver = 1.3.0-p1 | +| SYS-121H-TNR | X1.05.10 | 2.7 | SAA Ver = 1.3.0-p5 | +| SYS-821GE-TNHR | 1.03.18 | 2.7 | SAA Ver = 1.3.0-p7 | +| Dell R760xd2 | iDRAC 7.20.80.50 | 2.9.4 | 1.1.2 | +| Dell R670 | iDRAC 1.20.80.51 | 1.7.5 | | +| Dell R770 | iDRAC 1.20.80.51 | 1.7.5 | | +| SYS-421GE-TNRT | 1.03.19 | 2.6 | SAA Ver = 1.2.0-p8 | +| Dell PowerEdge R640 | iDRAC 7.00.00.182 | 2.24.0 | 1.0.6 | \* OVX may not show up as an option; check the Server Serial Number to confirm. -### Hosts -- Under Development +### Hosts -- Under Development Upcoming This list outlines platforms that are under development and have not undergone full unit testing. -| Host Machine | BMC/Management Firmware Version | BIOS/UEFI Version | Provisioning Manager Version | -|-------------------------------------|-------------------------------------|---------------------|--------------------------------------------| -| Lenovo GB300 Compute Tray | 3.0.0 | 1.0.0GA | 1.0.0GA | +| Host Machine | BMC/Management Firmware Version | BIOS/UEFI Version | Provisioning Manager Version | +| ------------ | ------------------------------- | ----------------- | ---------------------------- | +| Lenovo GB300 Compute Tray | 3.0.0 | 1.0.0GA | 1.0.0GA | ## DPUs -| DPU | Firmware / Software Version | -|--------------|---------------------------------------------------| -| BlueField 2 | DOCA 3.2.0 | -| BlueField 3 | DOCA 3.2.0 | +| DPU | Firmware / Software Version | +| --- | --------------------------- | +| BlueField 2 | DOCA 3.2.0 | +| BlueField 3 | DOCA 3.2.0 | ## GPUs @@ -71,4 +71,3 @@ Unlike host platforms, GPU compatibility is not validated on a per-platform basi | GPU Family | Support | | ---------- | ------- | | Current NVIDIA Data Center GPUs | Supported | - diff --git a/docs/manuals/vpc/flat_vpcs_zero_dpu.md b/docs/manuals/vpc/flat_vpcs_zero_dpu.md index 9f1a71c417..9597459977 100644 --- a/docs/manuals/vpc/flat_vpcs_zero_dpu.md +++ b/docs/manuals/vpc/flat_vpcs_zero_dpu.md @@ -1,4 +1,4 @@ -# Flat VPCs and Zero-DPU Hosts +# Flat VPCs and Zero-DPU Hosts v2.0 New `Flat` is a VPC virtualization type for tenant instances that run on hosts **without a NICo-managed DPU** — either hosts with no DPU hardware at all, or @@ -371,7 +371,7 @@ Rules enforced at allocation: Read instance status through the REST API or `nicocli`: -``` +```text GET /v2/org/{org}/nico/instance/{id} nicocli instance get ``` diff --git a/docs/operations/firmware-updates.md b/docs/operations/firmware-updates.md index a681047e89..105a5d052b 100644 --- a/docs/operations/firmware-updates.md +++ b/docs/operations/firmware-updates.md @@ -1,4 +1,4 @@ -# Firmware Updates +# Firmware Updates New This guide first gives a high-level view of firmware updates in NICo and helps you choose the workflow that applies to your hardware. The linked workflow diff --git a/docs/operations/monitoring-health.md b/docs/operations/monitoring-health.md index 658583d531..23039323e5 100644 --- a/docs/operations/monitoring-health.md +++ b/docs/operations/monitoring-health.md @@ -1,4 +1,4 @@ -# Monitoring and Health +# Monitoring and Health New This page covers monitoring and health workflows for NICo sites after deployment: hardware health, DPU health, aggregate host health, health diff --git a/docs/operations/nvos-password-rotation.md b/docs/operations/nvos-password-rotation.md index a94473f1d0..a96f8a0b3e 100644 --- a/docs/operations/nvos-password-rotation.md +++ b/docs/operations/nvos-password-rotation.md @@ -1,4 +1,4 @@ -# NVOS Password Rotation +# NVOS Password Rotation v2.0 New Use this workflow to rotate the NVOS admin password across managed NVLink switches, monitor site-wide convergence, and recover switches that do not reach the published target. diff --git a/docs/operations/tenant-lifecycle-cleanup.md b/docs/operations/tenant-lifecycle-cleanup.md index b4284ec40a..84f8cdca3d 100644 --- a/docs/operations/tenant-lifecycle-cleanup.md +++ b/docs/operations/tenant-lifecycle-cleanup.md @@ -1,4 +1,4 @@ -# Tenant Lifecycle Cleanup +# Tenant Lifecycle Cleanup New Use this workflow to release an instance, track NICo cleanup progress, and verify that the host is ready for reuse. diff --git a/docs/provisioning/boot-interfaces-and-dpu-modes.md b/docs/provisioning/boot-interfaces-and-dpu-modes.md index aef0e9ebdd..33d517c5d8 100644 --- a/docs/provisioning/boot-interfaces-and-dpu-modes.md +++ b/docs/provisioning/boot-interfaces-and-dpu-modes.md @@ -1,4 +1,4 @@ -# Boot Interfaces and DPU Policies +# Boot Interfaces and DPU Policies v2.0 New This guide explains how NICo decides **which interface a host boots from**, how a host's **DPUs are managed**, and how operators configure both through the Expected Machines table. It is the deep companion to [Ingesting Hosts](ingesting-hosts.md): that page covers the end-to-end ingest flow and the basic `expected_machines.json`; this page covers the per-host and per-NIC knobs (`dpu_policy`, `host_nics`), **what the defaults do when you set nothing**, and how a boot device is chosen and applied behind the scenes. diff --git a/docs/provisioning/machine-validation.md b/docs/provisioning/machine-validation.md index 3ec36e7807..faae2aaaa2 100644 --- a/docs/provisioning/machine-validation.md +++ b/docs/provisioning/machine-validation.md @@ -1,4 +1,4 @@ -# Machine Validation +# Machine Validation v2.0 Machine Validation is NVIDIA Infra Controller's in-band validation framework for checking a machine before it is made available to tenants. NICo uses Scout to run diff --git a/fern/changelog/2026-01-23.mdx b/fern/changelog/2026-01-23.mdx index e2d58681f4..97ccd3ba78 100644 --- a/fern/changelog/2026-01-23.mdx +++ b/fern/changelog/2026-01-23.mdx @@ -2,7 +2,7 @@ tags: ["EOL", "deployment", "gb200", "nvlink"] --- -## NVIDIA Infra Controller EA End of Life +## NVIDIA Infra Controller EA EOL NVIDIA Infra Controller EA introduced a self-hosted, disconnected deployment model with source and build instructions that do not depend on NGC. diff --git a/fern/changelog/2026-02-12.mdx b/fern/changelog/2026-02-12.mdx index 6e027a50dd..41989fd06c 100644 --- a/fern/changelog/2026-02-12.mdx +++ b/fern/changelog/2026-02-12.mdx @@ -1,8 +1,8 @@ --- -tags: ["EOL", "rest-api", "provisioning", "nvlink"] +tags: ["EOL", "nvlink", "provisioning", "rest-api"] --- -## NVIDIA Infra Controller v0.2 End of Life +## NVIDIA Infra Controller v0.2 EOL NVIDIA Infra Controller v0.2 made the project available as open-source software and expanded its APIs, orchestration, and hardware lifecycle support. diff --git a/fern/changelog/2026-05-15.mdx b/fern/changelog/2026-05-15.mdx index bee429f76f..6d475e516f 100644 --- a/fern/changelog/2026-05-15.mdx +++ b/fern/changelog/2026-05-15.mdx @@ -1,8 +1,8 @@ --- -tags: ["current", "deployment", "rack-level-administration", "gb200"] +tags: ["maintenance", "deployment", "gb200", "rack-level-administration"] --- -## NVIDIA Infra Controller v0.8 Current +## NVIDIA Infra Controller v0.8 Maintenance NVIDIA Infra Controller v0.8 introduced a unified documentation site, simplified deployment, and expanded rack-level administration for GB200 systems. diff --git a/fern/changelog/2026-07-31.mdx b/fern/changelog/2026-07-31.mdx index 74e6365797..8a55e50c0e 100644 --- a/fern/changelog/2026-07-31.mdx +++ b/fern/changelog/2026-07-31.mdx @@ -1,8 +1,12 @@ --- -tags: ["next", "deployment", "ntp", "security", "database"] +tags: ["current", "database", "deployment", "dpu", "machine-identity", "multi-arch", "networking", "ntp", "rest-api", "security", "telemetry", "validation", "zero-dpu"] --- -## NVIDIA Infra Controller v2.0 Next +## NVIDIA Infra Controller v2.0 Current + +NICo v2.0 expands bare-metal lifecycle management with built-in NTP, broader hardware telemetry, multi-architecture support, and stronger security and operational controls. + +Before upgrading an existing deployment, review the REST API database migration requirements below. The NICo REST API database has moved from a standalone PostgreSQL instance to the shared Zalando-managed `nico-pg-cluster`. A new `nico_rest` database and `nico-rest.nico` user are provisioned automatically (gated on `rest.enabled: true`). Database credentials are now synced by External Secrets Operator as the `nico-rest-pg-creds` secret in the `nico-rest` namespace, replacing the former `db-creds` secret. From b0397c7f9652be22af2cc3331f2b7d5f6e9c59bd Mon Sep 17 00:00:00 2001 From: Alex Ball Date: Fri, 31 Jul 2026 10:42:12 -0700 Subject: [PATCH 02/15] fern bump, add redirects Signed-off-by: Alex Ball --- fern/docs.yml | 6 ++++++ fern/fern.config.json | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/fern/docs.yml b/fern/docs.yml index ceaaa2bc4a..6bf942cd1d 100644 --- a/fern/docs.yml +++ b/fern/docs.yml @@ -44,5 +44,11 @@ redirects: destination: "/infra-controller/documentation/getting-started/quick-start-guide" - source: "/infra-controller/documentation/introduction" destination: "/infra-controller/documentation/overview/what-is-nico" + - source: "/infra-controller/documentation/operations-day-2/firmware-updates" + destination: "/infra-controller/documentation/operations-day-2/firmware-updates/overview" + - source: "/infra-controller/documentation/operations-day-2/observability/core-metrics" + destination: "/infra-controller/documentation/operations-day-2/observability/core-metrics-list" + - source: "/infra-controller/documentation/provisioning-day-0/boot-interfaces-and-dpu-modes" + destination: "/infra-controller/documentation/provisioning-day-0/boot-interfaces-and-dpu-policies" - source: "/infra-controller/documentation/reference/release-notes" destination: "/infra-controller/documentation/release-notes" diff --git a/fern/fern.config.json b/fern/fern.config.json index 22bc767300..162a343345 100644 --- a/fern/fern.config.json +++ b/fern/fern.config.json @@ -1,4 +1,4 @@ { "organization": "nvidia", - "version": "5.80.4" + "version": "5.89.1" } From 565a1691289c7f0fa924c4d1877862fcbd60c5b8 Mon Sep 17 00:00:00 2001 From: Jay Zhu Date: Thu, 30 Jul 2026 21:28:10 -0600 Subject: [PATCH 03/15] fix(migration): dedup SQL migration versions (#4428) Two database migrations on `main` use SQLx version `20260722120000`. Because SQLx records migration versions as primary keys, fresh database migration fails when the second migration is recorded, blocking Core CI and dependent PRs. This change assigns the newer BMC suppression migration version `20260722120001` without changing its SQL content. It also adds a regression test covering all legacy, squash, and post-squash migrations so duplicate versions fail with a focused diagnostic. ## Related issues Resolves #4427 ## Type of Change - [ ] **Add** - New feature or capability - [ ] **Change** - Changes in existing functionality - [x] **Fix** - Bug fixes - [ ] **Remove** - Removed features or deprecated functionality - [ ] **Internal** - Internal changes (refactoring, tests, docs, etc.) ## Breaking Changes - [ ] **This PR contains breaking changes** ## Testing - [x] Unit tests added/updated - [ ] Integration tests added/updated - [x] Manual testing performed - [ ] No testing required (docs, internal refactor, etc.) ## Additional Notes Databases that recorded `preserve_machine_ipv6_loopback` as version `20260722120000` will apply the BMC suppression migration as `20260722120001` normally. Any database that already recorded `bmc_suppressions` as `20260722120000` requires migration-history repair before upgrading. Signed-off-by: Jay Zhu Signed-off-by: Alex Ball --- ...ql => 20260722120001_bmc_suppressions.sql} | 0 crates/api-db/src/migrations/mod.rs | 21 +++++++++++++++++++ 2 files changed, 21 insertions(+) rename crates/api-db/migrations/{20260722120000_bmc_suppressions.sql => 20260722120001_bmc_suppressions.sql} (100%) diff --git a/crates/api-db/migrations/20260722120000_bmc_suppressions.sql b/crates/api-db/migrations/20260722120001_bmc_suppressions.sql similarity index 100% rename from crates/api-db/migrations/20260722120000_bmc_suppressions.sql rename to crates/api-db/migrations/20260722120001_bmc_suppressions.sql diff --git a/crates/api-db/src/migrations/mod.rs b/crates/api-db/src/migrations/mod.rs index a31ee059c8..89d27fa6e2 100644 --- a/crates/api-db/src/migrations/mod.rs +++ b/crates/api-db/src/migrations/mod.rs @@ -199,6 +199,27 @@ async fn load_and_validate_history( mod tests { use super::*; + #[test] + fn migration_versions_are_unique() { + let mut versions = HashSet::new(); + + for migration in std::iter::once(&MIGRATION_LAYOUT.legacy) + .chain( + MIGRATION_LAYOUT + .epochs + .iter() + .flat_map(|epoch| [&epoch.squash, &epoch.post_squash]), + ) + .flat_map(|migrator| migrator.iter()) + { + assert!( + versions.insert(migration.version), + "duplicate migration version {}", + migration.version + ); + } + } + #[test] fn epochs_are_ordered_and_point_to_their_squash_migration() { let mut previous = None; From 595c86295af465ce2225c3fc8eee8248a0153216 Mon Sep 17 00:00:00 2001 From: Chet Nichols III Date: Thu, 30 Jul 2026 20:57:47 -0700 Subject: [PATCH 04/15] feat(machine-controller): converge pending boot interface intent (#4389) A desired boot interface can change while an unassigned host is already `Ready`, but the machine controller did not have a durable answer to "has Redfish actually been checked for this desired version?" Treating the desired row itself as proof would make restart and race behavior ambiguous, and could let allocation win while repair is pending. So, this adds a persisted verification observation and a restart-safe `BootConfiguring` path. A `Ready` host captures the exact desired target and version, observes before mutation, reuses the shared HostInit BIOS/job/boot-order work, restores and verifies lockdown, then rereads Redfish and compare-and-swaps only the version it actually checked to verified. An ordinary host that is already correct stays observation-only. The migration gives rows whose machines are already `Ready` or `Assigned` an explicitly assumed baseline instead of scheduling fleet-wide Redfish work. In-flight lifecycle rows remain pending a real observation. This is a small machine-state-gated update with no discovery-dependent joins. MAC-only to full-pair enrichment also carries a current verification forward because adding the Redfish id still names the same physical NIC. This intentionally leaves assigned hosts pending and does not periodically look for later external drift; those are separate follow-ups. It also keeps `BootConfiguring` hosts on discovery iPXE during required reboots and keeps a pending host out of allocation until its desired version is verified. ## Related issues Closes #4246 ## Type of Change - [x] **Add** - New feature or capability - [ ] **Change** - Changes in existing functionality - [ ] **Fix** - Bug fixes - [ ] **Remove** - Removed features or deprecated functionality - [ ] **Internal** - Internal changes (refactoring, tests, docs, etc.) ## Breaking Changes - [ ] **This PR contains breaking changes** ## Testing - [x] Unit tests added/updated - [x] Integration tests added/updated - [ ] Manual testing performed - [ ] No testing required (docs, internal refactor, etc.) ## Additional Notes - `cargo make check-format-nightly` - `cargo make clippy` - `cargo make carbide-lints` - 17 focused API-model boot-interface tests plus the state-label regression - 6 machine-controller Ready boot-config tests - 6 PostgreSQL-backed Ready recovery tests - 12 PostgreSQL migration, versioning, compare-and-swap, and lock-order tests - End-to-end desired-change to verified-Ready progression test - iPXE reboot-state regression test - Independent state-controller, Rust-simplicity, and comment-voice audits - Cloud CodeRabbit review with all substantive threads addressed --------- Signed-off-by: Chet Nichols III Signed-off-by: Alex Ball --- crates/api-core/src/instance/mod.rs | 74 +- crates/api-core/src/ipxe.rs | 1 + .../src/tests/common/api_fixtures/mod.rs | 1 + crates/api-core/src/tests/ipxe.rs | 32 + crates/api-core/src/tests/machine_states.rs | 864 +++++++++- .../tests/integration/compute_allocation.rs | 3 +- .../tests/integration/forge_agent_control.rs | 2 +- ...120001_preserve_machine_ipv6_loopback.sql} | 0 ...30120000_machine_boot_interface_status.sql | 34 + crates/api-db/src/machine.rs | 18 +- .../src/machine_desired_boot_interface.rs | 540 +++++- .../src/sql/machine_snapshots.sql.template | 3 + .../api-db/src/sql/managed_hosts.sql.template | 6 + crates/api-model/src/machine/json.rs | 142 +- crates/api-model/src/machine/mod.rs | 337 +++- crates/api-model/src/machine/slas.rs | 4 + crates/api-model/src/machine/status.rs | 3 + .../api-model/src/machine_boot_interface.rs | 17 + .../src/test_support/machine_snapshot.rs | 9 +- crates/api-web/src/tests/env.rs | 3 +- .../bmc-mock/src/redfish/computer_system.rs | 163 +- crates/machine-controller/src/handler.rs | 1477 ++++++++++++++++- crates/machine-controller/src/io.rs | 54 +- crates/machine-controller/src/metrics.rs | 1 + .../tests/integration/bmc_rotation.rs | 6 +- .../tests/integration/maintenance.rs | 2 +- .../tests/integration/power_management.rs | 2 +- crates/redfish/src/libredfish/test_support.rs | 15 +- crates/test-harness/src/managed_host.rs | 40 + 29 files changed, 3715 insertions(+), 138 deletions(-) rename crates/api-db/migrations/{20260722120000_preserve_machine_ipv6_loopback.sql => 20260722120001_preserve_machine_ipv6_loopback.sql} (100%) create mode 100644 crates/api-db/migrations/20260730120000_machine_boot_interface_status.sql diff --git a/crates/api-core/src/instance/mod.rs b/crates/api-core/src/instance/mod.rs index bcb653646f..18c1a0e9c6 100644 --- a/crates/api-core/src/instance/mod.rs +++ b/crates/api-core/src/instance/mod.rs @@ -1324,6 +1324,27 @@ pub async fn allocate_instance( .ok_or_else(|| CarbideError::internal("instance allocation returned no result".to_string())) } +fn not_allocatable_error(machine_id: MachineId, reason: NotAllocatableReason) -> CarbideError { + match reason { + NotAllocatableReason::InvalidState(state) => CarbideError::InvalidArgument(format!( + "could not create instance on machine {machine_id} given machine state {state:?}" + )), + NotAllocatableReason::PendingInstanceCreation => CarbideError::InvalidArgument(format!( + "could not create instance on machine {machine_id}. machine is already used by another instance creation request", + )), + NotAllocatableReason::PendingBootConfiguration => { + CarbideError::FailedPrecondition(format!( + "machine {machine_id} has a pending boot configuration; retry after it has been applied" + )) + } + NotAllocatableReason::NoDpuSnapshots => { + CarbideError::internal(format!("machine {machine_id} has no DPU. cannot allocate")) + } + NotAllocatableReason::MaintenanceMode => CarbideError::MaintenanceMode, + NotAllocatableReason::HealthAlert(_) => CarbideError::UnhealthyHost, + } +} + /// Allocates multiple instances in a single transaction. /// Rolls back entirely if any allocation fails. /// @@ -1517,26 +1538,20 @@ pub async fn batch_allocate_instances( })?; if let Err(e) = mh_snapshot.is_usable_as_instance(request.allow_unhealthy_machine) { - tracing::error!( - %machine_id, - error = %e, - "Host can not be used as instance due to reason", - ); - return Err(match e { - NotAllocatableReason::InvalidState(s) => CarbideError::InvalidArgument(format!( - "could not create instance on machine {machine_id} given machine state {s:?}" - )), - NotAllocatableReason::PendingInstanceCreation => { - CarbideError::InvalidArgument(format!( - "could not create instance on machine {machine_id}. machine is already used by another instance creation request", - )) - } - NotAllocatableReason::NoDpuSnapshots => CarbideError::internal(format!( - "machine {machine_id} has no DPU. cannot allocate" - )), - NotAllocatableReason::MaintenanceMode => CarbideError::MaintenanceMode, - NotAllocatableReason::HealthAlert(_) => CarbideError::UnhealthyHost, - }); + if matches!(&e, NotAllocatableReason::PendingBootConfiguration) { + tracing::info!( + %machine_id, + error = %e, + "Host can not be used as instance due to reason", + ); + } else { + tracing::error!( + %machine_id, + error = %e, + "Host can not be used as instance due to reason", + ); + } + return Err(not_allocatable_error(machine_id, e)); } if mh_snapshot.host_snapshot.config.dpf.used_for_ingestion @@ -2335,6 +2350,25 @@ mod tests { }, ); } + + #[test] + fn pending_boot_configuration_has_a_safe_allocation_error() { + let machine_id = "fm100htes3rn1npvbtm5qd57dkilaag7ljugl1llmm7rfuq1ov50i0rpl30" + .parse() + .unwrap(); + + assert!(matches!( + not_allocatable_error( + machine_id, + NotAllocatableReason::PendingBootConfiguration, + ), + CarbideError::FailedPrecondition(message) + if message + == format!( + "machine {machine_id} has a pending boot configuration; retry after it has been applied" + ) + )); + } } #[cfg(test)] diff --git a/crates/api-core/src/ipxe.rs b/crates/api-core/src/ipxe.rs index 69dffb0bc3..7e4130ef1a 100644 --- a/crates/api-core/src/ipxe.rs +++ b/crates/api-core/src/ipxe.rs @@ -469,6 +469,7 @@ exit || let pxe_script = match &machine.current_state() { ManagedHostState::Ready | ManagedHostState::HostInit { .. } + | ManagedHostState::BootConfiguring { .. } | ManagedHostState::BomValidating { .. } | ManagedHostState::Measuring { measuring_state: MeasuringState::WaitingForMeasurements, diff --git a/crates/api-core/src/tests/common/api_fixtures/mod.rs b/crates/api-core/src/tests/common/api_fixtures/mod.rs index f99bdd69d1..ae96dc084d 100644 --- a/crates/api-core/src/tests/common/api_fixtures/mod.rs +++ b/crates/api-core/src/tests/common/api_fixtures/mod.rs @@ -424,6 +424,7 @@ impl TestEnv { ManagedHostState::HostInit { machine_state: mc } } ManagedHostState::Ready => state.clone(), + ManagedHostState::BootConfiguring { .. } => state.clone(), ManagedHostState::Maintenance { .. } => state.clone(), ManagedHostState::Assigned { .. } => state.clone(), ManagedHostState::WaitingForCleanup { .. } => state.clone(), diff --git a/crates/api-core/src/tests/ipxe.rs b/crates/api-core/src/tests/ipxe.rs index d000e6d24e..7d610fc719 100644 --- a/crates/api-core/src/tests/ipxe.rs +++ b/crates/api-core/src/tests/ipxe.rs @@ -22,12 +22,15 @@ use chrono::Utc; use common::api_fixtures::{ TestEnv, TestEnvOverrides, create_test_env, create_test_env_with_overrides, get_config, }; +use config_version::ConfigVersion; use db::{self}; use futures_util::FutureExt; use mac_address::MacAddress; use model::machine::{ CleanupContext, DpuInitState, HostReprovisionState, MachineState, ManagedHostState, + ReadyBootConfigState, SetBootOrderInfo, SetBootOrderState, }; +use model::machine_boot_interface::MachineBootInterfaceTarget; use model::test_support::ManagedHostConfig; use rpc::forge::CloudInitInstructionsRequest; use rpc::forge::forge_server::Forge; @@ -341,6 +344,35 @@ async fn test_pxe_host(pool: sqlx::PgPool) { .await; assert!(instructions.pxe_script.contains("x86_64/scout.efi")); + move_machine_to_needed_state( + host_id, + &ManagedHostState::BootConfiguring { + desired_version: ConfigVersion::new(7), + desired_boot_interface: MachineBootInterfaceTarget::MacOnly( + "02:00:00:00:00:01".parse().unwrap(), + ), + post_lock_verification_retry_count: 0, + boot_config_state: ReadyBootConfigState::SetBootOrder { + set_boot_order_info: SetBootOrderInfo { + set_boot_order_jid: None, + set_boot_order_state: SetBootOrderState::SetBootOrder, + retry_count: 0, + }, + }, + }, + &env.pool, + ) + .await; + + let instructions = get_pxe_instructions( + &env, + host_interface_id, + rpc::forge::MachineArchitecture::X86, + None, + ) + .await; + assert!(instructions.pxe_script.contains("x86_64/scout.efi")); + move_machine_to_needed_state( host_id, &ManagedHostState::HostReprovision { diff --git a/crates/api-core/src/tests/machine_states.rs b/crates/api-core/src/tests/machine_states.rs index ff7df726fe..141fdde1a6 100644 --- a/crates/api-core/src/tests/machine_states.rs +++ b/crates/api-core/src/tests/machine_states.rs @@ -30,7 +30,7 @@ use carbide_site_explorer::config::SiteExplorerConfig; use carbide_utils::arch::CpuArchitecture; use carbide_uuid::machine::MachineId; use carbide_uuid::machine_validation::MachineValidationId; -use chrono::Duration; +use chrono::{Duration, Utc}; use common::api_fixtures::dpu::{ create_dpu_machine, create_dpu_machine_in_waiting_for_network_install, }; @@ -57,12 +57,15 @@ use model::hardware_info::TpmEkCertificate; use model::machine::health_override::HARDWARE_HEALTH_OVERRIDE_PREFIX; use model::machine::machine_search_config::MachineSearchConfig; use model::machine::{ - BiosConfigInfo, BiosConfigState, CleanupContext, CleanupState, DpuDiscoveringState, - DpuInitState, DpuReprovisionStates, FailureCause, FailureDetails, FailureSource, - HostPlatformConfigurationState, InstallDpuOsState, InstanceState, LockdownMode, MachineState, - MachineValidatingState, ManagedHostState, MeasuringState, PowerState, SetBootOrderInfo, - SetBootOrderState, SetSecureBootState, SpdmMeasuringState, StateMachineArea, ValidationState, + BiosConfigInfo, BiosConfigState, BomValidating, BomValidatingContext, CleanupContext, + CleanupState, DpuDiscoveringState, DpuInitState, DpuReprovisionStates, FailureCause, + FailureDetails, FailureSource, HostPlatformConfigurationState, InstallDpuOsState, + InstanceState, LockdownMode, MachineState, MachineValidatingState, MachineValidationContext, + ManagedHostState, MeasuringState, PowerState, ReadyBootConfigState, + ReadyBootConfigTerminalFailure, SetBootOrderInfo, SetBootOrderState, SetSecureBootState, + SpdmMeasuringState, StateMachineArea, ValidationState, }; +use model::machine_boot_interface::MachineBootInterfaceTarget; use model::machine_validation::MachineValidationState; use model::network_segment::NetworkSegmentType; use model::site_explorer::{EndpointExplorationReport, ExploredDpu, ExploredManagedHost}; @@ -2949,6 +2952,855 @@ async fn host_inband_nic_mac(env: &TestEnv, host_id: MachineId) -> MacAddress { .mac_address } +/// Replaces the fixture host's boot target with distinct operator intent. +/// +/// Site Explorer initializes the durable target during ingestion. Using +/// `machine_desired_boot_interface::set` with a different MAC exercises the +/// normal intent path and guarantees a new version without force-reconcile. +async fn set_pending_boot_interface( + env: &TestEnv, + mh: &TestManagedHost, +) -> config_version::Versioned { + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + let current = host + .config + .desired_boot_interface + .as_ref() + .expect("Site Explorer should initialize a durable boot target"); + let first_candidate = MacAddress::new([0x02, 0, 0, 0, 0xfe, 0x01]); + let second_candidate = MacAddress::new([0x02, 0, 0, 0, 0xfe, 0x02]); + let replacement_mac = if current.value.mac_address() == first_candidate { + second_candidate + } else { + first_candidate + }; + let replacement = MachineBootInterfaceTarget::MacOnly(replacement_mac); + + let pending = db::machine_desired_boot_interface::set(txn.as_mut(), &host.id, &replacement) + .await + .expect("setting distinct boot-interface intent should persist a new version"); + assert_ne!( + pending.version, current.version, + "distinct intent must create a pending desired version" + ); + txn.commit().await.unwrap(); + pending +} + +/// Replaces the fixture's DMI vendor so controller vendor branches can be +/// exercised after ordinary ingestion has completed. +async fn set_host_hardware_vendor(env: &TestEnv, mh: &TestManagedHost, vendor: &str) { + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + let mut hardware_info = host + .status + .hardware_info + .expect("fixture host should have hardware information"); + hardware_info + .dmi_data + .as_mut() + .expect("fixture host should have DMI information") + .sys_vendor = vendor.to_string(); + db::machine_topology::set_topology_update_needed(txn.as_mut(), &host.id, true) + .await + .unwrap(); + db::machine_topology::create_or_update(txn.as_mut(), &host.id, &hardware_info) + .await + .unwrap(); + txn.commit().await.unwrap(); +} + +/// A zero-DPU host skips the DPU reachability wait. Disable still polls before +/// platform configuration, while Enable preserves the established direct +/// validation path even when reported lockdown status is stale. +#[crate::sqlx_test] +async fn test_zero_dpu_lockdown_wait_preserves_mode_transition(pool: sqlx::PgPool) { + let env = create_zero_dpu_test_env(pool).await; + let mh = create_managed_host_with_config(&env, ManagedHostConfig::zero_dpu()).await; + let polling = ManagedHostState::HostInit { + machine_state: MachineState::WaitingForLockdown { + lockdown_info: model::machine::LockdownInfo { + state: model::machine::LockdownState::PollingLockdownStatus, + mode: LockdownMode::Disable, + }, + }, + }; + + set_host_controller_state_stuck_in( + &env, + mh.host().id, + &ManagedHostState::HostInit { + machine_state: MachineState::WaitingForLockdown { + lockdown_info: model::machine::LockdownInfo { + state: model::machine::LockdownState::TimeWaitForDPUDown, + mode: LockdownMode::Disable, + }, + }, + }, + 0, + ) + .await; + env.redfish_sim + .set_lockdown(libredfish::EnabledDisabled::Disabled); + + env.run_machine_state_controller_iteration().await; + + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + assert_eq!(host.current_state(), &polling); + drop(txn); + + env.run_machine_state_controller_iteration().await; + + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + assert_eq!( + host.current_state(), + &ManagedHostState::HostInit { + machine_state: MachineState::WaitingForPlatformConfiguration { retry_count: 0 }, + }, + ); + drop(txn); + + set_host_controller_state_stuck_in( + &env, + mh.host().id, + &ManagedHostState::HostInit { + machine_state: MachineState::WaitingForLockdown { + lockdown_info: model::machine::LockdownInfo { + state: model::machine::LockdownState::TimeWaitForDPUDown, + mode: LockdownMode::Enable, + }, + }, + }, + 0, + ) + .await; + env.redfish_sim + .set_lockdown(libredfish::EnabledDisabled::Disabled); + + env.run_machine_state_controller_iteration().await; + + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + assert_eq!( + host.current_state(), + &ManagedHostState::BomValidating { + bom_validating_state: BomValidating::MatchingSku(BomValidatingContext { + machine_validation_context: Some(MachineValidationContext::Discovery), + ..BomValidatingContext::default() + }), + }, + ); +} + +/// Discovery completion must not publish a transient Ready state while a +/// desired boot-interface version still needs controller convergence. +#[crate::sqlx_test] +async fn test_discovered_host_with_pending_boot_config_enters_convergence(pool: sqlx::PgPool) { + let env = create_zero_dpu_test_env(pool).await; + let mut host_config = ManagedHostConfig::zero_dpu(); + host_config.vendor = Some(bmc_vendor::BMCVendor::Supermicro); + let mh = create_managed_host_with_config(&env, host_config).await; + set_host_hardware_vendor(&env, &mh, "Supermicro").await; + let pending = set_pending_boot_interface(&env, &mh).await; + + set_host_controller_state_stuck_in( + &env, + mh.host().id, + &ManagedHostState::HostInit { + machine_state: MachineState::WaitingForLockdown { + lockdown_info: model::machine::LockdownInfo { + state: model::machine::LockdownState::PollingLockdownStatus, + mode: LockdownMode::Enable, + }, + }, + }, + 0, + ) + .await; + env.redfish_sim + .set_lockdown(libredfish::EnabledDisabled::Enabled); + let lockdown_checkpoint = env.redfish_sim.timepoint(); + + env.run_machine_state_controller_iteration().await; + + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + assert_eq!( + host.pending_boot_interface_config_version(), + Some(pending.version), + "a locked Supermicro read must not publish boot verification" + ); + drop(txn); + assert!( + !env.redfish_sim + .actions_since(&lockdown_checkpoint) + .all_hosts() + .iter() + .any(|action| matches!(action, RedfishSimAction::IsBootOrderSetup { .. })), + "HostInit must defer Supermicro verification until the unlocked Ready flow" + ); + + set_host_controller_state_stuck_in( + &env, + mh.host().id, + &ManagedHostState::HostInit { + machine_state: MachineState::Discovered { + skip_reboot_wait: true, + }, + }, + 0, + ) + .await; + + env.run_machine_state_controller_iteration().await; + + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + assert_eq!( + host.current_state(), + &ManagedHostState::BootConfiguring { + desired_version: pending.version, + desired_boot_interface: pending.value, + post_lock_verification_retry_count: 0, + boot_config_state: ReadyBootConfigState::Prepare, + }, + ); +} + +/// Supermicro boot-order reads can be stale under lockdown. Ready must create +/// its exact verification while unlocked, then retain that proof while +/// restoring lockdown instead of trusting a contradictory locked read. +#[crate::sqlx_test] +async fn test_supermicro_ready_boot_config_uses_unlocked_verification(pool: sqlx::PgPool) { + let env = create_zero_dpu_test_env(pool).await; + let mut host_config = ManagedHostConfig::zero_dpu(); + host_config.vendor = Some(bmc_vendor::BMCVendor::Supermicro); + let mh = create_managed_host_with_config(&env, host_config).await; + set_host_hardware_vendor(&env, &mh, "Supermicro").await; + let pending = set_pending_boot_interface(&env, &mh).await; + let expected_mac = pending.value.mac_address().to_string(); + let locking = ManagedHostState::BootConfiguring { + desired_version: pending.version, + desired_boot_interface: pending.value.clone(), + post_lock_verification_retry_count: 0, + boot_config_state: ReadyBootConfigState::LockHost { + terminal_failure: None, + }, + }; + + env.redfish_sim.set_is_bios_setup(true); + env.redfish_sim.set_is_boot_order_setup(true); + env.redfish_sim + .set_lockdown(libredfish::EnabledDisabled::Enabled); + let convergence_checkpoint = env.redfish_sim.timepoint(); + + env.run_machine_state_controller_iteration().await; + env.run_machine_state_controller_iteration_until_state_matches(&mh.id, 10, locking.clone()) + .await; + + let history = mh.host().parsed_history(None).await; + assert!( + history.iter().any(|state| matches!( + state, + ManagedHostState::BootConfiguring { + desired_version, + boot_config_state: ReadyBootConfigState::UnlockHost { .. }, + .. + } if *desired_version == pending.version + )), + "Supermicro convergence must pass through the unlock choreography: {history:#?}" + ); + let convergence_actions = env + .redfish_sim + .actions_since(&convergence_checkpoint) + .all_hosts(); + assert!( + convergence_actions.iter().any(|action| matches!( + action, + RedfishSimAction::Power(libredfish::SystemPowerControl::ForceRestart) + )), + "Supermicro must reboot after lockdown is disabled: {convergence_actions:?}" + ); + assert!( + convergence_actions.iter().any(|action| matches!( + action, + RedfishSimAction::IsBootOrderSetup { boot_interface_mac } + if boot_interface_mac == &expected_mac + )), + "Supermicro must verify the exact target before relocking: {convergence_actions:?}" + ); + + // Model the stale locked view reported by Supermicro after the durable + // unlocked verification has advanced the controller to LockHost. + env.redfish_sim.set_is_boot_order_setup(false); + let lock_checkpoint = env.redfish_sim.timepoint(); + + env.run_machine_state_controller_iteration().await; + + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + assert_eq!(host.current_state(), &ManagedHostState::Ready); + assert_eq!(host.pending_boot_interface_config_version(), None); + assert_eq!( + host.status + .boot_interface_status_observation + .as_ref() + .map(|observation| observation.config_version), + Some(pending.version), + ); + drop(txn); + + let lock_actions = env.redfish_sim.actions_since(&lock_checkpoint).all_hosts(); + assert!( + !lock_actions + .iter() + .any(|action| matches!(action, RedfishSimAction::IsBootOrderSetup { .. })), + "LockHost must not replace the durable unlocked proof with a stale locked read: {lock_actions:?}" + ); + assert!( + env.redfish_sim + .lockdown_states() + .iter() + .all(|state| *state == libredfish::EnabledDisabled::Enabled), + "the target must be verified only after lockdown is restored" + ); +} + +/// A pending desired version on an unassigned Ready host is fully owned by the +/// state controller: it persists each convergence phase, repairs Redfish, +/// restores lockdown, and only then records the exact desired version verified. +#[crate::sqlx_test] +async fn test_ready_converges_pending_desired_boot_interface(pool: sqlx::PgPool) { + let env = create_zero_dpu_test_env(pool).await; + let mh = create_managed_host_with_config(&env, ManagedHostConfig::zero_dpu()).await; + let pending = set_pending_boot_interface(&env, &mh).await; + let expected_mac = pending.value.mac_address().to_string(); + + env.redfish_sim.set_is_bios_setup(true); + env.redfish_sim.set_is_boot_order_setup(false); + env.redfish_sim + .set_lockdown(libredfish::EnabledDisabled::Enabled); + let checkpoint = env.redfish_sim.timepoint(); + let reconciliation_started_at = Utc::now(); + + env.run_machine_state_controller_iteration().await; + + { + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + assert_eq!( + host.current_state(), + &ManagedHostState::BootConfiguring { + desired_version: pending.version, + desired_boot_interface: pending.value.clone(), + post_lock_verification_retry_count: 0, + boot_config_state: ReadyBootConfigState::Prepare, + }, + "Ready must persist the captured target before Redfish work" + ); + assert_ne!( + host.status + .boot_interface_status_observation + .as_ref() + .map(|observation| observation.config_version), + Some(pending.version), + "entering BootConfiguring is not itself a Redfish verification" + ); + } + + env.run_machine_state_controller_iteration_until_state_matches( + &mh.id, + 20, + ManagedHostState::Ready, + ) + .await; + + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + let persisted_desired = host + .config + .desired_boot_interface + .as_ref() + .expect("convergence must retain operator intent"); + assert_eq!(persisted_desired.version, pending.version); + assert_eq!(persisted_desired.value, pending.value); + let observation = host + .status + .boot_interface_status_observation + .as_ref() + .expect("successful convergence should record a Redfish observation"); + assert_eq!( + observation.config_version, pending.version, + "only the exact desired version that was checked may be marked verified" + ); + assert!( + observation.observed_at >= reconciliation_started_at, + "verification must come from this convergence pass" + ); + assert!( + !observation.assumed, + "controller verification must replace any migration-assumed baseline" + ); + assert_eq!( + host.pending_boot_interface_config_version(), + None, + "the verified host should no longer have pending boot intent" + ); + drop(txn); + + let history = mh.host().parsed_history(None).await; + let prepare_index = history + .iter() + .position(|state| { + matches!( + state, + ManagedHostState::BootConfiguring { + desired_version, + desired_boot_interface, + boot_config_state: ReadyBootConfigState::Prepare, + .. + } if *desired_version == pending.version + && desired_boot_interface == &pending.value + ) + }) + .expect("Ready should persist BootConfiguring/Prepare"); + let set_order_index = history + .iter() + .position(|state| { + matches!( + state, + ManagedHostState::BootConfiguring { + desired_version, + desired_boot_interface, + boot_config_state: ReadyBootConfigState::SetBootOrder { .. }, + .. + } if *desired_version == pending.version + && desired_boot_interface == &pending.value + ) + }) + .expect("boot-order drift should persist its remediation substate"); + let lock_index = history + .iter() + .position(|state| { + matches!( + state, + ManagedHostState::BootConfiguring { + desired_version, + desired_boot_interface, + boot_config_state: ReadyBootConfigState::LockHost { + terminal_failure: None, + }, + .. + } if *desired_version == pending.version + && desired_boot_interface == &pending.value + ) + }) + .expect("successful repair should persist LockHost before verification"); + assert!( + prepare_index < set_order_index && set_order_index < lock_index, + "persisted convergence phases should retain their causal order: {history:#?}" + ); + + let actions = env.redfish_sim.actions_since(&checkpoint).all_hosts(); + assert!( + actions.iter().any(|action| matches!( + action, + RedfishSimAction::SetBootOrderDpuFirst { boot_interface_mac } + if boot_interface_mac == &expected_mac + )), + "the controller should repair the desired target's boot order: {actions:?}" + ); + assert!( + actions.iter().any(|action| matches!( + action, + RedfishSimAction::Power(libredfish::SystemPowerControl::ForceRestart) + )), + "boot-order remediation should reboot to apply the change: {actions:?}" + ); + assert!( + actions + .iter() + .filter(|action| matches!( + action, + RedfishSimAction::IsBootOrderSetup { boot_interface_mac } + if boot_interface_mac == &expected_mac + )) + .count() + >= 2, + "the controller should observe the desired target before repair and again at final verification: {actions:?}" + ); + assert!( + env.redfish_sim + .lockdown_states() + .iter() + .all(|state| *state == libredfish::EnabledDisabled::Enabled), + "the BMC must be locked before the desired version is published as verified" + ); +} + +/// A correct ordinary host proves its target before opening lockdown and must +/// not issue any boot mutation or reboot. +#[crate::sqlx_test] +async fn test_ready_boot_config_skips_unlock_when_already_correct(pool: sqlx::PgPool) { + let env = create_zero_dpu_test_env(pool).await; + let mh = create_managed_host_with_config(&env, ManagedHostConfig::zero_dpu()).await; + let pending = set_pending_boot_interface(&env, &mh).await; + let expected_mac = pending.value.mac_address().to_string(); + + env.redfish_sim.set_is_bios_setup(true); + env.redfish_sim.set_is_boot_order_setup(true); + env.redfish_sim + .set_lockdown(libredfish::EnabledDisabled::Enabled); + let checkpoint = env.redfish_sim.timepoint(); + + env.run_machine_state_controller_iteration().await; + env.run_machine_state_controller_iteration_until_state_matches( + &mh.id, + 5, + ManagedHostState::Ready, + ) + .await; + + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + let observation = host + .status + .boot_interface_status_observation + .as_ref() + .expect("the already-correct target should be verified"); + assert_eq!(observation.config_version, pending.version); + assert!(!observation.assumed); + drop(txn); + + let history = mh.host().parsed_history(None).await; + assert!(history.iter().any(|state| matches!( + state, + ManagedHostState::BootConfiguring { + desired_version, + boot_config_state: ReadyBootConfigState::LockHost { + terminal_failure: None, + }, + .. + } if *desired_version == pending.version + ))); + assert!( + !history.iter().any(|state| matches!( + state, + ManagedHostState::BootConfiguring { + desired_version, + boot_config_state: + ReadyBootConfigState::UnlockHost { .. } + | ReadyBootConfigState::ConfigureBios { .. } + | ReadyBootConfigState::SetBootOrder { .. }, + .. + } if *desired_version == pending.version + )), + "an already-correct ordinary host should not be opened or mutated: {history:#?}" + ); + + let actions = env.redfish_sim.actions_since(&checkpoint).all_hosts(); + assert!( + actions + .iter() + .filter(|action| matches!( + action, + RedfishSimAction::IsBootOrderSetup { boot_interface_mac } + if boot_interface_mac == &expected_mac + )) + .count() + >= 2, + "the target should be observed before cleanup and at final verification: {actions:?}" + ); + assert!( + !actions.iter().any(|action| matches!( + action, + RedfishSimAction::MachineSetup { .. } + | RedfishSimAction::SetBootOrderDpuFirst { .. } + | RedfishSimAction::Power(_) + )), + "the correct fast path must remain observation-only: {actions:?}" + ); +} + +/// Hosts whose lifecycle profile intentionally leaves lockdown disabled still +/// take the already-correct fast path. LockHost has no policy restoration to +/// perform for them, but it still re-observes the exact target before marking +/// the desired version verified. +#[crate::sqlx_test] +async fn test_ready_boot_config_disable_lockdown_skips_unneeded_cleanup(pool: sqlx::PgPool) { + let env = create_zero_dpu_test_env(pool).await; + let mut expected_machine_data = ExpectedMachineData::default(); + expected_machine_data + .host_lifecycle_profile + .disable_lockdown = Some(true); + let mh = create_managed_host_with_config( + &env, + ManagedHostConfig::zero_dpu().with_expected_machine_data(expected_machine_data), + ) + .await; + let pending = set_pending_boot_interface(&env, &mh).await; + + env.redfish_sim.set_is_bios_setup(true); + env.redfish_sim.set_is_boot_order_setup(true); + env.redfish_sim + .set_lockdown(libredfish::EnabledDisabled::Enabled); + + env.run_machine_state_controller_iteration().await; + env.run_machine_state_controller_iteration().await; + let client_calls_after_preflight = env.redfish_sim.create_client_calls().len(); + + env.run_machine_state_controller_iteration_until_state_matches( + &mh.id, + 2, + ManagedHostState::Ready, + ) + .await; + + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + assert_eq!( + host.status + .boot_interface_status_observation + .as_ref() + .map(|observation| observation.config_version), + Some(pending.version), + ); + drop(txn); + + assert_eq!( + env.redfish_sim.create_client_calls().len(), + client_calls_after_preflight + 1, + "LockHost should reconnect only for the final exact-target observation" + ); + assert!( + env.redfish_sim + .lockdown_states() + .iter() + .all(|state| *state == libredfish::EnabledDisabled::Enabled), + "an already-correct host should not have lockdown changed" + ); + let history = mh.host().parsed_history(None).await; + assert!( + !history.iter().any(|state| matches!( + state, + ManagedHostState::BootConfiguring { + boot_config_state: ReadyBootConfigState::UnlockHost { .. }, + .. + } + )), + "an already-correct host must not enter the unlock choreography: {history:#?}" + ); +} + +/// A machine-level failure is more fundamental than boot convergence. Once it +/// has been durably captured on a host whose profile has no lockdown cleanup +/// barrier, it must remain reportable even without Redfish access. +#[crate::sqlx_test] +async fn test_ready_boot_config_machine_failure_does_not_wait_for_redfish(pool: sqlx::PgPool) { + let env = create_zero_dpu_test_env(pool).await; + let mut expected_machine_data = ExpectedMachineData::default(); + expected_machine_data + .host_lifecycle_profile + .disable_lockdown = Some(true); + let mh = create_managed_host_with_config( + &env, + ManagedHostConfig::zero_dpu().with_expected_machine_data(expected_machine_data), + ) + .await; + let pending = set_pending_boot_interface(&env, &mh).await; + let details = FailureDetails { + cause: FailureCause::Discovery { + err: "host BMC became unavailable".to_string(), + }, + failed_at: Utc::now(), + source: FailureSource::StateMachine, + }; + let locking = ManagedHostState::BootConfiguring { + desired_version: pending.version, + desired_boot_interface: pending.value, + post_lock_verification_retry_count: 0, + boot_config_state: ReadyBootConfigState::LockHost { + terminal_failure: Some(ReadyBootConfigTerminalFailure::Machine { + machine_id: mh.host().id, + details: details.clone(), + }), + }, + }; + + set_host_controller_state_stuck_in(&env, mh.host().id, &locking, 0).await; + let client_calls_before_failure = env.redfish_sim.create_client_calls().len(); + + env.run_machine_state_controller_iteration().await; + + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + assert_eq!( + host.current_state(), + &ManagedHostState::Failed { + details, + machine_id: mh.host().id, + retry_count: 0, + } + ); + assert_eq!( + env.redfish_sim.create_client_calls().len(), + client_calls_before_failure, + "machine failure must transition without opening a Redfish client" + ); +} + +/// `LockHost` is a durable restart boundary. The resumed invocation must +/// restore lockdown before directly re-reading the captured target. +#[crate::sqlx_test] +async fn test_ready_boot_config_lock_host_is_restart_safe(pool: sqlx::PgPool) { + let env = create_zero_dpu_test_env(pool).await; + let mh = create_managed_host_with_config(&env, ManagedHostConfig::zero_dpu()).await; + let pending = set_pending_boot_interface(&env, &mh).await; + let expected_mac = pending.value.mac_address().to_string(); + let locking = ManagedHostState::BootConfiguring { + desired_version: pending.version, + desired_boot_interface: pending.value.clone(), + post_lock_verification_retry_count: 0, + boot_config_state: ReadyBootConfigState::LockHost { + terminal_failure: None, + }, + }; + + set_host_controller_state_stuck_in(&env, mh.host().id, &locking, 0).await; + env.redfish_sim.set_is_bios_setup(true); + env.redfish_sim.set_is_boot_order_setup(true); + env.redfish_sim + .set_lockdown(libredfish::EnabledDisabled::Disabled); + let checkpoint = env.redfish_sim.timepoint(); + + env.run_machine_state_controller_iteration().await; + + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + assert_eq!(host.current_state(), &ManagedHostState::Ready); + let observation = host + .status + .boot_interface_status_observation + .as_ref() + .expect("resumed LockHost should publish a direct verification"); + assert_eq!(observation.config_version, pending.version); + assert!(!observation.assumed); + drop(txn); + + let actions = env.redfish_sim.actions_since(&checkpoint).all_hosts(); + assert!( + actions.iter().any(|action| matches!( + action, + RedfishSimAction::IsBootOrderSetup { boot_interface_mac } + if boot_interface_mac == &expected_mac + )), + "resumed LockHost must directly observe its captured target after lockdown: {actions:?}" + ); + assert!( + env.redfish_sim + .lockdown_states() + .iter() + .all(|state| *state == libredfish::EnabledDisabled::Enabled), + "resumed LockHost must restore lockdown before verification" + ); +} + +/// LockHost is a fail-closed cleanup boundary. Losing the BMC interface mapping +/// must keep the host parked there instead of propagating an error that could +/// let later failure handling bypass lockdown restoration. +#[crate::sqlx_test] +async fn test_ready_boot_config_lock_host_waits_for_redfish_access(pool: sqlx::PgPool) { + let env = create_zero_dpu_test_env(pool).await; + let mh = create_managed_host_with_config(&env, ManagedHostConfig::zero_dpu()).await; + let pending = set_pending_boot_interface(&env, &mh).await; + let locking = ManagedHostState::BootConfiguring { + desired_version: pending.version, + desired_boot_interface: pending.value, + post_lock_verification_retry_count: 0, + boot_config_state: ReadyBootConfigState::LockHost { + terminal_failure: None, + }, + }; + + set_host_controller_state_stuck_in(&env, mh.host().id, &locking, 0).await; + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + let bmc_ip = host + .bmc_addr() + .expect("fixture host has a BMC address") + .ip(); + sqlx::query("DELETE FROM machine_interface_addresses WHERE address = $1::inet") + .bind(bmc_ip) + .execute(txn.as_mut()) + .await + .unwrap(); + txn.commit().await.unwrap(); + + env.run_machine_state_controller_iteration().await; + + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + assert_eq!( + host.current_state(), + &locking, + "cleanup must wait in LockHost until Redfish access is restored" + ); + assert_eq!( + host.pending_boot_interface_config_version(), + Some(pending.version), + "a connectivity failure must not publish verification" + ); +} + +/// A successful Redfish write is not sufficient evidence that lockdown was +/// actually restored. Keep the desired version pending in LockHost until a +/// direct status read confirms the policy. +#[crate::sqlx_test] +async fn test_ready_boot_config_waits_for_observed_lockdown_before_verifying(pool: sqlx::PgPool) { + let env = create_zero_dpu_test_env(pool).await; + let mh = create_managed_host_with_config(&env, ManagedHostConfig::zero_dpu()).await; + let pending = set_pending_boot_interface(&env, &mh).await; + let locking = ManagedHostState::BootConfiguring { + desired_version: pending.version, + desired_boot_interface: pending.value.clone(), + post_lock_verification_retry_count: 0, + boot_config_state: ReadyBootConfigState::LockHost { + terminal_failure: None, + }, + }; + + set_host_controller_state_stuck_in(&env, mh.host().id, &locking, 0).await; + env.redfish_sim.set_is_bios_setup(true); + env.redfish_sim.set_is_boot_order_setup(true); + env.redfish_sim + .set_lockdown(libredfish::EnabledDisabled::Disabled); + env.redfish_sim.set_lockdown_bmc_applies(false); + let checkpoint = env.redfish_sim.timepoint(); + + env.run_machine_state_controller_iteration().await; + + let mut txn = env.db_txn().await; + let host = mh.host().db_machine(&mut txn).await; + assert_eq!(host.current_state(), &locking); + assert_eq!( + host.pending_boot_interface_config_version(), + Some(pending.version), + "an unobserved lockdown write must not publish verification" + ); + drop(txn); + + let actions = env.redfish_sim.actions_since(&checkpoint).all_hosts(); + assert!( + !actions + .iter() + .any(|action| matches!(action, RedfishSimAction::IsBootOrderSetup { .. })), + "final boot inspection must wait until lockdown is observed: {actions:?}" + ); +} + /// Asserts that, within the recorded boot-order actions, `machine_setup` ran /// before the `set_boot_order_dpu_first` reorder, and that BOTH targeted /// `expected_mac` -- the resolved boot NIC. This is the ordering the recovery diff --git a/crates/api-core/tests/integration/compute_allocation.rs b/crates/api-core/tests/integration/compute_allocation.rs index 92165f48e0..a6153599f1 100644 --- a/crates/api-core/tests/integration/compute_allocation.rs +++ b/crates/api-core/tests/integration/compute_allocation.rs @@ -28,7 +28,6 @@ use carbide_uuid::instance_type::InstanceTypeId; use carbide_uuid::machine::MachineId; use carbide_uuid::network::NetworkSegmentId; use model::instance_type::InstanceTypeMachineCapabilityFilter; -use model::machine::ManagedHostState; use model::machine::capabilities::MachineCapabilityType; use model::metadata::Metadata as DbMetadata; use model::test_support::ManagedHostConfig; @@ -181,7 +180,7 @@ async fn create_managed_host(env: &TestEnv) -> TestManagedHost { .await .0; mh.host.discover_primary_iface(env.admin_segment).await; - mh.advance_state(ManagedHostState::Ready).await; + mh.advance_to_converged_ready().await; TestManagedHost { id: mh.host.id } } diff --git a/crates/api-core/tests/integration/forge_agent_control.rs b/crates/api-core/tests/integration/forge_agent_control.rs index eea6bdc0c6..f1bac09798 100644 --- a/crates/api-core/tests/integration/forge_agent_control.rs +++ b/crates/api-core/tests/integration/forge_agent_control.rs @@ -383,7 +383,7 @@ async fn waiting_for_measurements_returns_measure(pool: PgPool) { #[sqlx_test] async fn ready_returns_noop(pool: PgPool) { let TestContext { env: _env, mh, .. } = init(pool).await; - mh.advance_state(ManagedHostState::Ready).await; + mh.advance_to_converged_ready().await; let response = mh.host.forge_agent_control().await; assert!(matches!(response.action, Some(Action::Noop(_)))); diff --git a/crates/api-db/migrations/20260722120000_preserve_machine_ipv6_loopback.sql b/crates/api-db/migrations/20260722120001_preserve_machine_ipv6_loopback.sql similarity index 100% rename from crates/api-db/migrations/20260722120000_preserve_machine_ipv6_loopback.sql rename to crates/api-db/migrations/20260722120001_preserve_machine_ipv6_loopback.sql diff --git a/crates/api-db/migrations/20260730120000_machine_boot_interface_status.sql b/crates/api-db/migrations/20260730120000_machine_boot_interface_status.sql new file mode 100644 index 0000000000..c921d15d3b --- /dev/null +++ b/crates/api-db/migrations/20260730120000_machine_boot_interface_status.sql @@ -0,0 +1,34 @@ +-- Track the desired boot-interface generation that machine-controller +-- currently treats as converged. The status is intentionally retained +-- when the desired target changes: comparing its version with desired_version +-- makes the new work pending without discarding the last convergence status. +ALTER TABLE machine_boot_interfaces + ADD COLUMN verified_version varchar(64), + ADD COLUMN observed_at timestamp with time zone, + ADD COLUMN assumed boolean NOT NULL DEFAULT false, + -- Existing rows satisfy this constraint through the unset-status branch. + -- Add it normally so this single migration remains atomic without implying + -- that validation releases the transaction's table lock. + ADD CONSTRAINT machine_boot_interfaces_status_consistent + CHECK ( + ( + verified_version IS NULL + AND observed_at IS NULL + AND NOT assumed + ) + OR ( + verified_version IS NOT NULL + AND observed_at IS NOT NULL + ) + ); + +-- Avoid scheduling a fleet-wide boot reconfiguration when this status first +-- rolls out. These rows predate status tracking, so record the baseline +-- explicitly as assumed rather than presenting it as a Redfish observation. +UPDATE machine_boot_interfaces AS boot_interface +SET verified_version = desired_version, + observed_at = CURRENT_TIMESTAMP, + assumed = true +FROM machines AS machine +WHERE machine.id = boot_interface.machine_id + AND machine.controller_state->>'state' IN ('ready', 'assigned'); diff --git a/crates/api-db/src/machine.rs b/crates/api-db/src/machine.rs index ec3cd1aeb1..a3f8585b60 100644 --- a/crates/api-db/src/machine.rs +++ b/crates/api-db/src/machine.rs @@ -48,10 +48,10 @@ use model::machine::nvlink::MachineNvLinkStatusObservation; use model::machine::spx::MachineSpxStatusObservation; use model::machine::upgrade_policy::AgentUpgradePolicy; use model::machine::{ - Dpf, DpuInfo, DpuInfoStatusObservation, DpuOsOperationalState, DpuRepresentorStatus, - FailureDetails, HostProfile, Machine, MachineInterfaceSnapshot, MachineLastRebootRequested, - MachineLastRebootRequestedMode, MachineMaintenanceOperation, MachineValidationContext, - ManagedHostState, ReprovisionRequest, UpgradeDecision, + CURRENT_STATE_MODEL_VERSION, Dpf, DpuInfo, DpuInfoStatusObservation, DpuOsOperationalState, + DpuRepresentorStatus, FailureDetails, HostProfile, Machine, MachineInterfaceSnapshot, + MachineLastRebootRequested, MachineLastRebootRequestedMode, MachineMaintenanceOperation, + MachineValidationContext, ManagedHostState, ReprovisionRequest, UpgradeDecision, }; use model::machine_interface_address::MachineInterfaceAssociation; use model::metadata::Metadata; @@ -134,7 +134,15 @@ pub async fn get_or_create( // Host and DPU machines are created in same `discover_machine` call. Update same // state in both machines. let state = ManagedHostState::Created; - let machine = create(txn, common_pools, stable_machine_id, state, None, 2).await?; + let machine = create( + txn, + common_pools, + stable_machine_id, + state, + None, + CURRENT_STATE_MODEL_VERSION, + ) + .await?; crate::machine_interface::associate_interface_with_machine( &interface.id, MachineInterfaceAssociation::Machine(machine.id), diff --git a/crates/api-db/src/machine_desired_boot_interface.rs b/crates/api-db/src/machine_desired_boot_interface.rs index 4a9e0a084e..4cd1b96143 100644 --- a/crates/api-db/src/machine_desired_boot_interface.rs +++ b/crates/api-db/src/machine_desired_boot_interface.rs @@ -16,6 +16,7 @@ */ use carbide_uuid::machine::{MachineId, MachineType}; +use chrono::{DateTime, Utc}; use config_version::{ConfigVersion, Versioned}; use mac_address::MacAddress; use model::machine_boot_interface::{ @@ -32,6 +33,7 @@ struct DesiredBootInterfaceRow { desired_mac_address: Option, desired_interface_id: Option, desired_version: Option, + rollout_baseline_eligible: bool, } impl DesiredBootInterfaceRow { @@ -104,7 +106,11 @@ async fn load( machine.version AS machine_version, boot_interface.desired_mac_address, boot_interface.desired_interface_id, - boot_interface.desired_version + boot_interface.desired_version, + COALESCE( + machine.controller_state->>'state' IN ('ready', 'assigned'), + false + ) AS rollout_baseline_eligible FROM machines machine LEFT JOIN machine_boot_interfaces boot_interface ON boot_interface.machine_id = machine.id @@ -131,7 +137,11 @@ async fn load_for_update( machine.version AS machine_version, boot_interface.desired_mac_address, boot_interface.desired_interface_id, - boot_interface.desired_version + boot_interface.desired_version, + COALESCE( + machine.controller_state->>'state' IN ('ready', 'assigned'), + false + ) AS rollout_baseline_eligible FROM machines machine LEFT JOIN machine_boot_interfaces boot_interface ON boot_interface.machine_id = machine.id @@ -222,13 +232,26 @@ fn next_version(expected_version: Option) -> ConfigVersion { .unwrap_or_else(ConfigVersion::initial) } +#[derive(Clone, Copy)] +enum VerificationPolicy { + Pending, + AssumeVerified, + CarryCurrentForward, +} + async fn update( txn: &mut PgConnection, machine_id: &MachineId, current_machine_version: ConfigVersion, expected_version: Option, target: &MachineBootInterfaceTarget, + verification_policy: VerificationPolicy, ) -> DatabaseResult> { + let (assume_verified, carry_current_forward) = match verification_policy { + VerificationPolicy::Pending => (false, false), + VerificationPolicy::AssumeVerified => (true, false), + VerificationPolicy::CarryCurrentForward => (false, true), + }; let desired_version = next_version(expected_version); let machine_version = current_machine_version.increment(); let updated: Option = if let Some(expected_version) = expected_version { @@ -236,7 +259,11 @@ async fn update( UPDATE machine_boot_interfaces SET desired_mac_address = $1, desired_interface_id = $2, - desired_version = $3 + desired_version = $3, + verified_version = CASE + WHEN $6 AND verified_version = $5 THEN $3 + ELSE verified_version + END WHERE machine_id = $4 AND desired_version = $5 RETURNING machine_id @@ -247,6 +274,7 @@ async fn update( .bind(desired_version) .bind(machine_id) .bind(expected_version) + .bind(carry_current_forward) .fetch_optional(&mut *txn) .await .map_err(|error| DatabaseError::query(query, error))? @@ -256,9 +284,20 @@ async fn update( machine_id, desired_mac_address, desired_interface_id, - desired_version + desired_version, + verified_version, + observed_at, + assumed + ) + VALUES ( + $1, + $2, + $3, + $4, + CASE WHEN $5 THEN $4 END, + CASE WHEN $5 THEN CURRENT_TIMESTAMP END, + $5 ) - VALUES ($1, $2, $3, $4) ON CONFLICT (machine_id) DO NOTHING RETURNING machine_id "#; @@ -267,6 +306,7 @@ async fn update( .bind(target.mac_address()) .bind(target.interface_id()) .bind(desired_version) + .bind(assume_verified) .fetch_optional(&mut *txn) .await .map_err(|error| DatabaseError::query(query, error))? @@ -337,6 +377,7 @@ pub async fn try_set( current_machine_version, expected_version, target, + VerificationPolicy::Pending, ) .await? .is_some()) @@ -371,6 +412,7 @@ pub async fn set( current_machine_version, expected_version, target, + VerificationPolicy::Pending, ) .await? else { @@ -389,6 +431,12 @@ pub async fn set( /// `initialize_if_unset` stores Site Explorer's initial target without /// replacing a target that is already present. +/// +/// A stable host already in `Ready` or `Assigned` receives an explicitly +/// assumed compatibility observation. This covers a missing row during a +/// mixed-component rollout or later repair without scheduling fleet-wide +/// remediation. Normal lifecycle initialization happens on a predicted host or +/// in `HostInit`, so newly provisioned hosts remain pending real verification. pub async fn initialize_if_unset( txn: &mut PgConnection, machine_id: &MachineId, @@ -399,11 +447,25 @@ pub async fn initialize_if_unset( let row = load_for_update(txn, machine_id).await?; let current_machine_version = row.machine_version; + let assume_verified = machine_id.machine_type().is_host() && row.rollout_baseline_eligible; + let verification_policy = if assume_verified { + VerificationPolicy::AssumeVerified + } else { + VerificationPolicy::Pending + }; if let Some(current) = row.decode(machine_id)? { return Ok(current); } - let Some(version) = update(txn, machine_id, current_machine_version, None, target).await? + let Some(version) = update( + txn, + machine_id, + current_machine_version, + None, + target, + verification_policy, + ) + .await? else { return Err(DatabaseError::Internal { message: format!( @@ -421,6 +483,8 @@ pub async fn initialize_if_unset( /// `enrich_interface_id` adds a Redfish id to a matching MAC-only target. /// /// Once a pair is stored, later observations do not replace its id. +/// Enrichment advances a current status because it strengthens the same +/// physical target's identity; stale or pending status remains unchanged. pub async fn enrich_interface_id( txn: &mut PgConnection, machine_id: &MachineId, @@ -458,6 +522,7 @@ pub async fn enrich_interface_id( current_machine_version, expected_version, &target, + VerificationPolicy::CarryCurrentForward, ) .await? else { @@ -474,15 +539,60 @@ pub async fn enrich_interface_id( })) } +/// Records a Redfish observation only if the desired boot-interface version +/// still matches the version the caller observed. +/// +/// A `false` return means the desired target was removed or replaced before +/// the observation could be committed. The caller must not treat that newer +/// target as verified. +pub async fn mark_verified( + txn: &mut PgConnection, + machine_id: &MachineId, + expected_desired_version: ConfigVersion, + observed_at: DateTime, +) -> Result { + validate_machine_id(machine_id)?; + + // Desired-target writers lock the parent machine row before touching this + // child row. Preserve that order so a concurrent operator write cannot + // deadlock verification against the state-controller transition, which + // also updates the parent before commit. + load_for_update(txn, machine_id).await?; + + let query = r#" + UPDATE machine_boot_interfaces + SET verified_version = desired_version, + observed_at = $1, + assumed = false + WHERE machine_id = $2 + AND desired_version = $3 + RETURNING machine_id + "#; + let updated: Option = sqlx::query_scalar(query) + .bind(observed_at) + .bind(machine_id) + .bind(expected_desired_version) + .fetch_optional(txn) + .await + .map_err(|error| DatabaseError::query(query, error))?; + + Ok(updated.is_some()) +} + #[cfg(test)] mod tests { use carbide_uuid::machine::{MachineIdSource, MachineType}; + use model::machine::machine_search_config::MachineSearchConfig; + use model::machine::{InstanceState, MachineState, ManagedHostState}; use sqlx::PgPool; + use sqlx::types::Json; use super::*; const MIGRATION: &str = include_str!("../migrations/20260728120000_machine_boot_interfaces.sql"); + const STATUS_MIGRATION: &str = + include_str!("../migrations/20260730120000_machine_boot_interface_status.sql"); fn machine_id(machine_type: MachineType, marker: u8) -> MachineId { let mut hardware_id = [0u8; 32]; @@ -500,7 +610,10 @@ mod tests { ) -> Result { let query = r#" INSERT INTO machines (id, dpf) - VALUES ($1, '{}'::jsonb) + VALUES ( + $1, + '{"enabled": false, "used_for_ingestion": false}'::jsonb + ) RETURNING version "#; sqlx::query_scalar(query) @@ -525,6 +638,33 @@ mod tests { .await } + async fn set_controller_state( + txn: &mut PgConnection, + machine_id: &MachineId, + state: ManagedHostState, + ) -> Result<(), sqlx::Error> { + sqlx::query("UPDATE machines SET controller_state = $1 WHERE id = $2") + .bind(Json(state)) + .bind(machine_id) + .execute(txn) + .await?; + Ok(()) + } + + async fn status_observation( + txn: &mut PgConnection, + machine_id: &MachineId, + ) -> Result<(Option, Option>, bool), sqlx::Error> { + sqlx::query_as( + "SELECT verified_version, observed_at, assumed + FROM machine_boot_interfaces + WHERE machine_id = $1", + ) + .bind(machine_id) + .fetch_one(txn) + .await + } + fn assert_target( actual: &Versioned, expected: &MachineBootInterfaceTarget, @@ -562,6 +702,234 @@ mod tests { Ok(()) } + #[crate::sqlx_test] + async fn rollout_baseline_only_applies_to_existing_stable_ready_or_assigned_hosts( + pool: PgPool, + ) -> Result<(), Box> { + let mut txn = pool.begin().await?; + let cases = [ + ( + machine_id(MachineType::Host, 30), + ManagedHostState::Ready, + true, + ), + ( + machine_id(MachineType::Host, 31), + ManagedHostState::Assigned { + instance_state: InstanceState::Init, + }, + true, + ), + ( + machine_id(MachineType::Host, 32), + ManagedHostState::HostInit { + machine_state: MachineState::WaitingForPlatformConfiguration { retry_count: 0 }, + }, + false, + ), + ( + machine_id(MachineType::PredictedHost, 33), + ManagedHostState::Ready, + false, + ), + ]; + + for (index, (machine_id, state, expect_assumed)) in cases.into_iter().enumerate() { + seed_machine(txn.as_mut(), &machine_id).await?; + set_controller_state(txn.as_mut(), &machine_id, state).await?; + let target = + MachineBootInterfaceTarget::MacOnly(MacAddress::new([2, 0, 0, 0, 3, index as u8])); + let initialized = initialize_if_unset(txn.as_mut(), &machine_id, &target).await?; + let (verified_version, observed_at, assumed) = + status_observation(txn.as_mut(), &machine_id).await?; + + if expect_assumed { + assert_eq!(verified_version, Some(initialized.version)); + assert!(observed_at.is_some()); + assert!(assumed); + } else { + assert_eq!(verified_version, None); + assert_eq!(observed_at, None); + assert!(!assumed); + } + } + + Ok(()) + } + + #[crate::sqlx_test] + async fn verification_is_a_cas_and_is_exposed_in_machine_snapshots( + pool: PgPool, + ) -> Result<(), Box> { + let mut txn = pool.begin().await?; + let machine_id = machine_id(MachineType::Host, 34); + seed_machine(txn.as_mut(), &machine_id).await?; + let target = MachineBootInterfaceTarget::MacOnly(MacAddress::new([2, 0, 0, 0, 3, 4])); + let initialized = initialize_if_unset(txn.as_mut(), &machine_id, &target).await?; + let observed_at = + DateTime::from_timestamp(1_722_000_000, 123_000_000).expect("fixture timestamp"); + + assert!( + !mark_verified( + txn.as_mut(), + &machine_id, + ConfigVersion::invalid(), + observed_at, + ) + .await? + ); + assert_eq!( + status_observation(txn.as_mut(), &machine_id).await?, + (None, None, false) + ); + + assert!(mark_verified(txn.as_mut(), &machine_id, initialized.version, observed_at,).await?); + assert_eq!( + status_observation(txn.as_mut(), &machine_id).await?, + (Some(initialized.version), Some(observed_at), false) + ); + + let machine = + crate::machine::find_one(txn.as_mut(), &machine_id, MachineSearchConfig::default()) + .await? + .expect("machine snapshot"); + let observation = machine + .status + .boot_interface_status_observation + .expect("boot interface status observation"); + assert_eq!(observation.config_version, initialized.version); + assert_eq!(observation.observed_at, observed_at); + assert!(!observation.assumed); + + let replacement = MachineBootInterfaceTarget::MacOnly(MacAddress::new([2, 0, 0, 0, 3, 5])); + let updated = set(txn.as_mut(), &machine_id, &replacement).await?; + assert_ne!(updated.version, initialized.version); + assert!(!mark_verified(txn.as_mut(), &machine_id, initialized.version, Utc::now(),).await?); + assert_eq!( + status_observation(txn.as_mut(), &machine_id).await?, + (Some(initialized.version), Some(observed_at), false), + "changing the target keeps the last factual observation but makes its version stale", + ); + + Ok(()) + } + + #[crate::sqlx_test] + #[allow(txn_held_across_await)] // Intentionally hold a row lock while exercising concurrency. + async fn concurrent_set_and_mark_verified_use_parent_first_lock_order( + pool: PgPool, + ) -> Result<(), Box> { + let machine_id = machine_id(MachineType::Host, 35); + let initial_target = + MachineBootInterfaceTarget::MacOnly(MacAddress::new([2, 0, 0, 0, 3, 6])); + let replacement = MachineBootInterfaceTarget::MacOnly(MacAddress::new([2, 0, 0, 0, 3, 7])); + + let mut setup_txn = pool.begin().await?; + seed_machine(setup_txn.as_mut(), &machine_id).await?; + let initial = initialize_if_unset(setup_txn.as_mut(), &machine_id, &initial_target).await?; + setup_txn.commit().await?; + + // Hold the parent lock as `set` does. `mark_verified` must wait here, + // before it can lock the child intent row. + let mut setter_txn = pool.begin().await?; + let setter_pid: i32 = sqlx::query_scalar("SELECT pg_backend_pid()") + .fetch_one(setter_txn.as_mut()) + .await?; + load_for_update(setter_txn.as_mut(), &machine_id).await?; + + let (verification_pid_tx, verification_pid_rx) = tokio::sync::oneshot::channel(); + let verification_pool = pool.clone(); + let mut verification_task = tokio::spawn(async move { + let mut txn = verification_pool + .begin() + .await + .map_err(|error| error.to_string())?; + let verification_pid: i32 = sqlx::query_scalar("SELECT pg_backend_pid()") + .fetch_one(txn.as_mut()) + .await + .map_err(|error| error.to_string())?; + verification_pid_tx + .send(verification_pid) + .map_err(|_| "could not signal verification backend pid".to_string())?; + + let marked = mark_verified(txn.as_mut(), &machine_id, initial.version, Utc::now()) + .await + .map_err(|error| error.to_string())?; + + // The real state-controller transaction updates the parent state + // after recording verification. This remains safe because + // mark_verified already owns the parent lock. + sqlx::query("UPDATE machines SET updated = updated WHERE id = $1") + .bind(machine_id) + .execute(txn.as_mut()) + .await + .map_err(|error| error.to_string())?; + txn.commit().await.map_err(|error| error.to_string())?; + + Ok::(marked) + }); + + let verification_pid = verification_pid_rx + .await + .map_err(|error| std::io::Error::other(error.to_string()))?; + let wait_for_parent_lock = tokio::time::timeout(std::time::Duration::from_secs(5), async { + loop { + let blocked_by_setter: bool = + sqlx::query_scalar("SELECT $1 = ANY(pg_blocking_pids($2))") + .bind(setter_pid) + .bind(verification_pid) + .fetch_one(&pool) + .await?; + if blocked_by_setter { + return Ok::<(), sqlx::Error>(()); + } + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + }) + .await; + match wait_for_parent_lock { + Ok(result) => result?, + Err(_) => { + verification_task.abort(); + let _ = verification_task.await; + return Err(std::io::Error::other( + "verification did not wait on the setter's parent lock", + ) + .into()); + } + } + + let updated = set(setter_txn.as_mut(), &machine_id, &replacement).await?; + setter_txn.commit().await?; + + let marked = + match tokio::time::timeout(std::time::Duration::from_secs(5), &mut verification_task) + .await + { + Ok(result) => result + .map_err(|error| std::io::Error::other(error.to_string()))? + .map_err(std::io::Error::other)?, + Err(_) => { + verification_task.abort(); + let _ = verification_task.await; + return Err(std::io::Error::other("set and verification deadlocked").into()); + } + }; + + assert!(!marked, "a superseded desired version must not be verified"); + let desired = get(&pool, &machine_id).await?.expect("replacement target"); + assert_target(&desired, &replacement); + assert_eq!(desired.version, updated.version); + let mut conn = pool.acquire().await?; + assert_eq!( + status_observation(conn.as_mut(), &machine_id).await?, + (None, None, false), + "stale verification must not stamp the replacement version", + ); + + Ok(()) + } + #[crate::sqlx_test] async fn try_set_uses_cas_without_bumping_noops( pool: PgPool, @@ -693,6 +1061,10 @@ mod tests { let mac_address = MacAddress::new([2, 0, 0, 0, 0, 5]); let target = MachineBootInterfaceTarget::MacOnly(mac_address); let initialized = initialize_if_unset(txn.as_mut(), &machine_id, &target).await?; + assert_eq!( + status_observation(txn.as_mut(), &machine_id).await?, + (None, None, false), + ); let mismatch = enrich_interface_id( txn.as_mut(), @@ -722,6 +1094,11 @@ mod tests { enriched.version.version_nr(), initialized.version.version_nr() + 1 ); + assert_eq!( + status_observation(txn.as_mut(), &machine_id).await?, + (None, None, false), + "enrichment must not make an unverified target look converged", + ); let unchanged = enrich_interface_id(txn.as_mut(), &machine_id, mac_address, "replacement") .await? @@ -737,6 +1114,51 @@ mod tests { Ok(()) } + #[crate::sqlx_test] + async fn enrichment_carries_current_verification_without_hiding_target_changes( + pool: PgPool, + ) -> Result<(), Box> { + let mut txn = pool.begin().await?; + let machine_id = machine_id(MachineType::Host, 36); + seed_machine(txn.as_mut(), &machine_id).await?; + let mac_address = MacAddress::new([2, 0, 0, 0, 3, 8]); + let initialized = initialize_if_unset( + txn.as_mut(), + &machine_id, + &MachineBootInterfaceTarget::MacOnly(mac_address), + ) + .await?; + let observed_at = + DateTime::from_timestamp(1_722_000_100, 123_000_000).expect("fixture timestamp"); + assert!(mark_verified(txn.as_mut(), &machine_id, initialized.version, observed_at,).await?); + + let enriched = + enrich_interface_id(txn.as_mut(), &machine_id, mac_address, "NIC.Slot.8-1-1") + .await? + .expect("enriched target"); + assert_eq!( + enriched.version.version_nr(), + initialized.version.version_nr() + 1, + "the enriched value remains a distinct immutable desired generation", + ); + assert_eq!( + status_observation(txn.as_mut(), &machine_id).await?, + (Some(enriched.version), Some(observed_at), false), + "adding an id for the same MAC must not schedule redundant convergence", + ); + + let replacement = MachineBootInterfaceTarget::MacOnly(MacAddress::new([2, 0, 0, 0, 3, 9])); + let replaced = set(txn.as_mut(), &machine_id, &replacement).await?; + assert_ne!(replaced.version, enriched.version); + assert_eq!( + status_observation(txn.as_mut(), &machine_id).await?, + (Some(enriched.version), Some(observed_at), false), + "a different target must retain only the stale factual observation", + ); + + Ok(()) + } + #[crate::sqlx_test] async fn desired_targets_reject_dpu_ids( pool: PgPool, @@ -771,6 +1193,16 @@ mod tests { enrich_interface_id(txn.as_mut(), &machine_id, mac_address, "id").await, Err(DatabaseError::InvalidArgument(_)) )); + assert!(matches!( + mark_verified( + txn.as_mut(), + &machine_id, + ConfigVersion::initial(), + Utc::now(), + ) + .await, + Err(DatabaseError::InvalidArgument(_)) + )); Ok(()) } @@ -913,4 +1345,98 @@ mod tests { Ok(()) } + + #[crate::sqlx_test] + async fn status_migration_scopes_rollout_baseline_and_constrains_status( + pool: PgPool, + ) -> Result<(), Box> { + sqlx::query("DROP TABLE machine_boot_interfaces") + .execute(&pool) + .await?; + + let mut txn = pool.begin().await?; + let existing_host = machine_id(MachineType::Host, 40); + let in_flight_host = machine_id(MachineType::Host, 42); + seed_machine(txn.as_mut(), &existing_host).await?; + seed_machine(txn.as_mut(), &in_flight_host).await?; + set_controller_state(txn.as_mut(), &existing_host, ManagedHostState::Ready).await?; + set_controller_state( + txn.as_mut(), + &in_flight_host, + ManagedHostState::HostInit { + machine_state: MachineState::WaitingForPlatformConfiguration { retry_count: 0 }, + }, + ) + .await?; + txn.commit().await?; + + sqlx::raw_sql(MIGRATION).execute(&pool).await?; + let existing_version = ConfigVersion::initial(); + for (machine_id, mac_address) in [ + (existing_host, MacAddress::new([2, 0, 0, 0, 4, 0])), + (in_flight_host, MacAddress::new([2, 0, 0, 0, 4, 2])), + ] { + sqlx::query( + "INSERT INTO machine_boot_interfaces ( + machine_id, + desired_mac_address, + desired_version + ) + VALUES ($1, $2, $3)", + ) + .bind(machine_id) + .bind(mac_address) + .bind(existing_version) + .execute(&pool) + .await?; + } + + sqlx::raw_sql(STATUS_MIGRATION).execute(&pool).await?; + let (verified_version, observed_at, assumed): ( + Option, + Option>, + bool, + ) = sqlx::query_as( + "SELECT verified_version, observed_at, assumed + FROM machine_boot_interfaces + WHERE machine_id = $1", + ) + .bind(existing_host) + .fetch_one(&pool) + .await?; + assert_eq!(verified_version, Some(existing_version)); + assert!(observed_at.is_some()); + assert!(assumed); + + let in_flight_status: (Option, Option>, bool) = + sqlx::query_as( + "SELECT verified_version, observed_at, assumed + FROM machine_boot_interfaces + WHERE machine_id = $1", + ) + .bind(in_flight_host) + .fetch_one(&pool) + .await?; + assert_eq!(in_flight_status, (None, None, false)); + + for inconsistent_update in [ + "UPDATE machine_boot_interfaces + SET observed_at = CURRENT_TIMESTAMP + WHERE machine_id = $1", + "UPDATE machine_boot_interfaces + SET assumed = true + WHERE machine_id = $1", + ] { + let result = sqlx::query(inconsistent_update) + .bind(in_flight_host) + .execute(&pool) + .await; + assert!( + result.is_err(), + "the status consistency constraint should reject {inconsistent_update:?}", + ); + } + + Ok(()) + } } diff --git a/crates/api-db/src/sql/machine_snapshots.sql.template b/crates/api-db/src/sql/machine_snapshots.sql.template index 49a2d0fb8c..1c116335bc 100644 --- a/crates/api-db/src/sql/machine_snapshots.sql.template +++ b/crates/api-db/src/sql/machine_snapshots.sql.template @@ -9,6 +9,9 @@ m.*, boot_interface.desired_mac_address AS desired_boot_interface_mac, boot_interface.desired_interface_id AS desired_boot_interface_id, boot_interface.desired_version AS desired_boot_interface_version, +boot_interface.verified_version AS boot_interface_verified_version, +boot_interface.observed_at AS boot_interface_observed_at, +COALESCE(boot_interface.assumed, false) AS boot_interface_observation_assumed, sku.device_type as hw_sku_device_type, COALESCE(i.json, '[]') AS interfaces, COALESCE(t.json, '[]') AS topology, diff --git a/crates/api-db/src/sql/managed_hosts.sql.template b/crates/api-db/src/sql/managed_hosts.sql.template index 96421adb93..52f00df197 100644 --- a/crates/api-db/src/sql/managed_hosts.sql.template +++ b/crates/api-db/src/sql/managed_hosts.sql.template @@ -26,6 +26,9 @@ SELECT m.* FROM ( boot_interface.desired_mac_address AS desired_boot_interface_mac, boot_interface.desired_interface_id AS desired_boot_interface_id, boot_interface.desired_version AS desired_boot_interface_version, + boot_interface.verified_version AS boot_interface_verified_version, + boot_interface.observed_at AS boot_interface_observed_at, + COALESCE(boot_interface.assumed, false) AS boot_interface_observation_assumed, sku.device_type as hw_sku_device_type, COALESCE(i.json, '[]') AS interfaces, COALESCE(t.json, '[]') AS topology, @@ -131,6 +134,9 @@ SELECT m.* FROM ( boot_interface.desired_mac_address AS desired_boot_interface_mac, boot_interface.desired_interface_id AS desired_boot_interface_id, boot_interface.desired_version AS desired_boot_interface_version, + boot_interface.verified_version AS boot_interface_verified_version, + boot_interface.observed_at AS boot_interface_observed_at, + COALESCE(boot_interface.assumed, false) AS boot_interface_observation_assumed, COALESCE(i.json, '[]') AS interfaces, COALESCE(t.json, '[]') AS topology, COALESCE(bmc.json, t.bmc_info - 'ip' - 'mac', '{}'::jsonb) AS bmc_info diff --git a/crates/api-model/src/machine/json.rs b/crates/api-model/src/machine/json.rs index b405b43066..acfe16aa94 100644 --- a/crates/api-model/src/machine/json.rs +++ b/crates/api-model/src/machine/json.rs @@ -41,7 +41,7 @@ use crate::machine::{ ManagedHostState, ReprovisionRequest, UpgradeDecision, }; use crate::machine_boot_interface::{ - MachineBootInterfaceTarget, canonical_redfish_boot_interface_id, + BootInterfaceStatusObservation, MachineBootInterfaceTarget, canonical_redfish_boot_interface_id, }; use crate::metadata::Metadata; use crate::power_manager::PowerOptions; @@ -109,6 +109,10 @@ pub struct MachineSnapshotPgJson { pub desired_boot_interface_mac: Option, pub desired_boot_interface_id: Option, pub desired_boot_interface_version: Option, + pub boot_interface_verified_version: Option, + pub boot_interface_observed_at: Option>, + #[serde(default)] + pub boot_interface_observation_assumed: bool, pub hw_sku_status: Option, #[serde(default)] // Power options are valid only for host, not for DPUs. pub power_options: Option, @@ -171,6 +175,37 @@ fn decode_desired_boot_interface( } } +fn decode_boot_interface_status_observation( + config_version: Option, + observed_at: Option>, + assumed: bool, +) -> sqlx::Result> { + match (config_version, observed_at, assumed) { + (None, None, false) => Ok(None), + (Some(config_version), Some(observed_at), assumed) => { + let config_version = + config_version + .parse() + .map_err(|error| sqlx::Error::ColumnDecode { + index: "boot_interface_verified_version".to_string(), + source: Box::new(error), + })?; + Ok(Some(BootInterfaceStatusObservation { + config_version, + observed_at, + assumed, + })) + } + _ => Err(sqlx::Error::ColumnDecode { + index: "boot_interface_(verified_version,observed_at,assumed)".to_string(), + source: Box::new(std::io::Error::new( + std::io::ErrorKind::InvalidData, + "boot interface verified version and observation time must both be set or both be null, and assumed requires an observation", + )), + }), + } +} + impl TryFrom for Machine { type Error = sqlx::Error; @@ -196,6 +231,11 @@ impl TryFrom for Machine { value.desired_boot_interface_id, value.desired_boot_interface_version, )?; + let boot_interface_status_observation = decode_boot_interface_status_observation( + value.boot_interface_verified_version, + value.boot_interface_observed_at, + value.boot_interface_observation_assumed, + )?; let version: ConfigVersion = value @@ -261,6 +301,7 @@ impl TryFrom for Machine { }, status: MachineStatus { interfaces: value.interfaces, + boot_interface_status_observation, hardware_info, bmc_info: value.bmc_info, last_reboot_time: value.last_reboot_time, @@ -332,6 +373,13 @@ mod tests { }, } + #[derive(Debug)] + struct ObservationInput { + config_version: Option, + observed_at: Option>, + assumed: bool, + } + fn summarize(value: Option>) -> Decoded { match value { None => Decoded::Unset, @@ -464,4 +512,96 @@ mod tests { }, ); } + + #[test] + fn boot_interface_status_columns_decode_atomically() { + let observed_at = DateTime::from_timestamp(1_722_000_000, 123_000_000) + .expect("fixture timestamp is valid"); + let version = ConfigVersion::new(7); + let config_version = version.version_string(); + + check_cases( + [ + Case { + scenario: "no observation", + input: ObservationInput { + config_version: None, + observed_at: None, + assumed: false, + }, + expect: Yields(None), + }, + Case { + scenario: "Redfish observation", + input: ObservationInput { + config_version: Some(config_version.clone()), + observed_at: Some(observed_at), + assumed: false, + }, + expect: Yields(Some(BootInterfaceStatusObservation { + config_version: version, + observed_at, + assumed: false, + })), + }, + Case { + scenario: "rollout baseline", + input: ObservationInput { + config_version: Some(config_version.clone()), + observed_at: Some(observed_at), + assumed: true, + }, + expect: Yields(Some(BootInterfaceStatusObservation { + config_version: version, + observed_at, + assumed: true, + })), + }, + Case { + scenario: "version without time", + input: ObservationInput { + config_version: Some(config_version), + observed_at: None, + assumed: false, + }, + expect: Fails, + }, + Case { + scenario: "time without version", + input: ObservationInput { + config_version: None, + observed_at: Some(observed_at), + assumed: false, + }, + expect: Fails, + }, + Case { + scenario: "assumed without observation", + input: ObservationInput { + config_version: None, + observed_at: None, + assumed: true, + }, + expect: Fails, + }, + Case { + scenario: "malformed version", + input: ObservationInput { + config_version: Some("not-a-version".to_string()), + observed_at: Some(observed_at), + assumed: false, + }, + expect: Fails, + }, + ], + |ObservationInput { + config_version, + observed_at, + assumed, + }| { + decode_boot_interface_status_observation(config_version, observed_at, assumed) + .map_err(drop) + }, + ); + } } diff --git a/crates/api-model/src/machine/mod.rs b/crates/api-model/src/machine/mod.rs index d3660262c1..da4b5a07ce 100644 --- a/crates/api-model/src/machine/mod.rs +++ b/crates/api-model/src/machine/mod.rs @@ -42,7 +42,7 @@ use super::StateSla; use super::instance::snapshot::InstanceSnapshot; use super::instance::status::extension_service::InstanceExtensionServiceStatusObservation; use super::instance::status::network::InstanceNetworkStatusObservation; -use super::machine_boot_interface::MachineBootInterface; +use super::machine_boot_interface::{MachineBootInterface, MachineBootInterfaceTarget}; use super::metadata::Metadata; use crate::controller_outcome::PersistentStateHandlerOutcome; use crate::dpa_interface::DpaInterface; @@ -120,7 +120,14 @@ fn default_true() -> bool { } // This should be updated on each new model introduction -pub const CURRENT_STATE_MODEL_VERSION: i16 = 2; +pub const CURRENT_STATE_MODEL_VERSION: i16 = 3; + +fn pending_boot_interface_config_version( + desired_version: Option, + verified_version: Option, +) -> Option { + desired_version.filter(|desired_version| Some(*desired_version) != verified_version) +} /// Represents the current state of `Machine` #[derive(Debug, Clone)] @@ -228,6 +235,8 @@ pub enum NotAllocatableReason { "the machine has a pending instance creation request, that has not yet been processed by the state handler" )] PendingInstanceCreation, + #[error("the machine has a pending boot configuration")] + PendingBootConfiguration, #[error("there are no dpu_snapshots, but associated_dpu_machine_ids is non-empty")] NoDpuSnapshots, #[error("the machine is in maintenance mode")] @@ -462,6 +471,7 @@ impl ManagedHostStateSnapshot { /// - the Machine has not yet been target of an instance creation request /// - no health alerts which classification `PreventAllocations` to be set /// - the machine not to be in Maintenance Mode + /// - the desired boot-interface generation to have a matching observation pub fn is_usable_as_instance(&self, allow_unhealthy: bool) -> Result<(), NotAllocatableReason> { // TODO: allow other states than Ready when allow_unhealthy=true. Will require changes to state machine (see Matthias). if !matches!(self.managed_state, ManagedHostState::Ready) { @@ -477,6 +487,18 @@ impl ManagedHostStateSnapshot { return Err(NotAllocatableReason::PendingInstanceCreation); } + // A desired boot-interface update and instance allocation can race + // before machine-controller has persisted BootConfiguring. Keep the + // host unavailable as soon as the desired version lacks a matching + // convergence status. + if self + .host_snapshot + .pending_boot_interface_config_version() + .is_some() + { + return Err(NotAllocatableReason::PendingBootConfiguration); + } + if self.dpu_snapshots.is_empty() && !self.host_snapshot.associated_dpu_machine_ids().is_empty() { @@ -939,6 +961,25 @@ impl Machine { self.state.version } + /// Returns the desired boot-interface version whose persisted convergence + /// status is not current, if any. + /// + /// Comparing versions keeps the Ready-state decision DB-only. Redfish is + /// queried only after the controller has persisted + /// [`ManagedHostState::BootConfiguring`]. + pub fn pending_boot_interface_config_version(&self) -> Option { + pending_boot_interface_config_version( + self.config + .desired_boot_interface + .as_ref() + .map(|desired| desired.version), + self.status + .boot_interface_status_observation + .as_ref() + .map(|observation| observation.config_version), + ) + } + /// Latest health report received from forge-dpu-agent. pub fn dpu_agent_health_report(&self) -> Option<&HealthReport> { self.health_reports @@ -1166,6 +1207,23 @@ pub enum ManagedHostState { /// Host is Ready for instance creation. Ready, + /// An unassigned Ready host is converging its Redfish boot configuration + /// to the desired boot interface persisted on the machine. + /// + /// The desired target and version are captured when the repair starts. + /// The controller checks that version before issuing new Redfish writes, + /// uses the captured target while work is in flight, and records it + /// verified only when the version is still current after final observation. + BootConfiguring { + desired_version: ConfigVersion, + desired_boot_interface: MachineBootInterfaceTarget, + /// Number of complete reconciliation passes retried because the final + /// Redfish observation drifted after lockdown was restored. + #[serde(default)] + post_lock_verification_retry_count: u32, + boot_config_state: ReadyBootConfigState, + }, + /// Host is executing an operator-requested maintenance operation. Maintenance { operation: MachineMaintenanceOperation, @@ -1309,6 +1367,74 @@ pub enum MachineValidatingState { validation_id: MachineValidationId, }, } + +/// `ReadyBootConfigTerminalFailure` defers a terminal condition until Ready +/// boot convergence restores lockdown. +#[derive(Debug, Clone, Serialize, Deserialize, Eq, PartialEq)] +#[serde(tag = "kind", rename_all = "lowercase")] +pub enum ReadyBootConfigTerminalFailure { + /// The boot-config convergence flow could not complete automatically. + Convergence { failure: String }, + /// An independent host or DPU failure appeared while lockdown was open. + /// Preserve its original attribution while routing through LockHost. + Machine { + machine_id: MachineId, + details: FailureDetails, + }, +} + +/// `ReadyBootConfigState` persists progress while an unassigned Ready host +/// converges its desired Redfish boot configuration. +/// +/// BIOS and boot-order job details reuse the same model types as HostInit, +/// assigned platform configuration, and validation so controller restarts +/// retain vendor job IDs, recovery substates, and retry budgets. +#[derive(Debug, Clone, Serialize, Deserialize, Eq, PartialEq)] +#[serde(tag = "state", rename_all = "lowercase")] +pub enum ReadyBootConfigState { + /// Observe the target, then inspect lockdown only when a repair may write. + Prepare, + /// Disable lockdown, including any vendor-specific reboot and wait. + UnlockHost { + #[serde(default)] + unlock_host_state: UnlockHostState, + }, + /// Observe BIOS and boot order and select the smallest required repair. + CheckHostConfig, + /// Run `machine_setup` for the desired boot interface. + ConfigureBios { + #[serde(default)] + retry_count: u32, + }, + /// Wait for the vendor BIOS configuration job returned by `machine_setup`. + WaitingForBiosJob { bios_config_info: BiosConfigInfo }, + /// Verify that the BIOS configuration has been applied. + PollingBiosSetup { + #[serde(default)] + retry_count: u32, + }, + /// Set, apply, and verify boot order. + SetBootOrder { + set_boot_order_info: SetBootOrderInfo, + }, + /// Restore the configured lockdown policy before either conditionally + /// marking the desired boot-interface version verified or surfacing a + /// terminal convergence failure. + LockHost { + /// Failure deferred until lockdown has been restored. Absent on the + /// successful convergence path. + #[serde(default, skip_serializing_if = "Option::is_none")] + terminal_failure: Option, + }, + /// Automated convergence could not complete safely after lockdown was + /// restored. The host remains unavailable until an operator changes its + /// desired boot interface, starting a fresh pass from + /// [`ReadyBootConfigState::Prepare`], or successfully completes a + /// maintenance operation, which returns the host to + /// [`ManagedHostState::Ready`]. + Failed { failure: String }, +} + #[derive(Debug, Clone, Serialize, Deserialize, Eq, PartialEq)] #[serde(tag = "validation_type", rename_all = "lowercase")] pub enum ValidationState { @@ -2335,6 +2461,23 @@ impl Display for SpdmMeasuringState { } } +impl Display for ReadyBootConfigState { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + let name = match self { + Self::Prepare => "Prepare", + Self::UnlockHost { .. } => "UnlockHost", + Self::CheckHostConfig => "CheckHostConfig", + Self::ConfigureBios { .. } => "ConfigureBios", + Self::WaitingForBiosJob { .. } => "WaitingForBiosJob", + Self::PollingBiosSetup { .. } => "PollingBiosSetup", + Self::SetBootOrder { .. } => "SetBootOrder", + Self::LockHost { .. } => "LockHost", + Self::Failed { .. } => "Failed", + }; + f.write_str(name) + } +} + impl Display for ManagedHostState { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { match self { @@ -2363,6 +2506,11 @@ impl Display for ManagedHostState { write!(f, "HostInitializing/{machine_state}") } ManagedHostState::Ready => write!(f, "Ready"), + ManagedHostState::BootConfiguring { + boot_config_state, .. + } => { + write!(f, "BootConfiguring/{boot_config_state}") + } ManagedHostState::Maintenance { operation } => { write!(f, "Maintenance({operation:?})") } @@ -2459,6 +2607,11 @@ impl ManagedHostState { format!("HostInitializing/{machine_state}") } ManagedHostState::Ready => "Ready".to_string(), + ManagedHostState::BootConfiguring { + boot_config_state, .. + } => { + format!("BootConfiguring/{boot_config_state}") + } ManagedHostState::Maintenance { operation } => { format!("Maintenance({operation:?})") } @@ -2662,6 +2815,13 @@ pub fn state_sla( _ => StateSla::with_sla(slas::HOST_INIT, time_in_state), }, ManagedHostState::Ready => StateSla::no_sla(), + ManagedHostState::BootConfiguring { + boot_config_state: ReadyBootConfigState::Failed { .. }, + .. + } => StateSla::with_sla(std::time::Duration::ZERO, time_in_state), + ManagedHostState::BootConfiguring { .. } => { + StateSla::with_sla(slas::BOOT_CONFIGURING, time_in_state) + } ManagedHostState::Maintenance { .. } => { StateSla::with_sla(slas::MAINTENANCE, time_in_state) } @@ -3095,7 +3255,7 @@ mod tests { use std::str::FromStr; use carbide_test_support::Outcome::*; - use carbide_test_support::{Check, check_values, scenarios}; + use carbide_test_support::{Check, check_values, scenarios, value_scenarios}; use super::*; use crate::test_support::machine_snapshot::{ @@ -3133,6 +3293,151 @@ mod tests { reprovision_requested: false, }; + #[test] + fn pending_boot_interface_version_requires_matching_verification() { + let desired = ConfigVersion::new(7); + let stale = ConfigVersion::new(6); + + value_scenarios!( + run = |(desired_version, verified_version)| { + pending_boot_interface_config_version(desired_version, verified_version) + }; + "no desired target needs no verification" { + (None, None) => None, + } + "an unobserved desired target needs verification" { + (Some(desired), None) => Some(desired), + } + "a stale observation needs verification" { + (Some(desired), Some(stale)) => Some(desired), + } + "a matching observation is converged" { + (Some(desired), Some(desired)) => None, + } + ); + } + + #[test] + fn ready_boot_config_defaults_survive_persisted_state_loading() { + scenarios!( + run = |json| serde_json::from_str::(json).map_err(drop); + "unlock starts by disabling lockdown" { + r#"{"state":"unlockhost"}"# => Yields(ReadyBootConfigState::UnlockHost { + unlock_host_state: UnlockHostState::DisableLockdown, + }), + } + + "BIOS setup starts with no retries" { + r#"{"state":"configurebios"}"# => Yields(ReadyBootConfigState::ConfigureBios { + retry_count: 0, + }), + } + + "BIOS verification starts with no retries" { + r#"{"state":"pollingbiossetup"}"# => Yields( + ReadyBootConfigState::PollingBiosSetup { retry_count: 0 }, + ), + } + + "lockdown restoration defaults to the success path" { + r#"{"state":"lockhost"}"# => Yields(ReadyBootConfigState::LockHost { + terminal_failure: None, + }), + } + ); + } + + #[test] + fn ready_boot_config_terminal_outcomes_round_trip() { + let machine_id = + MachineId::from_str("fm100ds7blqjsadm2uuh3qqbf1h7k8pmf47um6v9uckrg7l03po8mhqgvng") + .unwrap(); + let failure_details = FailureDetails { + cause: FailureCause::BiosSetupFailed { + err: "BIOS job retries exhausted".to_string(), + }, + failed_at: DateTime::::UNIX_EPOCH, + source: FailureSource::StateMachine, + }; + + check_values( + [ + Check { + scenario: "convergence failure waits for lockdown", + input: ReadyBootConfigState::LockHost { + terminal_failure: Some(ReadyBootConfigTerminalFailure::Convergence { + failure: "BIOS job retries exhausted".to_string(), + }), + }, + expect: true, + }, + Check { + scenario: "independent machine failure keeps its attribution", + input: ReadyBootConfigState::LockHost { + terminal_failure: Some(ReadyBootConfigTerminalFailure::Machine { + machine_id, + details: failure_details, + }), + }, + expect: true, + }, + Check { + scenario: "terminal convergence failure persists", + input: ReadyBootConfigState::Failed { + failure: "BIOS job retries exhausted".to_string(), + }, + expect: true, + }, + ], + |state| { + serde_json::from_str::( + &serde_json::to_string(&state).unwrap(), + ) + .unwrap() + == state + }, + ); + } + + #[test] + fn ready_host_with_unverified_boot_interface_is_not_allocatable() { + let mut snapshot = managed_host_state_snapshot(); + let desired_version = ConfigVersion::new(7); + let desired_boot_interface = + MachineBootInterfaceTarget::MacOnly(MacAddress::new([1, 2, 3, 4, 5, 6])); + snapshot.host_snapshot.config.desired_boot_interface = + Some(Versioned::new(desired_boot_interface, desired_version)); + snapshot + .host_snapshot + .status + .boot_interface_status_observation = None; + + assert_eq!( + snapshot.is_usable_as_instance(false), + Err(NotAllocatableReason::PendingBootConfiguration) + ); + } + + #[test] + fn boot_configuring_state_has_stable_state_strings() { + let state = ManagedHostState::BootConfiguring { + desired_version: ConfigVersion::new(7), + desired_boot_interface: MachineBootInterfaceTarget::MacOnly(MacAddress::new([ + 1, 2, 3, 4, 5, 6, + ])), + post_lock_verification_retry_count: 0, + boot_config_state: ReadyBootConfigState::Failed { + failure: "payload must not enter state labels".to_string(), + }, + }; + let dpu_id = + MachineId::from_str("fm100ds7blqjsadm2uuh3qqbf1h7k8pmf47um6v9uckrg7l03po8mhqgvng") + .unwrap(); + + assert_eq!(state.to_string(), "BootConfiguring/Failed"); + assert_eq!(state.dpu_state_string(&dpu_id), "BootConfiguring/Failed"); + } + #[test] fn machine_bmc_vendor_delegates_to_hardware_info() { let mut with_hardware_info = host_machine(); @@ -3874,6 +4179,32 @@ mod tests { input: stale(ManagedHostState::Ready), expect: (None, false), }, + Check { + scenario: "active boot configuration uses the convergence SLA", + input: stale(ManagedHostState::BootConfiguring { + desired_version: ConfigVersion::initial(), + desired_boot_interface: MachineBootInterfaceTarget::MacOnly( + MacAddress::new([1, 2, 3, 4, 5, 6]), + ), + post_lock_verification_retry_count: 0, + boot_config_state: ReadyBootConfigState::Prepare, + }), + expect: (seconds(5_400), true), + }, + Check { + scenario: "terminal boot configuration immediately breaches its SLA", + input: stale(ManagedHostState::BootConfiguring { + desired_version: ConfigVersion::initial(), + desired_boot_interface: MachineBootInterfaceTarget::MacOnly( + MacAddress::new([1, 2, 3, 4, 5, 6]), + ), + post_lock_verification_retry_count: 0, + boot_config_state: ReadyBootConfigState::Failed { + failure: "BIOS job retries exhausted".to_string(), + }, + }), + expect: (seconds(0), true), + }, Check { scenario: "maintenance uses the maintenance SLA", input: stale(ManagedHostState::Maintenance { diff --git a/crates/api-model/src/machine/slas.rs b/crates/api-model/src/machine/slas.rs index 74f55c500a..097796da75 100644 --- a/crates/api-model/src/machine/slas.rs +++ b/crates/api-model/src/machine/slas.rs @@ -29,6 +29,10 @@ pub const DPUINIT_NOTINIT: Duration = Duration::from_secs(30 * 60); // EnableIpmiOverLan WaitingForPlatformConfiguration PollingBiosSetup UefiSetup Discovered Lockdown PollingLockdownStatus MachineValidating pub const HOST_INIT: Duration = Duration::from_secs(30 * 60); +// Ready-state boot-interface convergence may include vendor BIOS jobs, +// controlled reboots, and final observation after lockdown is restored. +pub const BOOT_CONFIGURING: Duration = Duration::from_secs(90 * 60); + pub const WAITING_FOR_CLEANUP: Duration = Duration::from_secs(30 * 60); pub const CREATED: Duration = Duration::from_secs(30 * 60); diff --git a/crates/api-model/src/machine/status.rs b/crates/api-model/src/machine/status.rs index 91ae96b2df..86e43f5640 100644 --- a/crates/api-model/src/machine/status.rs +++ b/crates/api-model/src/machine/status.rs @@ -22,6 +22,7 @@ use crate::machine::infiniband::MachineInfinibandStatusObservation; use crate::machine::nvlink::MachineNvLinkStatusObservation; use crate::machine::spx::MachineSpxStatusObservation; use crate::machine::{FailureDetails, MachineInterfaceSnapshot, MachineLastRebootRequested}; +use crate::machine_boot_interface::BootInterfaceStatusObservation; use crate::power_manager::PowerOptions; use crate::sku::SkuStatus; @@ -31,6 +32,8 @@ use crate::sku::SkuStatus; #[derive(Debug, Clone)] pub struct MachineStatus { pub interfaces: Vec, + /// Latest persisted convergence status for the desired boot interface. + pub boot_interface_status_observation: Option, pub hardware_info: Option, pub bmc_info: BmcInfo, pub last_reboot_time: Option>, diff --git a/crates/api-model/src/machine_boot_interface.rs b/crates/api-model/src/machine_boot_interface.rs index c8383b0a8b..bc3e2d3fd5 100644 --- a/crates/api-model/src/machine_boot_interface.rs +++ b/crates/api-model/src/machine_boot_interface.rs @@ -15,6 +15,8 @@ * limitations under the License. */ use carbide_utils::none_if_empty::NoneIfEmpty; +use chrono::{DateTime, Utc}; +use config_version::ConfigVersion; use mac_address::MacAddress; use serde::{Deserialize, Serialize}; @@ -98,6 +100,21 @@ pub enum MachineBootInterfaceTarget { MacOnly(MacAddress), } +/// Status for the desired boot-interface generation currently treated as converged. +/// +/// `assumed` is true for the compatibility baseline used when an already-stable +/// host has no persisted row, including during mixed-component rollout. Real +/// Redfish verification always records it as false. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct BootInterfaceStatusObservation { + /// Desired boot-interface configuration version this status applies to. + pub config_version: ConfigVersion, + /// Time this status was recorded. + pub observed_at: DateTime, + /// Whether this is a compatibility baseline rather than a Redfish observation. + pub assumed: bool, +} + impl MachineBootInterfaceTarget { /// Builds the strongest usable target from an endpoint record. /// diff --git a/crates/api-model/src/test_support/machine_snapshot.rs b/crates/api-model/src/test_support/machine_snapshot.rs index a8b213fe4f..1169b437ba 100644 --- a/crates/api-model/src/test_support/machine_snapshot.rs +++ b/crates/api-model/src/test_support/machine_snapshot.rs @@ -45,8 +45,8 @@ use crate::machine::json::MachineSnapshotPgJson; use crate::machine::network::{MachineNetworkStatusObservation, ManagedHostNetworkConfig}; use crate::machine::topology::{DiscoveryData, MachineTopology, TopologyData}; use crate::machine::{ - Dpf, FailureCause, FailureDetails, FailureSource, HostProfile, Machine, - MachineInterfaceSnapshot, MachineLastRebootRequested, MachineLastRebootRequestedMode, + CURRENT_STATE_MODEL_VERSION, Dpf, FailureCause, FailureDetails, FailureSource, HostProfile, + Machine, MachineInterfaceSnapshot, MachineLastRebootRequested, MachineLastRebootRequestedMode, ManagedHostState, ManagedHostStateSnapshot, UpgradeDecision, }; use crate::machine_interface::InterfaceType; @@ -457,7 +457,7 @@ pub fn machine_snapshot_pg_json(machine_id: MachineId) -> MachineSnapshotPgJson asn: Some(4_200_000_042), controller_state_outcome: None, current_machine_validation_id: None, - machine_state_model_version: 2, + machine_state_model_version: i32::from(CURRENT_STATE_MODEL_VERSION), instance_type_id: Some(uuid::Uuid::from_u128(0x4001).into()), interfaces, topology: vec![MachineTopology { @@ -485,6 +485,9 @@ pub fn machine_snapshot_pg_json(machine_id: MachineId) -> MachineSnapshotPgJson desired_boot_interface_mac: None, desired_boot_interface_id: None, desired_boot_interface_version: None, + boot_interface_verified_version: None, + boot_interface_observed_at: None, + boot_interface_observation_assumed: false, hw_sku_status: None, power_options: None, hw_sku_device_type: Some("compute".to_string()), diff --git a/crates/api-web/src/tests/env.rs b/crates/api-web/src/tests/env.rs index 30fa7bdd8e..2a1d5ab585 100644 --- a/crates/api-web/src/tests/env.rs +++ b/crates/api-web/src/tests/env.rs @@ -21,7 +21,6 @@ use carbide_api_core::test_support::fixture_config::{ use carbide_test_harness::dns::TestDomain; use carbide_test_harness::network::segment::TestNetworkSegment; use carbide_test_harness::prelude::*; -use model::machine::ManagedHostState; use model::test_support::ManagedHostConfig; pub struct TestEnv { @@ -84,7 +83,7 @@ impl TestEnv { host.report_dpu_network_status().await; host.insert_empty_host_health_report("test-harness-health") .await; - host.advance_state(ManagedHostState::Ready).await; + host.advance_to_converged_ready().await; (host, build_data) } } diff --git a/crates/bmc-mock/src/redfish/computer_system.rs b/crates/bmc-mock/src/redfish/computer_system.rs index 6c5c29ad54..c7b2324502 100644 --- a/crates/bmc-mock/src/redfish/computer_system.rs +++ b/crates/bmc-mock/src/redfish/computer_system.rs @@ -59,6 +59,25 @@ pub fn reset_target(system_id: &str) -> String { ) } +/// Return the HPE iLO boot settings resource used for persistent boot ordering. +fn hpe_boot_resource(system_id: &str) -> redfish::Resource<'static> { + redfish::Resource { + odata_id: Cow::Owned(format!( + "/redfish/v1/Systems/{system_id}/Bios/oem/hpe/boot/" + )), + odata_type: Cow::Borrowed("#HpeServerBootSettings.v2_0_0.HpeServerBootSettings"), + id: Cow::Borrowed("boot"), + name: Cow::Borrowed("Boot Settings"), + } +} + +/// HPE iLO settings payload for its persistent boot order. +#[derive(serde::Deserialize)] +#[serde(rename_all = "PascalCase")] +struct HpeBootSettingsPatch { + persistent_boot_config_order: Vec, +} + pub fn add_routes(r: Router, bmc_vendor: redfish::oem::BmcVendor) -> Router { const SYSTEM_ID: &str = "{system_id}"; const ETH_ID: &str = "{eth_id}"; @@ -67,7 +86,8 @@ pub fn add_routes(r: Router, bmc_vendor: redfish::oem::BmcVendor) -> R const LOG_ENTRY_ID: &str = "{log_entry_id}"; const PROCESSOR_ID: &str = "{processor_id}"; let bios = redfish::bios::resource(SYSTEM_ID); - r.route(&collection().odata_id, get(get_system_collection)) + let routes = r + .route(&collection().odata_id, get(get_system_collection)) .route( &resource(SYSTEM_ID).odata_id, get(get_system).patch(patch_system), @@ -141,7 +161,20 @@ pub fn add_routes(r: Router, bmc_vendor: redfish::oem::BmcVendor) -> R .route( &redfish::bios::change_password_target(&bios), post(change_bios_password_action), + ); + if matches!(bmc_vendor, redfish::oem::BmcVendor::Hpe) { + let hpe_boot = hpe_boot_resource(SYSTEM_ID); + // CombinedServer normalizes requests by removing trailing slashes before + // routing them, while the Redfish resource still advertises canonical + // trailing-slash OData identifiers. + let hpe_boot_path = hpe_boot.odata_id.trim_end_matches('/'); + routes.route(hpe_boot_path, get(get_hpe_boot)).route( + &format!("{hpe_boot_path}/settings"), + patch(patch_hpe_boot_settings), ) + } else { + routes + } } pub struct SingleSystemConfig { @@ -183,6 +216,9 @@ pub struct SingleSystemState { config: SingleSystemConfig, virtual_media: Option, boot_order_override: Mutex>>, + // HPE iLO uses OEM structured boot strings here, not the BootOption IDs + // exposed by the standard ComputerSystem BootOrder property. + hpe_boot_order_override: Mutex>>, boot_source_override: Mutex, secure_boot_enabled: Arc, bios_overrides: Arc>, @@ -268,6 +304,7 @@ impl SingleSystemState { config, virtual_media, boot_order_override: Mutex::new(None), + hpe_boot_order_override: Mutex::new(None), boot_source_override: Mutex::new(BootSourceOverride::default()), secure_boot_enabled: Arc::new(AtomicBool::new(false)), bios_overrides: Arc::new(Mutex::new(serde_json::json!({}))), @@ -305,6 +342,33 @@ impl SingleSystemState { self.boot_order_override.lock().unwrap().clone() } + /// Return the HPE OEM persistent order without changing standard BootOrder state. + fn hpe_boot_order(&self) -> Vec { + self.hpe_boot_order_override + .lock() + .unwrap() + .clone() + .unwrap_or_else(|| { + self.config + .boot_options + .iter() + .flatten() + .map(|option| { + let prefix = match option.kind { + BootOptionKind::Disk => "HD", + BootOptionKind::Network => "NIC", + }; + format!("{prefix}.BootOption.{}", option.boot_reference()) + }) + .collect() + }) + } + + /// Persist an HPE OEM boot order independently from standard BootOrder state. + fn set_hpe_boot_order(&self, boot_order: Vec) { + *self.hpe_boot_order_override.lock().unwrap() = Some(boot_order); + } + pub(crate) fn virtual_media(&self) -> Option<&redfish::virtual_media::VirtualMediaState> { self.virtual_media.as_ref() } @@ -755,6 +819,35 @@ async fn get_boot_option( .unwrap_or_else(http::not_found) } +/// Return the HPE iLO persistent boot-order resource. +async fn get_hpe_boot(State(state): State, Path(system_id): Path) -> Response { + let Some(system_state) = state.system_state.find(&system_id) else { + return http::not_found(); + }; + let boot_order = system_state.hpe_boot_order(); + hpe_boot_resource(&system_id) + .json_patch() + .patch(json!({ + "BootSources": [], + "DefaultBootOrder": ["PcieSlotNic", "PcieSlotStorage"], + "PersistentBootConfigOrder": boot_order, + })) + .into_ok_response() +} + +/// Apply the HPE iLO persistent boot order staged through its settings resource. +async fn patch_hpe_boot_settings( + State(state): State, + Path(system_id): Path, + Json(request): Json, +) -> Response { + let Some(system_state) = state.system_state.find(&system_id) else { + return http::not_found(); + }; + system_state.set_hpe_boot_order(request.persistent_boot_config_order); + json!({}).into_ok_response() +} + async fn get_log_services_collection( State(state): State, Path(system_id): Path, @@ -1092,3 +1185,71 @@ impl SystemBuilder { self.value } } + +#[cfg(test)] +mod tests { + use axum::Router; + use axum::body::{Body, to_bytes}; + use axum::http::header::CONTENT_TYPE; + use axum::http::{Method, Request, StatusCode}; + use tower::ServiceExt; + use tower_http::normalize_path::NormalizePathLayer; + + use super::*; + use crate::test_support::{NoopCallbacks, host_info}; + use crate::{HardwareType, MachineRouterOptions, machine_router}; + + /// Reads one successful JSON response from the in-process mock router. + async fn get_json(router: &Router, path: &str) -> serde_json::Value { + let response = router + .clone() + .oneshot(Request::builder().uri(path).body(Body::empty()).unwrap()) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + let body = to_bytes(response.into_body(), usize::MAX).await.unwrap(); + serde_json::from_slice(&body).unwrap() + } + + /// HPE OEM ordering round-trips without corrupting standard BootOption IDs. + #[tokio::test] + async fn hpe_boot_order_is_persisted_separately_from_standard_boot_order() { + let router = machine_router( + &host_info(HardwareType::HpeProliantDl380aGen11), + Arc::new(NoopCallbacks), + "test-host-id".to_string(), + false, + MachineRouterOptions::default(), + ) + .0 + .layer(NormalizePathLayer::trim_trailing_slash()); + let boot_path = hpe_boot_resource("1").odata_id; + let initial = get_json(&router, &boot_path).await; + assert_eq!( + initial["PersistentBootConfigOrder"], + json!(["NIC.BootOption.Boot0000", "HD.BootOption.Boot0001",]) + ); + + let updated_order = json!(["HD.BootOption.Boot0001", "NIC.BootOption.Boot0000",]); + let response = router + .clone() + .oneshot( + Request::builder() + .method(Method::PATCH) + .uri(format!("{boot_path}settings/")) + .header(CONTENT_TYPE, "application/json") + .body(Body::from( + json!({"PersistentBootConfigOrder": updated_order}).to_string(), + )) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + + let updated = get_json(&router, &boot_path).await; + assert_eq!(updated["PersistentBootConfigOrder"], updated_order); + let system = get_json(&router, &resource("1").odata_id).await; + assert_eq!(system["Boot"]["BootOrder"], json!(["Boot0000", "Boot0001"])); + } +} diff --git a/crates/machine-controller/src/handler.rs b/crates/machine-controller/src/handler.rs index 89c3cf6b0a..79e206f0ca 100644 --- a/crates/machine-controller/src/handler.rs +++ b/crates/machine-controller/src/handler.rs @@ -80,11 +80,13 @@ use model::machine::{ MachineLastRebootRequested, MachineLastRebootRequestedMode, MachineNextStateResolver, MachineState, MachineValidationContext, ManagedHostState, ManagedHostStateSnapshot, MeasuringState, NetworkConfigUpdateState, NextStateBFBSupport, PerformPowerOperation, - PowerDrainState, PowerState, ReprovisionState, RetryInfo, SecureEraseBossContext, - SecureEraseBossState, SetBootOrderInfo, SetBootOrderState, SetSecureBootState, - SpdmMeasuringState, StateMachineArea, UefiSetupInfo, UefiSetupState, UnlockHostState, - ValidationState, dpf_based_dpu_provisioning_possible, get_display_ids, + PowerDrainState, PowerState, ReadyBootConfigState, ReadyBootConfigTerminalFailure, + ReprovisionState, RetryInfo, SecureEraseBossContext, SecureEraseBossState, SetBootOrderInfo, + SetBootOrderState, SetSecureBootState, SpdmMeasuringState, StateMachineArea, UefiSetupInfo, + UefiSetupState, UnlockHostState, ValidationState, dpf_based_dpu_provisioning_possible, + get_display_ids, }; +use model::machine_boot_interface::MachineBootInterfaceTarget; use model::power_manager::PowerHandlingOutcome; use model::predicted_machine_interface::PredictedMachineInterface; use model::resource_pool::common::CommonPools; @@ -136,8 +138,9 @@ use helpers::{ }; use host_boot_config::{ HostBootConfigCheckOutcome, HostBootConfigDecision, HostBootConfigDpuFreshness, - HostBootConfigOutcome, HostBootConfigStage, check_host_boot_config, - initial_set_boot_order_info, run_host_boot_config_stage, should_skip_boot_order_remediation, + HostBootConfigOutcome, HostBootConfigStage, check_host_boot_config, decide_host_boot_config, + initial_set_boot_order_info, inspect_host_boot_config, run_host_boot_config_stage, + should_skip_boot_order_remediation, }; use state_controller::db_write_batch::DbWriteBatch; @@ -757,27 +760,71 @@ impl MachineStateHandler { if !matches!(mh_state, ManagedHostState::Failed { .. }) && let Some((machine_id, details)) = get_failed_state(mh_snapshot) { - tracing::error!( - host_id = %mh_snapshot.host_snapshot.id, - dpu_ids = %get_display_ids(&mh_snapshot.dpu_snapshots), - failed_machine_id = %machine_id, - ?details, - "ManagedHost is moved to Failed state", + let already_relocking_machine_failure = matches!( + &mh_state, + ManagedHostState::BootConfiguring { + boot_config_state: + ReadyBootConfigState::LockHost { + terminal_failure: + Some(ReadyBootConfigTerminalFailure::Machine { + machine_id: pending_machine_id, + details: pending_details, + }), + }, + .. + } if *pending_machine_id == machine_id && *pending_details == details ); - let next_state = match mh_state { - ManagedHostState::Assigned { .. } => ManagedHostState::Assigned { - instance_state: InstanceState::Failed { + + // Boot reconciliation may have opened lockdown. Preserve a newly + // observed host or DPU failure, but restore lockdown before moving + // the managed host into its terminal state. + if already_relocking_machine_failure { + tracing::info!( + host_machine_id = %mh_snapshot.host_snapshot.id, + failed_machine_id = %machine_id, + "Deferring global failure transition until boot reconciliation restores lockdown", + ); + } else { + tracing::error!( + host_id = %mh_snapshot.host_snapshot.id, + dpu_ids = %get_display_ids(&mh_snapshot.dpu_snapshots), + failed_machine_id = %machine_id, + ?details, + "ManagedHost is moved to Failed state", + ); + let next_state = match &mh_state { + ManagedHostState::Assigned { .. } => ManagedHostState::Assigned { + instance_state: InstanceState::Failed { + details, + machine_id, + }, + }, + ManagedHostState::BootConfiguring { + desired_version, + desired_boot_interface, + post_lock_verification_retry_count, + boot_config_state, + } if !matches!(boot_config_state, ReadyBootConfigState::Failed { .. }) => { + ready_boot_config_locking( + Versioned { + value: desired_boot_interface.clone(), + version: *desired_version, + }, + *post_lock_verification_retry_count, + Some(ReadyBootConfigTerminalFailure::Machine { + machine_id, + details, + }), + ) + } + _ => ManagedHostState::Failed { details, machine_id, + retry_count: 0, }, - }, - _ => ManagedHostState::Failed { - details, - machine_id, - retry_count: 0, - }, - }; - return Ok(StateHandlerOutcome::transition(next_state)); + }; + return Ok(StateHandlerOutcome::transition(next_state)); + } } match &mh_state { @@ -852,7 +899,9 @@ impl MachineStateHandler { return Ok(outcome); } - // Check if instance to be created. + // An already-committed instance wins before disruptive boot + // reconciliation. Allocation locks the machine row, while its + // eligibility check rejects an earlier pending desired version. if mh_snapshot.instance.is_some() { return Ok(StateHandlerOutcome::transition( ManagedHostState::PreAssignedMeasuring { @@ -861,6 +910,12 @@ impl MachineStateHandler { )); } + if let Some(next_state) = + pending_ready_boot_config_state(&mh_snapshot.host_snapshot) + { + return Ok(StateHandlerOutcome::transition(next_state)); + } + if let Some(outcome) = handle_bom_validation_requested( &self.host_handler.host_handler_params, mh_snapshot, @@ -1033,6 +1088,26 @@ impl MachineStateHandler { Ok(StateHandlerOutcome::do_nothing()) } + ManagedHostState::BootConfiguring { + desired_version, + desired_boot_interface, + post_lock_verification_retry_count, + boot_config_state, + } => { + handle_ready_boot_config( + ctx, + mh_snapshot, + &self.host_handler.host_handler_params.reachability_params, + Versioned { + value: desired_boot_interface.clone(), + version: *desired_version, + }, + *post_lock_verification_retry_count, + boot_config_state.clone(), + ) + .await + } + ManagedHostState::RotatingBmc { retry_count } => { // One tick converges every BMC that needs work: force-requested // devices (bypassing backoff) and, when site-wide rotation is @@ -2411,10 +2486,36 @@ impl StateHandler for MachineStateHandler { .is_empty() && mh_snapshot.dpu_snapshots.is_empty() { - tracing::error!("No DPU snapshot found for host {}", host_machine_id); - return Err(StateHandlerError::GenericError(eyre!( - "no DPU snapshot found" - ))); + if let Some(next_state) = + ready_boot_config_missing_dpu_recovery(&mh_snapshot.managed_state) + { + tracing::error!( + machine_id = %host_machine_id, + "DPU snapshots disappeared during boot reconciliation; restoring lockdown before parking the repair", + ); + return Ok(StateHandlerOutcome::transition(next_state)); + } + + let can_continue_without_dpu = match &mh_snapshot.managed_state { + ManagedHostState::BootConfiguring { + boot_config_state, .. + } => !ready_boot_config_may_have_opened_lockdown(boot_config_state), + _ => false, + }; + if !can_continue_without_dpu { + tracing::error!(machine_id = %host_machine_id, "No DPU snapshot found for host"); + return Err(StateHandlerError::GenericError(eyre!( + "no DPU snapshot found" + ))); + } + + // Prepare and Failed must still process desired-state changes, and + // LockHost only needs host Redfish. Keep those recovery paths + // dispatchable through a transient DPU snapshot gap. + tracing::warn!( + machine_id = %host_machine_id, + "Continuing boot reconciliation recovery without DPU snapshots", + ); } self.record_metrics(mh_snapshot, ctx); @@ -5394,41 +5495,917 @@ pub struct HostMachineStateHandler { host_handler_params: HostHandlerParams, } -impl HostMachineStateHandler { - pub fn new(host_handler_params: HostHandlerParams) -> Self { - Self { - host_handler_params, - } - } -} +impl HostMachineStateHandler { + pub fn new(host_handler_params: HostHandlerParams) -> Self { + Self { + host_handler_params, + } + } +} + +fn managed_host_network_config_version_synced_and_dpu_healthy( + dpu_snapshot: &Machine, + host_version: ConfigVersion, +) -> bool { + if !dpu_snapshot.managed_host_network_config_version_synced(host_version) { + return false; + } + + let Some(dpu_health) = dpu_snapshot.dpu_agent_health_report() else { + return false; + }; + + // Note that DPU alerts may be surpressed (classifications removed) in the aggregate health + // report so the individual DPU's report is used. + !dpu_health + .has_classification(&health_report::HealthAlertClassification::prevent_host_state_changes()) +} + +fn check_host_health_for_alerts(state: &ManagedHostStateSnapshot) -> Result<(), StateHandlerError> { + // In some states, DPU alerts may be surpressed (classifications removed) in the aggregate health report. + // Since this is not called from a state that supresses DPU alerts, this is ok here. + match state + .aggregate_health + .has_classification(&health_report::HealthAlertClassification::prevent_host_state_changes()) + { + true => Err(StateHandlerError::HealthProbeAlert), + false => Ok(()), + } +} + +/// Whether a captured desired target can be replaced before this substate runs. +/// +/// Pure checks and pre-write states can adopt newer intent. Vendor jobs, +/// polling, unlock choreography, and cleanup must finish against the captured +/// target so a concurrent write cannot orphan an in-flight Redfish operation. +fn ready_boot_config_can_adopt_latest(state: &ReadyBootConfigState) -> bool { + match state { + ReadyBootConfigState::Prepare + | ReadyBootConfigState::CheckHostConfig + | ReadyBootConfigState::ConfigureBios { .. } => true, + ReadyBootConfigState::SetBootOrder { + set_boot_order_info, + } => matches!( + set_boot_order_info.set_boot_order_state, + SetBootOrderState::SetBootOrder | SetBootOrderState::CheckBootOrder + ), + ReadyBootConfigState::UnlockHost { .. } + | ReadyBootConfigState::WaitingForBiosJob { .. } + | ReadyBootConfigState::PollingBiosSetup { .. } + | ReadyBootConfigState::LockHost { .. } + | ReadyBootConfigState::Failed { .. } => false, + } +} + +// One complete persisted retry tolerates a transient final observation. +// Repeated drift after lockdown strongly suggests that lockdown or another +// actor is undoing the configuration. +const MAX_POST_LOCK_VERIFICATION_RETRIES: u32 = 1; + +/// Returns the state required by newer intent, or `None` while the captured +/// desired version remains current. +fn ready_boot_config_superseded_state( + captured_version: ConfigVersion, + current_desired: Option>, +) -> Option { + if current_desired + .as_ref() + .is_some_and(|desired| desired.version == captured_version) + { + return None; + } + + Some(current_desired.map_or(ManagedHostState::Ready, |desired| { + ready_boot_configuring(desired, 0, ReadyBootConfigState::Prepare) + })) +} + +fn ready_boot_config_state_from_stage(stage: HostBootConfigStage) -> ReadyBootConfigState { + match stage { + HostBootConfigStage::ConfigureBios { retry_count } => { + ReadyBootConfigState::ConfigureBios { retry_count } + } + HostBootConfigStage::WaitingForBiosJob { bios_config_info } => { + ReadyBootConfigState::WaitingForBiosJob { bios_config_info } + } + HostBootConfigStage::PollingBiosSetup { retry_count } => { + ReadyBootConfigState::PollingBiosSetup { retry_count } + } + HostBootConfigStage::SetBootOrder { + set_boot_order_info, + } => ReadyBootConfigState::SetBootOrder { + set_boot_order_info, + }, + } +} + +/// Returns whether a Ready boot substate may have left lockdown disabled. +/// +/// Keep this match exhaustive so a new substate cannot silently skip cleanup. +fn ready_boot_config_may_have_opened_lockdown(state: &ReadyBootConfigState) -> bool { + match state { + ReadyBootConfigState::Prepare + | ReadyBootConfigState::LockHost { .. } + | ReadyBootConfigState::Failed { .. } => false, + ReadyBootConfigState::UnlockHost { .. } + | ReadyBootConfigState::CheckHostConfig + | ReadyBootConfigState::ConfigureBios { .. } + | ReadyBootConfigState::WaitingForBiosJob { .. } + | ReadyBootConfigState::PollingBiosSetup { .. } + | ReadyBootConfigState::SetBootOrder { .. } => true, + } +} + +/// Routes an active repair through cleanup when expected DPU snapshots vanish. +/// +/// `Prepare` has not opened lockdown, `LockHost` is already cleanup, and +/// `Failed` is reached only after cleanup. Every other substate may have +/// disabled lockdown. +fn ready_boot_config_missing_dpu_recovery(state: &ManagedHostState) -> Option { + let ManagedHostState::BootConfiguring { + desired_version, + desired_boot_interface, + post_lock_verification_retry_count, + boot_config_state, + } = state + else { + return None; + }; + if !ready_boot_config_may_have_opened_lockdown(boot_config_state) { + return None; + } + + Some(ready_boot_config_locking( + Versioned { + value: desired_boot_interface.clone(), + version: *desired_version, + }, + *post_lock_verification_retry_count, + Some(ReadyBootConfigTerminalFailure::Convergence { + failure: "expected DPU snapshots disappeared while boot-interface reconciliation may have left lockdown disabled" + .to_string(), + }), + )) +} + +fn ready_boot_config_requires_timeout_cleanup( + state: &ReadyBootConfigState, + time_in_state: chrono::Duration, +) -> bool { + let cleanup_timeout = chrono::Duration::from_std(model::machine::slas::BOOT_CONFIGURING) + .expect("BootConfiguring SLA fits chrono::Duration"); + time_in_state >= cleanup_timeout && ready_boot_config_may_have_opened_lockdown(state) +} + +fn ready_boot_configuring( + desired: Versioned, + post_lock_verification_retry_count: u32, + boot_config_state: ReadyBootConfigState, +) -> ManagedHostState { + ManagedHostState::BootConfiguring { + desired_version: desired.version, + desired_boot_interface: desired.value, + post_lock_verification_retry_count, + boot_config_state, + } +} + +/// Builds the convergence state for a machine whose desired boot-interface +/// version has not yet been verified. +fn pending_ready_boot_config_state(machine: &Machine) -> Option { + let desired = machine.config.desired_boot_interface.as_ref()?; + machine + .pending_boot_interface_config_version() + .map(|_| ready_boot_configuring(desired.clone(), 0, ReadyBootConfigState::Prepare)) +} + +fn ready_boot_config_locking( + desired: Versioned, + post_lock_verification_retry_count: u32, + terminal_failure: Option, +) -> ManagedHostState { + ready_boot_configuring( + desired, + post_lock_verification_retry_count, + ReadyBootConfigState::LockHost { terminal_failure }, + ) +} + +fn ready_boot_config_after_post_lock_drift( + desired: Versioned, + post_lock_verification_retry_count: u32, + machine_id: MachineId, +) -> ManagedHostState { + if post_lock_verification_retry_count < MAX_POST_LOCK_VERIFICATION_RETRIES { + return ready_boot_configuring( + desired, + post_lock_verification_retry_count + 1, + ReadyBootConfigState::Prepare, + ); + } + + ready_boot_configuring( + desired, + post_lock_verification_retry_count, + ReadyBootConfigState::Failed { + failure: format!( + "Boot configuration on host {machine_id} repeatedly drifted after lockdown was restored; manual intervention required" + ), + }, + ) +} + +/// Maps the shared HostInit boot driver onto persisted Ready substates. +/// +/// Terminal driver failures pass through `LockHost` so convergence cannot park +/// while lockdown may be open. +async fn handle_ready_boot_config_stage( + ctx: &mut StateHandlerContext<'_, MachineStateHandlerContextObjects>, + mh_snapshot: &ManagedHostStateSnapshot, + reachability_params: &ReachabilityParams, + redfish_client: &dyn Redfish, + desired: Versioned, + post_lock_verification_retry_count: u32, + stage: HostBootConfigStage, +) -> Result, StateHandlerError> { + let boot_interface: BootInterfaceTarget = desired.value.clone().into(); + let outcome = match run_host_boot_config_stage( + ctx, + reachability_params, + redfish_client, + mh_snapshot, + Some(&boot_interface), + stage, + ) + .await + { + Ok(outcome) => outcome, + Err(StateHandlerError::ManualInterventionRequired(failure)) => { + return Ok(StateHandlerOutcome::transition(ready_boot_config_locking( + desired, + post_lock_verification_retry_count, + Some(ReadyBootConfigTerminalFailure::Convergence { failure }), + ))); + } + Err(error) => return Err(error), + }; + + match outcome { + HostBootConfigOutcome::Continue(stage) => { + Ok(StateHandlerOutcome::transition(ready_boot_configuring( + desired, + post_lock_verification_retry_count, + ready_boot_config_state_from_stage(stage), + ))) + } + HostBootConfigOutcome::Complete => Ok(StateHandlerOutcome::transition( + ready_boot_config_locking(desired, post_lock_verification_retry_count, None), + )), + HostBootConfigOutcome::Wait(reason) => Ok(StateHandlerOutcome::wait(reason)), + HostBootConfigOutcome::Failed { failure } => { + Ok(StateHandlerOutcome::transition(ready_boot_config_locking( + desired, + post_lock_verification_retry_count, + Some(ReadyBootConfigTerminalFailure::Convergence { failure }), + ))) + } + } +} + +/// Converges an unassigned Ready host to its persisted boot-interface target. +/// +/// The outer state captures one target and desired version. Safe boundaries +/// may adopt newer intent, while an in-flight vendor job or cleanup finishes +/// against the captured target before the controller switches versions. +async fn handle_ready_boot_config( + ctx: &mut StateHandlerContext<'_, MachineStateHandlerContextObjects>, + mh_snapshot: &ManagedHostStateSnapshot, + reachability_params: &ReachabilityParams, + desired: Versioned, + post_lock_verification_retry_count: u32, + boot_config_state: ReadyBootConfigState, +) -> Result, StateHandlerError> { + // Only states that can adopt replacement intent need an unlocked read. + // LockHost re-reads under the machine-row lock before it commits a + // transition, and in-flight vendor stages deliberately finish their + // captured generation. + let current_desired = if matches!(boot_config_state, ReadyBootConfigState::Failed { .. }) + || ready_boot_config_can_adopt_latest(&boot_config_state) + { + let mut conn = ctx.services.db_pool.acquire().await?; + db::machine_desired_boot_interface::get(conn.as_mut(), &mh_snapshot.host_snapshot.id) + .await? + } else { + None + }; + let captured_boot_interface: BootInterfaceTarget = desired.value.clone().into(); + + if ready_boot_config_requires_timeout_cleanup( + &boot_config_state, + mh_snapshot.host_snapshot.state.version.since_state_change(), + ) { + let failure = format!( + "boot-interface reconciliation stopped progressing in {boot_config_state:?} for longer than its {}-second cleanup deadline", + model::machine::slas::BOOT_CONFIGURING.as_secs(), + ); + tracing::error!( + machine_id = %mh_snapshot.host_snapshot.id, + desired_version = %desired.version, + ?boot_config_state, + reason = %failure, + "Restoring lockdown before parking timed-out boot reconciliation", + ); + return Ok(StateHandlerOutcome::transition(ready_boot_config_locking( + desired, + post_lock_verification_retry_count, + Some(ReadyBootConfigTerminalFailure::Convergence { failure }), + ))); + } + + // A terminal failure stays scoped to the captured desired generation. + // A maintenance request can take control, while changing intent creates a + // new version and restarts convergence. + if matches!(boot_config_state, ReadyBootConfigState::Failed { .. }) { + if let Some(outcome) = maintenance::maintenance_transition_if_requested(mh_snapshot) { + return Ok(outcome); + } + return Ok( + match ready_boot_config_superseded_state(desired.version, current_desired) { + Some(next_state) => StateHandlerOutcome::transition(next_state), + None => StateHandlerOutcome::do_nothing(), + }, + ); + } + + if ready_boot_config_can_adopt_latest(&boot_config_state) + && current_desired.as_ref().map(|current| current.version) != Some(desired.version) + { + return Ok(match current_desired { + Some(current_desired) => { + let next_state = if matches!(boot_config_state, ReadyBootConfigState::Prepare) { + ReadyBootConfigState::Prepare + } else { + // Re-observe the replacement before any Redfish write. + ReadyBootConfigState::CheckHostConfig + }; + StateHandlerOutcome::transition(ready_boot_configuring( + current_desired, + 0, + next_state, + )) + } + None if matches!(boot_config_state, ReadyBootConfigState::Prepare) => { + StateHandlerOutcome::transition(ManagedHostState::Ready) + } + None => { + // The target disappeared after lockdown may have opened. + StateHandlerOutcome::transition(ready_boot_config_locking( + desired, + post_lock_verification_retry_count, + None, + )) + } + }); + } + + if matches!(boot_config_state, ReadyBootConfigState::Prepare) + && !mh_snapshot + .host_snapshot + .associated_dpu_machine_ids() + .is_empty() + && mh_snapshot.dpu_snapshots.is_empty() + { + // Prepare has not opened lockdown, so it can safely process target + // replacement or removal above. Do not let the shared boot check + // mistake a transiently empty snapshot list for a zero-DPU host. + return Ok(StateHandlerOutcome::wait( + "Waiting for expected DPU snapshots before boot-interface reconciliation".to_string(), + )); + } + + match boot_config_state { + ReadyBootConfigState::Prepare => { + if mh_snapshot.host_snapshot.bmc_vendor().is_supermicro() { + // A locked Supermicro BMC can report stale boot order. Reboot + // after disabling lockdown before performing the exact read. + return Ok(StateHandlerOutcome::transition(ready_boot_configuring( + desired, + post_lock_verification_retry_count, + ReadyBootConfigState::UnlockHost { + unlock_host_state: UnlockHostState::DisableLockdown, + }, + ))); + } + + let redfish_client = ctx + .services + .create_redfish_client_from_machine(&mh_snapshot.host_snapshot) + .await?; + + let preflight_decision = match check_host_boot_config( + redfish_client.as_ref(), + mh_snapshot, + reachability_params, + HostBootConfigDpuFreshness::CurrentHostState, + Some(&captured_boot_interface), + ctx, + ) + .await? + { + HostBootConfigCheckOutcome::Wait(reason) => { + return Ok(StateHandlerOutcome::wait(reason)); + } + HostBootConfigCheckOutcome::Ready(decision) => decision, + }; + let preflight_complete = matches!(preflight_decision, HostBootConfigDecision::Complete); + + let next_state = if preflight_complete { + // Avoid opening an ordinary host that is already correct. + ReadyBootConfigState::LockHost { + terminal_failure: None, + } + } else { + match redfish_client.lockdown_status().await { + Err(RedfishError::NotSupported(_)) => { + tracing::info!( + machine_id = %mh_snapshot.host_snapshot.id, + "BMC vendor does not support checking lockdown status during Ready boot repair", + ); + ReadyBootConfigState::CheckHostConfig + } + Err(error) => { + tracing::warn!( + machine_id = %mh_snapshot.host_snapshot.id, + error = %error, + "Failed to fetch lockdown status during Ready boot repair", + ); + return Ok(StateHandlerOutcome::wait(format!( + "Failed to fetch lockdown status: {error}" + ))); + } + Ok(lockdown_status) if !lockdown_status.is_fully_disabled() => { + ReadyBootConfigState::UnlockHost { + unlock_host_state: UnlockHostState::DisableLockdown, + } + } + Ok(_) => ReadyBootConfigState::CheckHostConfig, + } + }; + + Ok(StateHandlerOutcome::transition(ready_boot_configuring( + desired, + post_lock_verification_retry_count, + next_state, + ))) + } + ReadyBootConfigState::UnlockHost { unlock_host_state } => { + let redfish_client = ctx + .services + .create_redfish_client_from_machine(&mh_snapshot.host_snapshot) + .await?; + + let next_state = match unlock_host_state { + UnlockHostState::DisableLockdown => { + match redfish_client.lockdown_bmc(EnabledDisabled::Disabled).await { + Ok(()) => {} + Err(RedfishError::NotSupported(_)) => { + tracing::info!( + machine_id = %mh_snapshot.host_snapshot.id, + "BMC vendor does not support disabling lockdown during Ready boot repair", + ); + } + Err(error) => return Err(redfish_error("lockdown_bmc", error)), + } + + if mh_snapshot.host_snapshot.bmc_vendor().is_supermicro() { + ReadyBootConfigState::UnlockHost { + unlock_host_state: UnlockHostState::RebootHost, + } + } else { + ReadyBootConfigState::CheckHostConfig + } + } + UnlockHostState::RebootHost => { + host_power_control( + redfish_client.as_ref(), + &mh_snapshot.host_snapshot, + SystemPowerControl::ForceRestart, + ctx, + ) + .await + .map_err(|error| { + StateHandlerError::GenericError(eyre!( + "failed to ForceRestart host after disabling BMC lockdown: {error}" + )) + })?; + + ReadyBootConfigState::UnlockHost { + unlock_host_state: UnlockHostState::WaitForUefiBoot, + } + } + UnlockHostState::WaitForUefiBoot => { + let entered_at = mh_snapshot.host_snapshot.state.version.timestamp(); + if wait(&entered_at, reachability_params.uefi_boot_wait) { + return Ok(StateHandlerOutcome::wait(format!( + "Waiting for UEFI boot to complete on {} after post-unlock reboot", + mh_snapshot.host_snapshot.id + ))); + } + ReadyBootConfigState::CheckHostConfig + } + }; + + Ok(StateHandlerOutcome::transition(ready_boot_configuring( + desired, + post_lock_verification_retry_count, + next_state, + ))) + } + ReadyBootConfigState::CheckHostConfig => { + let redfish_client = ctx + .services + .create_redfish_client_from_machine(&mh_snapshot.host_snapshot) + .await?; + + let next_state = match check_host_boot_config( + redfish_client.as_ref(), + mh_snapshot, + reachability_params, + HostBootConfigDpuFreshness::CurrentHostState, + Some(&captured_boot_interface), + ctx, + ) + .await? + { + HostBootConfigCheckOutcome::Wait(reason) => { + return Ok(StateHandlerOutcome::wait(reason)); + } + HostBootConfigCheckOutcome::Ready(HostBootConfigDecision::ConfigureBios) => { + ReadyBootConfigState::ConfigureBios { retry_count: 0 } + } + HostBootConfigCheckOutcome::Ready(HostBootConfigDecision::SetBootOrder) => { + ReadyBootConfigState::SetBootOrder { + set_boot_order_info: initial_set_boot_order_info(), + } + } + HostBootConfigCheckOutcome::Ready(HostBootConfigDecision::Complete) => { + ReadyBootConfigState::LockHost { + terminal_failure: None, + } + } + }; + + Ok(StateHandlerOutcome::transition(ready_boot_configuring( + desired, + post_lock_verification_retry_count, + next_state, + ))) + } + ReadyBootConfigState::ConfigureBios { retry_count } => { + let redfish_client = ctx + .services + .create_redfish_client_from_machine(&mh_snapshot.host_snapshot) + .await?; + handle_ready_boot_config_stage( + ctx, + mh_snapshot, + reachability_params, + redfish_client.as_ref(), + desired, + post_lock_verification_retry_count, + HostBootConfigStage::ConfigureBios { retry_count }, + ) + .await + } + ReadyBootConfigState::WaitingForBiosJob { bios_config_info } => { + let redfish_client = ctx + .services + .create_redfish_client_from_machine(&mh_snapshot.host_snapshot) + .await?; + handle_ready_boot_config_stage( + ctx, + mh_snapshot, + reachability_params, + redfish_client.as_ref(), + desired, + post_lock_verification_retry_count, + HostBootConfigStage::WaitingForBiosJob { bios_config_info }, + ) + .await + } + ReadyBootConfigState::PollingBiosSetup { retry_count } => { + let redfish_client = ctx + .services + .create_redfish_client_from_machine(&mh_snapshot.host_snapshot) + .await?; + handle_ready_boot_config_stage( + ctx, + mh_snapshot, + reachability_params, + redfish_client.as_ref(), + desired, + post_lock_verification_retry_count, + HostBootConfigStage::PollingBiosSetup { retry_count }, + ) + .await + } + ReadyBootConfigState::SetBootOrder { + set_boot_order_info, + } => { + let redfish_client = ctx + .services + .create_redfish_client_from_machine(&mh_snapshot.host_snapshot) + .await?; + handle_ready_boot_config_stage( + ctx, + mh_snapshot, + reachability_params, + redfish_client.as_ref(), + desired, + post_lock_verification_retry_count, + HostBootConfigStage::SetBootOrder { + set_boot_order_info, + }, + ) + .await + } + ReadyBootConfigState::LockHost { terminal_failure } => { + let lockdown_disabled = mh_snapshot.host_snapshot.host_profile.disable_lockdown; + + // A profile that deliberately leaves lockdown disabled has no + // cleanup barrier. Terminal failures can therefore be published + // without requiring Redfish access. Successful convergence still + // performs the final exact-target observation below. + if lockdown_disabled && let Some(terminal_failure) = &terminal_failure { + match terminal_failure { + ReadyBootConfigTerminalFailure::Machine { + machine_id, + details, + } => { + return Ok(StateHandlerOutcome::transition(ManagedHostState::Failed { + details: details.clone(), + machine_id: *machine_id, + retry_count: 0, + })); + } + ReadyBootConfigTerminalFailure::Convergence { failure } => { + let mut txn = ctx.services.db_pool.begin().await?; + let current_desired = db::machine_desired_boot_interface::lock( + txn.as_mut(), + &mh_snapshot.host_snapshot.id, + ) + .await?; + let next_state = + ready_boot_config_superseded_state(desired.version, current_desired) + .unwrap_or_else(|| { + ready_boot_configuring( + desired, + post_lock_verification_retry_count, + ReadyBootConfigState::Failed { + failure: failure.clone(), + }, + ) + }); + return Ok(StateHandlerOutcome::transition(next_state).with_txn(txn)); + } + } + } + + let redfish_client = match ctx + .services + .create_redfish_client_from_machine(&mh_snapshot.host_snapshot) + .await + { + Ok(redfish_client) => redfish_client, + Err(error) => { + tracing::warn!( + machine_id = %mh_snapshot.host_snapshot.id, + error = %error, + "Waiting for Redfish access before completing Ready boot repair cleanup", + ); + return Ok(StateHandlerOutcome::wait( + "Waiting for host Redfish access before completing Ready boot repair cleanup" + .to_string(), + )); + } + }; + + if lockdown_disabled { + tracing::info!( + machine_id = %mh_snapshot.host_snapshot.id, + "Skipping lockdown re-enable after Ready boot repair per expected-machine config", + ); + } else { + let (lockdown_command_required, verify_after_command, require_supported_command) = + match redfish_client.lockdown_status().await { + Ok(lockdown_status) if lockdown_status.is_fully_enabled() => { + (false, false, false) + } + Ok(lockdown_status) => { + tracing::info!( + machine_id = %mh_snapshot.host_snapshot.id, + ?lockdown_status, + "Restoring lockdown after Ready boot repair", + ); + (true, true, true) + } + Err(RedfishError::NotSupported(_)) => { + // The command may still be supported even when the + // vendor has no corresponding status read. + (true, false, false) + } + Err(error) => { + tracing::warn!( + machine_id = %mh_snapshot.host_snapshot.id, + error = %error, + "Could not read lockdown status before Ready boot repair cleanup; attempting restoration", + ); + (true, true, true) + } + }; + + let lockdown_command_supported = if lockdown_command_required { + match redfish_client.lockdown_bmc(EnabledDisabled::Enabled).await { + Ok(()) => true, + Err(RedfishError::NotSupported(_)) if require_supported_command => { + return Ok(StateHandlerOutcome::wait( + "BMC lockdown status is not verified and the BMC does not support restoring it" + .to_string(), + )); + } + Err(RedfishError::NotSupported(_)) => { + tracing::info!( + machine_id = %mh_snapshot.host_snapshot.id, + "BMC vendor does not support re-enabling lockdown after Ready boot repair", + ); + false + } + Err(error) => return Err(redfish_error("lockdown_bmc", error)), + } + } else { + false + }; + + if lockdown_command_supported && verify_after_command { + match redfish_client.lockdown_status().await { + Ok(lockdown_status) if lockdown_status.is_fully_enabled() => {} + Ok(lockdown_status) => { + tracing::info!( + machine_id = %mh_snapshot.host_snapshot.id, + ?lockdown_status, + "Waiting for lockdown policy restoration after Ready boot repair", + ); + return Ok(StateHandlerOutcome::wait(format!( + "Waiting for lockdown to be fully enabled after Ready boot repair; current status: {lockdown_status:?}" + ))); + } + Err(RedfishError::NotSupported(_)) => { + tracing::warn!( + machine_id = %mh_snapshot.host_snapshot.id, + "BMC stopped supporting lockdown verification after reporting a readable status", + ); + return Ok(StateHandlerOutcome::wait( + "BMC lockdown restoration could not be verified".to_string(), + )); + } + Err(error) => { + tracing::warn!( + machine_id = %mh_snapshot.host_snapshot.id, + error = %error, + "Failed to verify lockdown after Ready boot repair", + ); + return Ok(StateHandlerOutcome::wait(format!( + "Failed to verify lockdown after Ready boot repair: {error}" + ))); + } + } + } + } -fn managed_host_network_config_version_synced_and_dpu_healthy( - dpu_snapshot: &Machine, - host_version: ConfigVersion, -) -> bool { - if !dpu_snapshot.managed_host_network_config_version_synced(host_version) { - return false; - } + if let Some(terminal_failure) = terminal_failure { + match terminal_failure { + ReadyBootConfigTerminalFailure::Machine { + machine_id, + details, + } => { + return Ok(StateHandlerOutcome::transition(ManagedHostState::Failed { + details, + machine_id, + retry_count: 0, + })); + } + ReadyBootConfigTerminalFailure::Convergence { failure } => { + let mut txn = ctx.services.db_pool.begin().await?; + let current_desired = db::machine_desired_boot_interface::lock( + txn.as_mut(), + &mh_snapshot.host_snapshot.id, + ) + .await?; + let next_state = + ready_boot_config_superseded_state(desired.version, current_desired) + .unwrap_or_else(|| { + ready_boot_configuring( + desired, + post_lock_verification_retry_count, + ReadyBootConfigState::Failed { failure }, + ) + }); + return Ok(StateHandlerOutcome::transition(next_state).with_txn(txn)); + } + } + } - let Some(dpu_health) = dpu_snapshot.dpu_agent_health_report() else { - return false; - }; + let boot_config_verified = + if mh_snapshot.host_snapshot.bmc_vendor().is_supermicro() && !lockdown_disabled { + // For a still-current desired generation, entry into LockHost + // durably records an exact read while Supermicro lockdown was + // disabled. Locked boot-order reads can be stale, so that + // checkpoint is stronger evidence than another Redfish read. + tracing::info!( + machine_id = %mh_snapshot.host_snapshot.id, + desired_version = %desired.version, + "Using the pre-lock Supermicro boot verification", + ); + true + } else { + // This state is restartable, so re-observe after lockdown is + // restored rather than trusting an earlier pre-crash read. + let inspection = inspect_host_boot_config( + redfish_client.as_ref(), + mh_snapshot, + &captured_boot_interface, + ) + .await?; + matches!( + decide_host_boot_config(inspection), + HostBootConfigDecision::Complete + ) + }; + if !boot_config_verified { + let mut txn = ctx.services.db_pool.begin().await?; + let current_desired = db::machine_desired_boot_interface::lock( + txn.as_mut(), + &mh_snapshot.host_snapshot.id, + ) + .await?; + let next_state = + ready_boot_config_superseded_state(desired.version, current_desired) + .unwrap_or_else(|| { + if post_lock_verification_retry_count + < MAX_POST_LOCK_VERIFICATION_RETRIES + { + tracing::warn!( + machine_id = %mh_snapshot.host_snapshot.id, + desired_version = %desired.version, + post_lock_verification_retry_count = + post_lock_verification_retry_count + 1, + "Boot configuration drifted after lockdown restoration; retrying reconciliation", + ); + } else { + tracing::warn!( + machine_id = %mh_snapshot.host_snapshot.id, + desired_version = %desired.version, + post_lock_verification_retry_count, + "Boot configuration repeatedly drifted after lockdown restoration; parking reconciliation", + ); + } + ready_boot_config_after_post_lock_drift( + desired, + post_lock_verification_retry_count, + mh_snapshot.host_snapshot.id, + ) + }); + return Ok(StateHandlerOutcome::transition(next_state).with_txn(txn)); + } - // Note that DPU alerts may be surpressed (classifications removed) in the aggregate health - // report so the individual DPU's report is used. - !dpu_health - .has_classification(&health_report::HealthAlertClassification::prevent_host_state_changes()) -} + let mut txn = ctx.services.db_pool.begin().await?; + let verified = db::machine_desired_boot_interface::mark_verified( + txn.as_mut(), + &mh_snapshot.host_snapshot.id, + desired.version, + Utc::now(), + ) + .await?; + let next_state = if verified { + ManagedHostState::Ready + } else { + match db::machine_desired_boot_interface::get( + txn.as_mut(), + &mh_snapshot.host_snapshot.id, + ) + .await? + { + Some(current_desired) => { + ready_boot_configuring(current_desired, 0, ReadyBootConfigState::Prepare) + } + None => ManagedHostState::Ready, + } + }; -fn check_host_health_for_alerts(state: &ManagedHostStateSnapshot) -> Result<(), StateHandlerError> { - // In some states, DPU alerts may be surpressed (classifications removed) in the aggregate health report. - // Since this is not called from a state that supresses DPU alerts, this is ok here. - match state - .aggregate_health - .has_classification(&health_report::HealthAlertClassification::prevent_host_state_changes()) - { - true => Err(StateHandlerError::HealthProbeAlert), - false => Ok(()), + Ok(StateHandlerOutcome::transition(next_state).with_txn(txn)) + } + // Handled above so superseding intent can restart a parked failure. + ReadyBootConfigState::Failed { .. } => Ok(StateHandlerOutcome::do_nothing()), } } @@ -5498,6 +6475,107 @@ async fn handle_host_init_boot_config_stage( } } +/// Finishes HostInit lockdown and records a fresh target-specific verification +/// in the same state-transition transaction when the final Redfish read agrees. +/// +/// A failed or mismatched read does not block ingestion. The desired version +/// remains pending, keeps the host out of allocation, and is handled by the +/// Ready reconciliation path. If operator intent changes before the exact +/// version is committed, the replacement likewise remains pending. +async fn complete_host_init_lockdown( + ctx: &mut StateHandlerContext<'_, MachineStateHandlerContextObjects>, + mh_snapshot: &ManagedHostStateSnapshot, + next_state: ManagedHostState, +) -> Result, StateHandlerError> { + let outcome = StateHandlerOutcome::transition(next_state); + let Some(desired) = mh_snapshot + .host_snapshot + .config + .desired_boot_interface + .as_ref() + else { + return Ok(outcome); + }; + + if mh_snapshot.host_snapshot.bmc_vendor().is_supermicro() + && !mh_snapshot.host_snapshot.host_profile.disable_lockdown + { + // Supermicro boot-order reads can become stale after lockdown. HostInit + // has no persisted pre-lock verification boundary, so leave this + // generation pending for the Ready convergence flow. + tracing::info!( + machine_id = %mh_snapshot.host_snapshot.id, + desired_version = %desired.version, + "Deferring Supermicro boot verification until Ready convergence", + ); + return Ok(outcome); + } + + let redfish_client = match ctx + .services + .create_redfish_client_from_machine(&mh_snapshot.host_snapshot) + .await + { + Ok(redfish_client) => redfish_client, + Err(error) => { + tracing::warn!( + machine_id = %mh_snapshot.host_snapshot.id, + desired_version = %desired.version, + error = %error, + "Could not create a Redfish client for HostInit boot verification; leaving it pending for Ready reconciliation", + ); + return Ok(outcome); + } + }; + let boot_interface: BootInterfaceTarget = desired.value.clone().into(); + let inspection = match inspect_host_boot_config( + redfish_client.as_ref(), + mh_snapshot, + &boot_interface, + ) + .await + { + Ok(inspection) => inspection, + Err(error) => { + tracing::warn!( + machine_id = %mh_snapshot.host_snapshot.id, + desired_version = %desired.version, + error = %error, + "Could not verify boot configuration after HostInit lockdown; leaving it pending for Ready reconciliation", + ); + return Ok(outcome); + } + }; + if !matches!( + decide_host_boot_config(inspection), + HostBootConfigDecision::Complete + ) { + tracing::warn!( + machine_id = %mh_snapshot.host_snapshot.id, + desired_version = %desired.version, + "Boot configuration differs after HostInit lockdown; leaving it pending for Ready reconciliation", + ); + return Ok(outcome); + } + + let mut txn = ctx.services.db_pool.begin().await?; + let verified = db::machine_desired_boot_interface::mark_verified( + txn.as_mut(), + &mh_snapshot.host_snapshot.id, + desired.version, + Utc::now(), + ) + .await?; + if !verified { + tracing::info!( + machine_id = %mh_snapshot.host_snapshot.id, + desired_version = %desired.version, + "Desired boot interface changed during HostInit verification; leaving it pending", + ); + } + Ok(outcome.with_txn(txn)) +} + /// TODO: we need to handle the case where the job is deleted for some reason /// Resolve the current site-wide UEFI target version (host_uefi or dpu_uefi) /// from `sitewide_credential_rotation.target_version` so ingestion drives a @@ -6083,7 +7161,9 @@ impl StateHandler for HostMachineStateHandler { machine_id = %host_machine_id, "Lockdown disabled per expected-machine config, skipping lockdown enable" ); - return Ok(StateHandlerOutcome::transition( + return complete_host_init_lockdown( + ctx, + mh_snapshot, ManagedHostState::BomValidating { bom_validating_state: BomValidating::MatchingSku( BomValidatingContext { @@ -6094,7 +7174,8 @@ impl StateHandler for HostMachineStateHandler { }, ), }, - )); + ) + .await; } tracing::info!( @@ -6138,23 +7219,39 @@ impl StateHandler for HostMachineStateHandler { } LockdownState::TimeWaitForDPUDown => { if !mh_snapshot.has_managed_dpus() { - // No DPU to wait for going down/up -- skip - // straight to BomValidating. Covers - // `Nic`/`Ignore` hosts and anything else - // with no DPU snapshots; otherwise we'd - // wait `dpu_wait_time` for a DPU that's - // never going to come up. - let next_state = ManagedHostState::BomValidating { - bom_validating_state: BomValidating::MatchingSku( - BomValidatingContext { - machine_validation_context: Some( - MachineValidationContext::Discovery, + // There is no DPU power cycle to observe for + // `Nic`/`Ignore` hosts. Disable still needs the + // shared status poll to reach platform + // configuration. Preserve the established + // Enable path, which does not require a + // separately readable lockdown status. + if lockdown_info.mode == LockdownMode::Enable { + return complete_host_init_lockdown( + ctx, + mh_snapshot, + ManagedHostState::BomValidating { + bom_validating_state: BomValidating::MatchingSku( + BomValidatingContext { + machine_validation_context: Some( + MachineValidationContext::Discovery, + ), + ..BomValidatingContext::default() + }, ), - reboot_retry_count: None, }, - ), - }; - return Ok(StateHandlerOutcome::transition(next_state)); + ) + .await; + } + return Ok(StateHandlerOutcome::transition( + ManagedHostState::HostInit { + machine_state: MachineState::WaitingForLockdown { + lockdown_info: LockdownInfo { + state: LockdownState::PollingLockdownStatus, + mode: lockdown_info.mode.clone(), + }, + }, + }, + )); } // Lets wait for some time before checking if DPU is up or not. // Waiting is needed because DPU takes some time to go down. If we check DPU @@ -6268,7 +7365,16 @@ impl StateHandler for HostMachineStateHandler { mode = ?lockdown_info.mode, "Lockdown status verified successfully" ); - Ok(StateHandlerOutcome::transition(next_state)) + if lockdown_info.mode == LockdownMode::Enable { + complete_host_init_lockdown( + ctx, + mh_snapshot, + next_state, + ) + .await + } else { + Ok(StateHandlerOutcome::transition(next_state)) + } } else { Ok(StateHandlerOutcome::wait(format!( "Polling lockdown status, waiting for {:?} to be applied. Current status: {:?}", @@ -6281,7 +7387,12 @@ impl StateHandler for HostMachineStateHandler { machine_id = %host_machine_id, "BMC vendor does not support checking lockdown status." ); - Ok(StateHandlerOutcome::transition(next_state)) + if lockdown_info.mode == LockdownMode::Enable { + complete_host_init_lockdown(ctx, mh_snapshot, next_state) + .await + } else { + Ok(StateHandlerOutcome::transition(next_state)) + } } Err(e) => { tracing::warn!( @@ -6305,7 +7416,10 @@ impl StateHandler for HostMachineStateHandler { // or Measuring state, depending on if machine attestation // is enabled or not. if rebooted(&mh_snapshot.host_snapshot) || *skip_reboot { - Ok(StateHandlerOutcome::transition(ManagedHostState::Ready)) + let next_state = + pending_ready_boot_config_state(&mh_snapshot.host_snapshot) + .unwrap_or(ManagedHostState::Ready); + Ok(StateHandlerOutcome::transition(next_state)) } else { let status = trigger_reboot_if_needed( &mh_snapshot.host_snapshot, @@ -12069,6 +13183,211 @@ mod tests { use super::*; + #[test] + fn terminal_ready_boot_config_failure_is_deferred_until_lockdown_restoration() { + let desired_version = ConfigVersion::initial(); + let desired_boot_interface = + MachineBootInterfaceTarget::MacOnly("02:00:00:00:00:01".parse().unwrap()); + let failure = "BIOS job retries exhausted".to_string(); + + assert_eq!( + ready_boot_config_locking( + Versioned::new(desired_boot_interface.clone(), desired_version), + 0, + Some(ReadyBootConfigTerminalFailure::Convergence { + failure: failure.clone(), + }), + ), + ManagedHostState::BootConfiguring { + desired_version, + desired_boot_interface, + post_lock_verification_retry_count: 0, + boot_config_state: ReadyBootConfigState::LockHost { + terminal_failure: Some(ReadyBootConfigTerminalFailure::Convergence { failure }), + }, + } + ); + } + + #[test] + fn missing_dpus_during_ready_boot_config_fails_closed() { + let desired_version = ConfigVersion::initial(); + let desired_boot_interface = + MachineBootInterfaceTarget::MacOnly("02:00:00:00:00:01".parse().unwrap()); + let active = ready_boot_configuring( + Versioned::new(desired_boot_interface.clone(), desired_version), + 0, + ReadyBootConfigState::CheckHostConfig, + ); + + assert!(matches!( + ready_boot_config_missing_dpu_recovery(&active), + Some(ManagedHostState::BootConfiguring { + desired_version: version, + desired_boot_interface: target, + boot_config_state: ReadyBootConfigState::LockHost { + terminal_failure: + Some(ReadyBootConfigTerminalFailure::Convergence { failure }), + }, + .. + }) if version == desired_version + && target == desired_boot_interface + && failure.contains("DPU snapshots disappeared") + )); + + for safe_state in [ + ReadyBootConfigState::Prepare, + ReadyBootConfigState::LockHost { + terminal_failure: None, + }, + ReadyBootConfigState::Failed { + failure: "already parked".to_string(), + }, + ] { + let state = ready_boot_configuring( + Versioned::new(desired_boot_interface.clone(), desired_version), + 0, + safe_state, + ); + assert_eq!(ready_boot_config_missing_dpu_recovery(&state), None); + } + } + + #[test] + fn stalled_ready_boot_config_cleans_up_only_after_work_can_start() { + let timed_out = chrono::Duration::from_std(model::machine::slas::BOOT_CONFIGURING).unwrap(); + let still_within_sla = timed_out - chrono::Duration::seconds(1); + + assert!(!ready_boot_config_requires_timeout_cleanup( + &ReadyBootConfigState::CheckHostConfig, + still_within_sla, + )); + assert!(ready_boot_config_requires_timeout_cleanup( + &ReadyBootConfigState::CheckHostConfig, + timed_out, + )); + + for safe_state in [ + ReadyBootConfigState::Prepare, + ReadyBootConfigState::LockHost { + terminal_failure: None, + }, + ReadyBootConfigState::Failed { + failure: "already parked".to_string(), + }, + ] { + assert!(!ready_boot_config_requires_timeout_cleanup( + &safe_state, + timed_out, + )); + } + } + + #[test] + fn post_lock_drift_retries_once_then_requires_intervention() { + let machine_id = + MachineId::from_str("fm100htes3rn1npvbtm5qd57dkilaag7ljugl1llmm7rfuq1ov50i0rpl30") + .unwrap(); + let desired_version = ConfigVersion::initial(); + let desired_boot_interface = + MachineBootInterfaceTarget::MacOnly("02:00:00:00:00:01".parse().unwrap()); + let desired = Versioned::new(desired_boot_interface.clone(), desired_version); + + assert_eq!( + ready_boot_config_after_post_lock_drift(desired.clone(), 0, machine_id), + ManagedHostState::BootConfiguring { + desired_version, + desired_boot_interface: desired_boot_interface.clone(), + post_lock_verification_retry_count: 1, + boot_config_state: ReadyBootConfigState::Prepare, + } + ); + + assert!(matches!( + ready_boot_config_after_post_lock_drift(desired, 1, machine_id), + ManagedHostState::BootConfiguring { + desired_version: version, + desired_boot_interface: target, + post_lock_verification_retry_count: 1, + boot_config_state: ReadyBootConfigState::Failed { failure }, + } if version == desired_version + && target == desired_boot_interface + && failure.contains("repeatedly drifted") + )); + } + + #[test] + fn ready_boot_config_adopts_targets_only_at_safe_boundaries() { + let set_boot_order = |set_boot_order_state| ReadyBootConfigState::SetBootOrder { + set_boot_order_info: SetBootOrderInfo { + set_boot_order_jid: None, + set_boot_order_state, + retry_count: 0, + }, + }; + + for state in [ + ReadyBootConfigState::Prepare, + ReadyBootConfigState::CheckHostConfig, + ReadyBootConfigState::ConfigureBios { retry_count: 0 }, + set_boot_order(SetBootOrderState::SetBootOrder), + set_boot_order(SetBootOrderState::CheckBootOrder), + ] { + assert!(ready_boot_config_can_adopt_latest(&state), "{state:?}"); + } + + for state in [ + ReadyBootConfigState::UnlockHost { + unlock_host_state: UnlockHostState::RebootHost, + }, + ReadyBootConfigState::PollingBiosSetup { retry_count: 0 }, + set_boot_order(SetBootOrderState::WaitForSetBootOrderJobCompletion), + ReadyBootConfigState::LockHost { + terminal_failure: Some(ReadyBootConfigTerminalFailure::Convergence { + failure: "exhausted".to_string(), + }), + }, + ReadyBootConfigState::Failed { + failure: "exhausted".to_string(), + }, + ] { + assert!(!ready_boot_config_can_adopt_latest(&state), "{state:?}"); + } + } + + #[test] + fn failed_ready_boot_config_restarts_only_for_superseding_intent() { + let failed_version = ConfigVersion::initial(); + let next_version = failed_version.increment(); + let failed_target = + MachineBootInterfaceTarget::MacOnly("02:00:00:00:00:01".parse().unwrap()); + let next_target = MachineBootInterfaceTarget::MacOnly("02:00:00:00:00:02".parse().unwrap()); + + assert_eq!( + ready_boot_config_superseded_state( + failed_version, + Some(Versioned::new(failed_target, failed_version)), + ), + None, + ); + assert_eq!( + ready_boot_config_superseded_state( + failed_version, + Some(Versioned::new(next_target.clone(), next_version)), + ), + Some(ManagedHostState::BootConfiguring { + desired_version: next_version, + desired_boot_interface: next_target, + post_lock_verification_retry_count: 0, + boot_config_state: ReadyBootConfigState::Prepare, + }), + ); + assert_eq!( + ready_boot_config_superseded_state(failed_version, None), + Some(ManagedHostState::Ready), + ); + } + #[test] fn ipmi_socket_address_uses_reported_or_default_port() { let ip_address = IpAddr::V4("192.0.2.10".parse().unwrap()); diff --git a/crates/machine-controller/src/io.rs b/crates/machine-controller/src/io.rs index 0a84cf7062..e4695decb9 100644 --- a/crates/machine-controller/src/io.rs +++ b/crates/machine-controller/src/io.rs @@ -28,7 +28,8 @@ use model::machine::slas::MachineSlaConfig; use model::machine::{ self, AttestationMode, DpuDiscoveringState, DpuInitState, HostHealthConfig, MachineMaintenanceOperation, MachineValidatingState, ManagedHostState, - ManagedHostStateSnapshot, MeasuringState, SpdmMeasuringState, ValidationState, + ManagedHostStateSnapshot, MeasuringState, ReadyBootConfigState, SpdmMeasuringState, + ValidationState, }; use sqlx::PgConnection; use state_controller::io::StateControllerIO; @@ -276,6 +277,21 @@ impl StateControllerIO for MachineStateControllerIO { MachineValidatingState::LockAfterBootRepair { .. } => "lockafterbootrepair", } } + + fn ready_boot_config_state_name(state: &ReadyBootConfigState) -> &'static str { + match state { + ReadyBootConfigState::Prepare => "prepare", + ReadyBootConfigState::UnlockHost { .. } => "unlockhost", + ReadyBootConfigState::CheckHostConfig => "checkhostconfig", + ReadyBootConfigState::ConfigureBios { .. } => "configurebios", + ReadyBootConfigState::WaitingForBiosJob { .. } => "waitingforbiosjob", + ReadyBootConfigState::PollingBiosSetup { .. } => "pollingbiossetup", + ReadyBootConfigState::SetBootOrder { .. } => "setbootorder", + ReadyBootConfigState::LockHost { .. } => "lockhost", + ReadyBootConfigState::Failed { .. } => "failed", + } + } + match state { ManagedHostState::DpuDiscoveringState { dpu_states } => { // Min state indicates the least processed DPU. The state machine is blocked @@ -299,6 +315,12 @@ impl StateControllerIO for MachineStateControllerIO { ("hostnotready", machine_state_name(machine_state)) } ManagedHostState::Ready => ("ready", ""), + ManagedHostState::BootConfiguring { + boot_config_state, .. + } => ( + "bootconfiguring", + ready_boot_config_state_name(boot_config_state), + ), ManagedHostState::Maintenance { operation } => { let op = match operation { MachineMaintenanceOperation::PowerOn => "power_on", @@ -376,6 +398,10 @@ impl StateControllerIO for MachineStateControllerIO { { Some(details.cause.metric_label()) } + ManagedHostState::BootConfiguring { + boot_config_state: ReadyBootConfigState::Failed { .. }, + .. + } => Some("boot_config_convergence_failed"), _ => None, } } @@ -394,3 +420,29 @@ impl StateControllerIO for MachineStateControllerIO { ) } } + +#[cfg(test)] +mod tests { + use model::machine_boot_interface::MachineBootInterfaceTarget; + + use super::*; + + #[test] + fn terminal_ready_boot_config_requires_manual_intervention() { + let state = ManagedHostState::BootConfiguring { + desired_version: ConfigVersion::initial(), + desired_boot_interface: MachineBootInterfaceTarget::MacOnly( + "02:00:00:00:00:01".parse().unwrap(), + ), + post_lock_verification_retry_count: 0, + boot_config_state: ReadyBootConfigState::Failed { + failure: "exhausted".to_string(), + }, + }; + + assert_eq!( + ::manual_intervention_reason(&state), + Some("boot_config_convergence_failed"), + ); + } +} diff --git a/crates/machine-controller/src/metrics.rs b/crates/machine-controller/src/metrics.rs index 232854ffd3..4ff6b9a6b2 100644 --- a/crates/machine-controller/src/metrics.rs +++ b/crates/machine-controller/src/metrics.rs @@ -56,6 +56,7 @@ pub struct MachineMetrics { /// - the Machine has not yet been target of an instance creation request /// - no health alerts which classification `PreventAllocations` to be set /// - the machine not to be in Maintenance Mode + /// - the desired boot-interface generation to have a matching observation pub is_usable_as_instance: bool, /// is the host's bios password set pub is_host_bios_password_set: bool, diff --git a/crates/machine-controller/tests/integration/bmc_rotation.rs b/crates/machine-controller/tests/integration/bmc_rotation.rs index f9854ae92a..709d25ba1e 100644 --- a/crates/machine-controller/tests/integration/bmc_rotation.rs +++ b/crates/machine-controller/tests/integration/bmc_rotation.rs @@ -93,7 +93,7 @@ async fn ready_host_converges_bmc_to_site_target( .build() .await .0; - mh.advance_state(ManagedHostState::Ready).await; + mh.advance_to_converged_ready().await; let host_mac = mh .host @@ -236,7 +236,7 @@ async fn feature_flag_off_suppresses_passive_rotation( .build() .await .0; - mh.advance_state(ManagedHostState::Ready).await; + mh.advance_to_converged_ready().await; let host_mac = mh .host @@ -297,7 +297,7 @@ async fn force_request_converges_quarantined_bmc_when_disabled( .build() .await .0; - mh.advance_state(ManagedHostState::Ready).await; + mh.advance_to_converged_ready().await; let host = mh.host.machine().await; let machine_id = host.id; diff --git a/crates/machine-controller/tests/integration/maintenance.rs b/crates/machine-controller/tests/integration/maintenance.rs index f23e6f37aa..04ad21d2bc 100644 --- a/crates/machine-controller/tests/integration/maintenance.rs +++ b/crates/machine-controller/tests/integration/maintenance.rs @@ -193,7 +193,7 @@ async fn create_ready_host( .build() .await .0; - host.advance_state(ManagedHostState::Ready).await; + host.advance_to_converged_ready().await; host } diff --git a/crates/machine-controller/tests/integration/power_management.rs b/crates/machine-controller/tests/integration/power_management.rs index 130e4f01de..963b88e517 100644 --- a/crates/machine-controller/tests/integration/power_management.rs +++ b/crates/machine-controller/tests/integration/power_management.rs @@ -67,7 +67,7 @@ impl TestContext { .build() .await .0; - mh.advance_state(ManagedHostState::Ready).await; + mh.advance_to_converged_ready().await; Self { env, mh } } } diff --git a/crates/redfish/src/libredfish/test_support.rs b/crates/redfish/src/libredfish/test_support.rs index dc744e76da..e6a212921e 100644 --- a/crates/redfish/src/libredfish/test_support.rs +++ b/crates/redfish/src/libredfish/test_support.rs @@ -58,6 +58,10 @@ struct RedfishSimState { machine_setup_bios_job_id: Option, is_bios_setup: Option, default_lockdown: Option, + /// Override whether `lockdown_bmc` changes the observed state. `None` + /// preserves the normal successful behavior; `Some(false)` models a BMC + /// accepting the write without applying the requested policy. + lockdown_bmc_applies: Option, job_state_sequence: VecDeque, /// Offset (in seconds) applied to the BMC `DateTime` returned by /// `get_manager`, relative to the controller's `Utc::now()`. Defaults to 0 @@ -309,6 +313,11 @@ impl RedfishSim { } } + /// Control whether `lockdown_bmc` updates the observed lockdown state. + pub fn set_lockdown_bmc_applies(&self, applies: bool) { + self.state.lock().unwrap().lockdown_bmc_applies = Some(applies); + } + /// Set the offset (in seconds) applied to the BMC `DateTime` returned by /// `get_manager`, relative to the controller clock. Use a value larger than /// the time-sync threshold to simulate an out-of-sync BMC clock. @@ -1667,8 +1676,10 @@ impl Redfish for RedfishSimClient { ) -> libredfish::RedfishFuture<'a, Result<(), RedfishError>> { Box::pin(async move { let mut state = self.state.lock().unwrap(); - let host_state = state.hosts.get_mut(&self._host).unwrap(); - host_state.lockdown = target; + if state.lockdown_bmc_applies.unwrap_or(true) { + let host_state = state.hosts.get_mut(&self._host).unwrap(); + host_state.lockdown = target; + } Ok(()) }) } diff --git a/crates/test-harness/src/managed_host.rs b/crates/test-harness/src/managed_host.rs index baa15fb32f..eddc3149d9 100644 --- a/crates/test-harness/src/managed_host.rs +++ b/crates/test-harness/src/managed_host.rs @@ -22,6 +22,7 @@ use carbide_api_core::test_support::Api; use carbide_api_core::test_support::fixture_config::FixtureDefault as _; use carbide_site_explorer::test_support::TestSiteExplorer; use carbide_uuid::machine::MachineId; +use chrono::Utc; use mac_address::MacAddress; use model::expected_machine::{ExpectedMachine, ExpectedMachineData}; use model::hardware_info::HardwareInfo; @@ -90,6 +91,45 @@ impl TestManagedHost { .expect("database transaction should commit"); } + /// Advances this fixture to a stable Ready state after recording its + /// current desired boot interface as verified. + /// + /// This models the invariant established by the machine controller before + /// it returns a host to Ready. Use [`Self::advance_state`] instead when a + /// test intentionally needs a Ready host with pending boot configuration. + pub async fn advance_to_converged_ready(&self) { + let mut txn = self + .api + .database_connection + .begin() + .await + .expect("database transaction should start"); + let machine = self.host.db_machine(&mut txn).await; + let desired_version = machine + .config + .desired_boot_interface + .as_ref() + .expect("test host should have a desired boot interface") + .version; + assert!( + db::machine_desired_boot_interface::mark_verified( + txn.as_mut(), + &self.host.id, + desired_version, + Utc::now(), + ) + .await + .expect("boot-interface verification should be recorded"), + "test host's desired boot interface should still be current" + ); + machine + .advance_state(&mut txn, ManagedHostState::Ready) + .await; + txn.commit() + .await + .expect("database transaction should commit"); + } + pub async fn report_dpu_network_status(&self) { for dpu in &self.dpus { dpu.record_network_status().await; From 5f46d56fcdce658a92834bf1300227a365b0e8f2 Mon Sep 17 00:00:00 2001 From: Jay Zhu Date: Thu, 30 Jul 2026 23:29:36 -0600 Subject: [PATCH 05/15] fix(machine-a-tron): invalidate stale boot actions (#4430) A retrying Machine-a-Tron boot action can remain at the front of the action queue after a power change. This prevents the queued power-off cleanup and power-cycle timer from running. In the observed failure, a stale `InitialDiscoveryRequest` kept retrying while the BIOS job remained scheduled until the test timed out. This change invalidates work tied to the previous boot when the machine powers off or cycles. BMC-scoped setup, BMC DHCP, and existing power-off cleanup remain queued. Pending machine and agent deadlines are cleared. Action classification uses an exhaustive match so future `FsmAction`, `Timer`, or `BmcEvent` variants require an explicit power-change policy. ## Related issues Resolves #4429 ## Type of Change - [ ] **Add** - New feature or capability - [ ] **Change** - Changes in existing functionality - [x] **Fix** - Bug fixes - [ ] **Remove** - Removed features or deprecated functionality - [ ] **Internal** - Internal changes (refactoring, tests, docs, etc.) ## Breaking Changes - [ ] **This PR contains breaking changes** ## Testing - [x] Unit tests added/updated - [ ] Integration tests added/updated - [ ] Manual testing performed - [ ] No testing required (docs, internal refactor, etc.) Signed-off-by: Jay Zhu Signed-off-by: Alex Ball --- .../src/machine_state_machine.rs | 78 +++++++++++++++---- 1 file changed, 61 insertions(+), 17 deletions(-) diff --git a/crates/machine-a-tron/src/machine_state_machine.rs b/crates/machine-a-tron/src/machine_state_machine.rs index 15d7df5764..d7fc81e1a2 100644 --- a/crates/machine-a-tron/src/machine_state_machine.rs +++ b/crates/machine-a-tron/src/machine_state_machine.rs @@ -24,8 +24,9 @@ use std::time::Duration; use bmc_mock::injection::InjectionStore; use bmc_mock::ipmi_sim::IpmiEndpoint; use bmc_mock::{ - BmcCommand, BmcState, BootOptionKind, Callbacks, HostnameQuerying, MachineInfo, MockPowerState, - POWER_CYCLE_DELAY, SetSystemPowerError, SetSystemPowerResult, SystemPowerControl, + BmcCommand, BmcEvent, BmcState, BootOptionKind, Callbacks, HostnameQuerying, MachineInfo, + MockPowerState, POWER_CYCLE_DELAY, SetSystemPowerError, SetSystemPowerResult, + SystemPowerControl, }; use carbide_network::virtualization::build_dual_stack_list; use carbide_uuid::machine::MachineId; @@ -100,17 +101,38 @@ impl DhcpRetryState { } } -/// Abandon a queued machine DHCP request when the machine powers off or cycles -/// so that we can actively drain everything behind the Dhcp(_) action barrier. -/// If any future action needs to run regardless of DHCP status, need to include -/// that action here. -fn abandon_machine_dhcp_on_power_change( +/// Abandon queued work for the current boot when the machine powers off or +/// cycles. A retrying in-band action would otherwise block the power-change +/// cleanup and timer queued behind it. +/// +/// BMC initialization remains valid because the BMC stays powered independently +/// of the machine. Preserve an existing power-off cleanup in case another power +/// change arrives before that action runs. +fn abandon_machine_actions_on_power_change( actions: &mut VecDeque, dhcp_retry: &mut DhcpRetryState, ) { - let is_machine_dhcp = |action: &FsmAction| matches!(action, FsmAction::Dhcp(DhcpType::Machine)); - if actions.iter().any(is_machine_dhcp) { - actions.retain(|action| !is_machine_dhcp(action)); + let abandoned_machine_dhcp = actions + .iter() + .any(|action| matches!(action, FsmAction::Dhcp(DhcpType::Machine))); + + actions.retain(|action| match action { + FsmAction::SetupBmc | FsmAction::Dhcp(DhcpType::Bmc) | FsmAction::CleanupOnPowerOff => true, + FsmAction::SetTimer( + Timer::PowerCycle + | Timer::MachineOn + | Timer::ScoutAgentControlPoll + | Timer::DpuAgentControlPoll, + ) + | FsmAction::Dhcp(DhcpType::Machine) + | FsmAction::PxeBootRequest + | FsmAction::InitialDiscoveryRequest(_) + | FsmAction::AgentControlRequest(_) + | FsmAction::DpuAgentNetworkObservation + | FsmAction::BmcEvent(BmcEvent::PowerOn | BmcEvent::BootCompleted) => false, + }); + + if abandoned_machine_dhcp { dhcp_retry.reset(); } } @@ -605,8 +627,13 @@ impl MachineStateMachine { fn fsm_event(&mut self, event: Event) { if matches!(event, Event::PowerCycle | Event::PowerOff) { - abandon_machine_dhcp_on_power_change(&mut self.actions, &mut self.dhcp_retry); + abandon_machine_actions_on_power_change(&mut self.actions, &mut self.dhcp_retry); + + self.machine_on_deadline = None; + self.power_cycle_deadline = None; + self.agent_polling_deadline = None; } + let old_state = self.fsm; let (new_state, actions) = self.fsm.event(event); tracing::info!(previous_state = ?old_state, ?event, next_state = ?new_state, ?actions, "machine FSM step"); @@ -1522,7 +1549,7 @@ mod tests { } #[test] - fn power_change_abandons_a_queued_machine_dhcp() { + fn power_change_abandons_queued_machine_actions() { let queued = |actions: &[FsmAction]| VecDeque::from(actions.to_vec()); check_values( @@ -1537,11 +1564,21 @@ mod tests { }, }, Check { - scenario: "unrelated queued actions are preserved", + scenario: "in-band work from the previous boot is abandoned", input: ( queued(&[ FsmAction::SetupBmc, FsmAction::Dhcp(DhcpType::Machine), + FsmAction::SetTimer(Timer::PowerCycle), + FsmAction::SetTimer(Timer::MachineOn), + FsmAction::SetTimer(Timer::ScoutAgentControlPoll), + FsmAction::SetTimer(Timer::DpuAgentControlPoll), + FsmAction::PxeBootRequest, + FsmAction::InitialDiscoveryRequest(OsImage::Scout), + FsmAction::AgentControlRequest(OsImage::Scout), + FsmAction::DpuAgentNetworkObservation, + FsmAction::BmcEvent(BmcEvent::PowerOn), + FsmAction::BmcEvent(BmcEvent::BootCompleted), FsmAction::CleanupOnPowerOff, ]), 1, @@ -1565,10 +1602,16 @@ mod tests { }, }, Check { - scenario: "power change without a queued DHCP changes nothing", - input: (queued(&[FsmAction::PxeBootRequest]), 0), + scenario: "power change without queued in-band work changes nothing", + input: ( + queued(&[FsmAction::SetupBmc, FsmAction::CleanupOnPowerOff]), + 0, + ), expect: PowerChangeOutcome { - remaining_actions: vec![format!("{:?}", FsmAction::PxeBootRequest)], + remaining_actions: vec![ + format!("{:?}", FsmAction::SetupBmc), + format!("{:?}", FsmAction::CleanupOnPowerOff), + ], attempt: 0, backoff_pending: false, }, @@ -1580,7 +1623,8 @@ mod tests { for _ in 0..failures { retry.schedule_next(now, 0); } - abandon_machine_dhcp_on_power_change(&mut actions, &mut retry); + + abandon_machine_actions_on_power_change(&mut actions, &mut retry); outcome(&actions, &retry) }, ); From f544fb29b0b655ca169d22067b07241132bc89ef Mon Sep 17 00:00:00 2001 From: Chet Nichols III Date: Thu, 30 Jul 2026 23:06:11 -0700 Subject: [PATCH 06/15] fix(migration): put the published loopback migration timestamp back (#4432) PR #4389 and PR #4428 independently fixed the same duplicate migration version from opposite sides. The loopback preservation migration first reached `main` in #3913 as `20260722120000`, while the BMC suppression migration came later. #4428 correctly moved the newer BMC migration to `20260722120001`, but #4389 had already renamed the older loopback migration to that same version on an earlier base. Since #4428 merged first, #4389 left `main` with two `20260722120001` migrations and removed the identity existing databases may already have in `_sqlx_migrations`. So, restore `20260722120000_preserve_machine_ipv6_loopback.sql` and leave `20260722120001_bmc_suppressions.sql` where it is. The loopback migration SQL is byte-for-byte identical to #3913, existing database histories keep matching the published version and checksum, and SQLx sees each migration version exactly once again. ## Related issues None -- urgent post-merge repair for #4389 and #4428. ## Type of Change - [ ] **Add** - New feature or capability - [ ] **Change** - Changes in existing functionality - [x] **Fix** - Bug fixes - [ ] **Remove** - Removed features or deprecated functionality - [ ] **Internal** - Internal changes (refactoring, tests, docs, etc.) ## Breaking Changes - [ ] **This PR contains breaking changes** ## Testing - [ ] Unit tests added/updated - [ ] Integration tests added/updated - [x] Manual testing performed - [ ] No testing required (docs, internal refactor, etc.) ## Additional Notes - `cargo test -p carbide-api-db migrations::tests --lib` (4 passed) - `cargo make format-nightly` - `cargo make clippy` - `cargo make carbide-lints` - `git diff --check` - Restored migration blob matches the original #3913 blob exactly (`eaff208e4ea6d3956bdb158a84c45ee6c1362ecf`) - This restores the migration identities intended by #4428. A database first initialized during either brief duplicate-version window may have recorded the wrong checksum at `20260722120000` or `20260722120001`; inspect its schema and migration history and perform site-specific repair before retrying rather than blindly replaying these non-idempotent migrations Signed-off-by: Chet Nichols III Signed-off-by: Alex Ball --- ...back.sql => 20260722120000_preserve_machine_ipv6_loopback.sql} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename crates/api-db/migrations/{20260722120001_preserve_machine_ipv6_loopback.sql => 20260722120000_preserve_machine_ipv6_loopback.sql} (100%) diff --git a/crates/api-db/migrations/20260722120001_preserve_machine_ipv6_loopback.sql b/crates/api-db/migrations/20260722120000_preserve_machine_ipv6_loopback.sql similarity index 100% rename from crates/api-db/migrations/20260722120001_preserve_machine_ipv6_loopback.sql rename to crates/api-db/migrations/20260722120000_preserve_machine_ipv6_loopback.sql From c33ace59beaf0f21beb0b425a2a55964b20a64f1 Mon Sep 17 00:00:00 2001 From: Chet Nichols III Date: Fri, 31 Jul 2026 07:24:08 -0700 Subject: [PATCH 07/15] feat(ci): add migration-police for migration filenames (#4433) This enforces that migration files have their timestamp fully populated, and not end in `0000`. Core migrations have more recently been using `...120000` placeholder timestamps, which means two independently authored PRs can both look fine until their versions collide on `main` -- exactly what #4432 is repairing now. The duplicate-version test can tell us once both migrations are together, but it cannot prevent that collision while the PRs are still independent. So, this adds a separate `migration-police` job that looks only at new files under `crates/api-db/migrations/` and rejects timestamps whose minute and second fields are both `00`. Existing migrations stay accepted, and failures name the migration plus the required `YYYYMMDDhhmmss_description.sql` format in the job log. `STYLE_GUIDE.md` and `AGENTS.md` now point contributors at the same fully populated timestamp rule. ## Related issues This supports https://github.com/NVIDIA/infra-controller/issues/4431 ## Type of Change - [x] **Add** - New feature or capability - [ ] **Change** - Changes in existing functionality - [ ] **Fix** - Bug fixes - [ ] **Remove** - Removed features or deprecated functionality - [ ] **Internal** - Internal changes (refactoring, tests, docs, etc.) ## Breaking Changes - [ ] **This PR contains breaking changes** ## Testing - [ ] Unit tests added/updated - [ ] Integration tests added/updated - [x] Manual testing performed - [ ] No testing required (docs, internal refactor, etc.) Focused checks covered a valid full timestamp, a rejected `0000` placeholder, an accepted zero-seconds timestamp with a populated minute, historical migrations against `origin/main`, an invalid base revision, shell syntax, workflow YAML parsing, and `git diff --check`. The required local gates also passed: - `cargo make format-nightly` - `cargo make clippy` - `cargo make carbide-lints` ## Additional Notes `cargo test -p carbide-api-db migration_versions_are_unique --lib` currently fails on the pre-existing duplicate `20260722120001` migration version. #4432 repairs that collision and reports the migration test passing. Signed-off-by: Chet Nichols III Signed-off-by: Alex Ball --- .github/workflows/ci.yaml | 20 ++++++++ AGENTS.md | 5 ++ STYLE_GUIDE.md | 8 +++ scripts/check-migration-filenames.sh | 74 ++++++++++++++++++++++++++++ 4 files changed, 107 insertions(+) create mode 100755 scripts/check-migration-filenames.sh diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 88760ceb9d..bf8cd2e33f 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -1362,6 +1362,25 @@ jobs: - name: Run Protolint run: protolint lint -config_path=.protolint.yaml crates/rpc/proto/ + migration-police: + permissions: + contents: read + needs: + - changes + if: ${{ contains(github.ref, 'pull-request/') }} + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + persist-credentials: false + fetch-depth: 0 + + - name: Check new migration timestamps + run: | + git fetch --no-tags origin main:refs/remotes/origin/main + bash scripts/check-migration-filenames.sh --base origin/main + proto-breaking-changes: name: Proto Breaking Changes Check runs-on: ubuntu-latest @@ -2059,6 +2078,7 @@ jobs: - build-release-artifacts-arm-host - security-secret-scan - lint-police + - migration-police - check-rest-core-proto-sync - build-machine-a-tron - build-mat-k8s-controller diff --git a/AGENTS.md b/AGENTS.md index ecf2db1aa0..4c7517f8a5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -169,6 +169,11 @@ verification expectations. See [`STYLE_GUIDE.md`](STYLE_GUIDE.md) for detailed Rust coding conventions. Make sure to review it to ensure changes meet the expected style of the codebase. +Name new Core database migrations with the fully populated +`YYYYMMDDhhmmss_description.sql` format described in +[`STYLE_GUIDE.md`](STYLE_GUIDE.md#database-migrations). The `migration-police` +CI job checks only newly added migrations, so existing filenames remain accepted. + ### Documentation Give every fenced code block a language identifier. Use `bash` or `sh` for diff --git a/STYLE_GUIDE.md b/STYLE_GUIDE.md index c5ce25834c..a5be6305a9 100644 --- a/STYLE_GUIDE.md +++ b/STYLE_GUIDE.md @@ -342,6 +342,14 @@ your interface `async` just so you can use the tokio Mutex. That way callers can async themselves. Async work should generally be traceable to some I/O or timer that needs to be used, otherwise code should typically be synchronous. +## Database migrations + +Name new Core database migration files with a fully populated 14-digit timestamp: +`YYYYMMDDhhmmss_description.sql`. Use the actual hour, minute, and second values instead of a +trailing `0000` minute-and-second placeholder so independently authored migrations are less likely +to collide. Existing migration filenames remain unchanged, and migrations already on `main` are +immutable. + ## Database transactions Transactions should be used to group write operations together such that they can be rolled back on failure. But do diff --git a/scripts/check-migration-filenames.sh b/scripts/check-migration-filenames.sh new file mode 100755 index 0000000000..ee897e5c4c --- /dev/null +++ b/scripts/check-migration-filenames.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +set -euo pipefail + +usage() { + echo "Usage: check-migration-filenames.sh --base REVISION" + echo " check-migration-filenames.sh MIGRATION_FILE..." +} + +escape_workflow_data() { + local value="$1" + + value="${value//'%'/'%25'}" + value="${value//$'\r'/'%0D'}" + value="${value//$'\n'/'%0A'}" + printf '%s' "${value}" +} + +migration_files=() + +if [[ "${1:-}" == "-h" || "${1:-}" == "--help" ]]; then + usage + exit 0 +elif [[ "${1:-}" == "--base" ]]; then + if (( $# != 2 )); then + usage >&2 + exit 2 + fi + + base_revision="$2" + migration_list="$(mktemp)" + trap 'rm -f "${migration_list}"' EXIT + + git diff --no-renames --diff-filter=A --name-only -z \ + "${base_revision}...HEAD" \ + -- ':(top,glob)crates/api-db/migrations/*.sql' >"${migration_list}" + + while IFS= read -r -d '' migration_file; do + migration_files+=("${migration_file}") + done <"${migration_list}" +elif (( $# > 0 )); then + migration_files=("$@") +else + usage >&2 + exit 2 +fi + +failed=0 + +for migration_file in "${migration_files[@]}"; do + filename="${migration_file##*/}" + timestamp="${filename%%_*}" + + if [[ ! "${filename}" =~ ^[0-9]{14}_.+\.sql$ ]]; then + message="New Core database migration ${migration_file} must use the YYYYMMDDhhmmss_description.sql filename format." + printf '::error title=Invalid migration filename::%s\n' \ + "$(escape_workflow_data "${message}")" + failed=1 + elif [[ "${timestamp}" =~ ^[0-9]{10}0000$ ]]; then + message="New Core database migration ${migration_file} must use a fully populated YYYYMMDDhhmmss timestamp; replace the trailing 0000 minute/second placeholder." + printf '::error title=Placeholder migration timestamp::%s\n' \ + "$(escape_workflow_data "${message}")" + failed=1 + fi +done + +if (( failed )); then + echo "migration-police rejected one or more new database migrations." + exit 1 +fi + +printf 'migration-police checked %d new database migration(s).\n' "${#migration_files[@]}" From f51359b45ecbf230bf6e6056e23141b29763826c Mon Sep 17 00:00:00 2001 From: Leah Itagaki Date: Fri, 31 Jul 2026 12:11:14 -0400 Subject: [PATCH 08/15] fix(api): Address misclassification of NVOS data interface IPs as BMC IPs (#4395) Previously our search IP address logic mistakenly categorized all statically allocated machine interface addresses as static BMC IPs. Now it looks to the interface type as well as the allocation type to categorize the interface. Additionally, if the response to an IP address search is associated with a switch, the CLI now prints the switch. ## Related issues Internal issue ## Type of Change - [x] **Change** - Changes in existing functionality - [x] **Fix** - Bug fixes ## Breaking Changes - [ ] **This PR contains breaking changes** ## Testing - [x] Unit tests added/updated - [x] Manual testing performed ## Additional Notes --------- Signed-off-by: Leah Itagaki Signed-off-by: Alex Ball --- crates/api-core/src/handlers/finder.rs | 111 +++++++++++-- crates/api-core/src/tests/finder.rs | 147 ++++++++++++++++-- .../api-db/src/machine_interface_address.rs | 19 ++- 3 files changed, 242 insertions(+), 35 deletions(-) diff --git a/crates/api-core/src/handlers/finder.rs b/crates/api-core/src/handlers/finder.rs index de2bd66923..f7b9fcb899 100644 --- a/crates/api-core/src/handlers/finder.rs +++ b/crates/api-core/src/handlers/finder.rs @@ -29,6 +29,7 @@ use carbide_uuid::network::NetworkSegmentId; use carbide_uuid::vpc::VpcId; use db::{DatabaseError, ObjectColumnFilter, instance, network_segment, vpc}; use model::allocation_type::AllocationType; +use model::machine_interface::InterfaceType; use model::network_segment::NetworkSegmentSearchConfig; use model::resource_pool::ResourcePoolEntryState; use model::route_server::RouteServerSourceType; @@ -36,19 +37,24 @@ use model::route_server::RouteServerSourceType; use crate::CarbideError; use crate::api::Api; -/// Returns true when this machine-interface address should be labeled as operator/static BMC -/// (`IpTypeStaticBmcIp`): either explicitly static allocation, or an address on the synthetic -/// `static-assignments` segment used for external IPs outside Carbide-managed prefixes. -fn machine_interface_address_is_operator_static( +/// Returns true when this machine-interface address should be labeled as an operator/static BMC +/// (`IpTypeStaticBmcIp`). +/// +/// Static allocation metadata and the synthetic `static-assignments` segment do not identify the +/// endpoint role, so either condition must be paired with `InterfaceType::Bmc`. +fn machine_interface_address_is_static_bmc( segment_name: &str, allocation_type: AllocationType, + interface_type: InterfaceType, ) -> bool { - allocation_type == AllocationType::Static - || segment_name == network_segment::STATIC_ASSIGNMENTS_SEGMENT_NAME + interface_type == InterfaceType::Bmc + && (allocation_type == AllocationType::Static + || segment_name == network_segment::STATIC_ASSIGNMENTS_SEGMENT_NAME) } /// Resolves an IP to zero or more typed matches (BMC, instance, static BMC, etc.). Static BMC -/// classification for `machine_interface_addresses` uses [`machine_interface_address_is_operator_static`]. +/// classification for `machine_interface_addresses` uses +/// [`machine_interface_address_is_static_bmc`]. pub(crate) async fn find_ip_address( api: &Api, request: tonic::Request, @@ -288,14 +294,17 @@ async fn search( }) } - // machine_interface_addresses: classify operator/static BMC as StaticBmcIp (see - // machine_interface_address_is_operator_static). + // machine_interface_addresses: classify operator/static BMC as StaticBmcIp while + // retaining static Data addresses as MachineAddress. MachineAddresses => { let out = db::machine_interface_address::find_by_address(db, addr).await?; match out { Some(e) => { - let is_static_bmc = - machine_interface_address_is_operator_static(&e.name, e.allocation_type); + let is_static_bmc = machine_interface_address_is_static_bmc( + &e.name, + e.allocation_type, + e.interface_type, + ); let (ip_type, type_label) = if is_static_bmc { (rpc::IpType::StaticBmcIp, "static BMC IP") @@ -303,12 +312,16 @@ async fn search( (rpc::IpType::MachineAddress, "machine address") }; - let message = match e.machine_id.as_ref() { - Some(machine_id) => format!( + let message = match (e.machine_id.as_ref(), e.switch_id.as_ref()) { + (Some(machine_id), _) => format!( "{ip} is a {type_label} on machine {} (interface {}) on network segment {} of type {}", machine_id, e.id, e.name, e.network_segment_type, ), - None => format!( + (None, Some(switch_id)) => format!( + "{ip} is a {type_label} on switch {} (interface {}) on network segment {} of type {}", + switch_id, e.id, e.name, e.network_segment_type, + ), + (None, None) => format!( "{ip} is a {type_label} on interface {} on network segment {} of type {}. It is not attached to a machine.", e.id, e.name, e.network_segment_type, ), @@ -333,9 +346,10 @@ async fn search( let is_static = db::machine_interface_address::find_by_address(db, addr) .await? .is_some_and(|row| { - machine_interface_address_is_operator_static( + machine_interface_address_is_static_bmc( &row.name, row.allocation_type, + row.interface_type, ) }); @@ -359,6 +373,7 @@ async fn search( None => None, } } + ExploredEndpoint => { let out = db::explored_endpoints::find_by_ips(db, vec![addr]).await?; out.first().map(|ee| rpc::IpAddressMatch { @@ -566,3 +581,69 @@ async fn by_mac( Ok(None) } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn static_bmc_classification_requires_bmc_interface_type() { + let static_assignments = network_segment::STATIC_ASSIGNMENTS_SEGMENT_NAME; + let cases = [ + ( + "static BMC allocation", + "underlay", + AllocationType::Static, + InterfaceType::Bmc, + true, + ), + ( + "BMC on static assignments", + static_assignments, + AllocationType::Dhcp, + InterfaceType::Bmc, + true, + ), + ( + "dynamic BMC", + "underlay", + AllocationType::Dhcp, + InterfaceType::Bmc, + false, + ), + ( + "static Data allocation", + "underlay", + AllocationType::Static, + InterfaceType::Data, + false, + ), + ( + "Data on static assignments", + static_assignments, + AllocationType::Dhcp, + InterfaceType::Data, + false, + ), + ( + "dynamic Data", + "underlay", + AllocationType::Dhcp, + InterfaceType::Data, + false, + ), + ]; + + for (name, segment_name, allocation_type, interface_type, expected) in cases { + assert_eq!( + machine_interface_address_is_static_bmc( + segment_name, + allocation_type, + interface_type, + ), + expected, + "{name}", + ); + } + } +} diff --git a/crates/api-core/src/tests/finder.rs b/crates/api-core/src/tests/finder.rs index c64cf4bb50..7e6c43e6db 100644 --- a/crates/api-core/src/tests/finder.rs +++ b/crates/api-core/src/tests/finder.rs @@ -147,10 +147,34 @@ async fn test_inner(ip: &str, ip_type: IpType, env: &TestEnv, caller: &str) { assert!(!res.matches.is_empty(), "{caller} not found"); // In integration testing DHCP relay is in a network segment, // so we get multiple matches. Wouldn't happen in live. - for m in res.matches { - if m.ip_type == ip_type as i32 { - return; // success + if let Some(ip_match) = res + .matches + .into_iter() + .find(|ip_match| ip_match.ip_type == ip_type as i32) + { + match ip_type { + IpType::MachineAddress => { + assert!( + ip_match.message.contains("machine address"), + "{caller} used incorrect Data-interface wording: {}", + ip_match.message, + ); + assert!( + !ip_match.message.contains("BMC"), + "{caller} described a Data interface as BMC: {}", + ip_match.message, + ); + } + IpType::StaticBmcIp => { + assert!( + ip_match.message.contains("static BMC IP"), + "{caller} used incorrect BMC-interface wording: {}", + ip_match.message, + ); + } + _ => {} } + return; } panic!("{caller} did not have correct IPType"); } @@ -423,7 +447,7 @@ async fn test_identify_serial(db_pool: sqlx::PgPool) -> Result<(), eyre::Report> } /// `FindIpAddress` returns `IpTypeStaticBmcIp` when the address is a static/operator BMC -/// allocation (`AllocationType::Static` after `preallocate_machine_interface` on underlay). +/// allocation. #[crate::sqlx_test] async fn test_static_bmc_ip_finder(db_pool: sqlx::PgPool) -> Result<(), eyre::Report> { use std::net::IpAddr; @@ -434,9 +458,14 @@ async fn test_static_bmc_ip_finder(db_pool: sqlx::PgPool) -> Result<(), eyre::Re let bmc_mac = "AA:BB:CC:DD:EE:99".parse().unwrap(); let mut txn = db_pool.begin().await.unwrap(); - db::machine_interface::preallocate_machine_interface(txn.as_mut(), bmc_mac, static_ip, None) - .await - .expect("preallocate static BMC interface"); + db::machine_interface::preallocate_bmc_machine_interface( + txn.as_mut(), + bmc_mac, + static_ip, + None, + ) + .await + .expect("preallocate static BMC interface"); txn.commit().await.unwrap(); // Query the IP via finder @@ -452,15 +481,107 @@ async fn test_static_bmc_ip_finder(db_pool: sqlx::PgPool) -> Result<(), eyre::Re assert!(!res.matches.is_empty(), "Should find at least one match"); - // Verify it's classified as StaticBmcIp - let has_static_bmc_ip = res + let static_bmc_match = res .matches - .iter() - .any(|m| m.ip_type == IpType::StaticBmcIp as i32); + .into_iter() + .find(|ip_match| ip_match.ip_type == IpType::StaticBmcIp as i32) + .expect("static BMC IP should be classified as IpTypeStaticBmcIp"); + assert!( + static_bmc_match.message.contains("static BMC IP"), + "static BMC wording should match the interface type: {}", + static_bmc_match.message, + ); + + Ok(()) +} + +/// A static NVOS/Data address remains a MachineAddress even when it lives on the synthetic +/// `static-assignments` segment. When the interface is associated with a switch, finder output +/// identifies that switch. +#[crate::sqlx_test] +async fn test_static_data_ip_finder(db_pool: sqlx::PgPool) -> Result<(), eyre::Report> { + use std::net::IpAddr; + + use carbide_uuid::switch::SwitchId; + use model::machine_interface_address::MachineInterfaceAssociation; + use model::switch::{NewSwitch, SwitchConfig}; + + let env = create_test_env(db_pool.clone()).await; + let static_ip: IpAddr = "10.86.241.49".parse().unwrap(); + let nvos_mac = "AA:BB:CC:DD:EE:98".parse().unwrap(); + let switch_id = SwitchId::from(uuid::Uuid::new_v4()); + + let mut txn = db_pool.begin().await.unwrap(); + db::machine_interface::preallocate_machine_interface(txn.as_mut(), nvos_mac, static_ip, None) + .await + .expect("preallocate static NVOS Data interface"); + let interface = db::machine_interface::find_by_mac_address(txn.as_mut(), nvos_mac) + .await + .expect("find static NVOS Data interface") + .into_iter() + .next() + .expect("static NVOS Data interface should exist"); + db::switch::create( + txn.as_mut(), + &NewSwitch { + id: switch_id, + config: SwitchConfig { + name: "gb-nvl-136-switch02".to_string(), + enable_nmxc: false, + fabric_manager_config: None, + }, + bmc_mac_address: None, + metadata: None, + rack_id: None, + slot_number: None, + tray_index: None, + }, + ) + .await + .expect("create associated switch"); + db::machine_interface::associate_interface_with_machine( + &interface.id, + MachineInterfaceAssociation::Switch(switch_id), + txn.as_mut(), + ) + .await + .expect("associate NVOS Data interface with switch"); + txn.commit().await.unwrap(); + + let req = rpc::forge::FindIpAddressRequest { + ip: static_ip.to_string(), + }; + let res = env + .api + .find_ip_address(tonic::Request::new(req)) + .await + .expect("find_ip_address should succeed") + .into_inner(); assert!( - has_static_bmc_ip, - "Static IP should be classified as IpTypeStaticBmcIp" + res.matches + .iter() + .all(|ip_match| ip_match.ip_type != IpType::StaticBmcIp as i32), + "static NVOS Data IP must never be classified as StaticBmcIp: {:?}", + res.matches, + ); + let machine_address_match = res + .matches + .into_iter() + .find(|ip_match| ip_match.ip_type == IpType::MachineAddress as i32) + .expect("static NVOS Data IP should remain a MachineAddress"); + assert!( + machine_address_match.owner_id.is_none(), + "static NVOS Data address must not expose a switch ID as a machine owner", + ); + assert!( + machine_address_match.message.contains("machine address") + && machine_address_match + .message + .contains(&switch_id.to_string()) + && !machine_address_match.message.contains("BMC"), + "static NVOS Data wording should identify the switch without calling it BMC: {}", + machine_address_match.message, ); Ok(()) diff --git a/crates/api-db/src/machine_interface_address.rs b/crates/api-db/src/machine_interface_address.rs index 23f50558f1..48cac96373 100644 --- a/crates/api-db/src/machine_interface_address.rs +++ b/crates/api-db/src/machine_interface_address.rs @@ -19,8 +19,10 @@ use std::net::IpAddr; use carbide_network::ip::{IdentifyAddressFamily, IpAddressFamily}; use carbide_uuid::machine::{MachineId, MachineInterfaceId}; use carbide_uuid::network::NetworkSegmentId; +use carbide_uuid::switch::SwitchId; use mac_address::MacAddress; use model::allocation_type::{AllocationType, AssignStaticResult}; +use model::machine_interface::InterfaceType; use model::network_segment::NetworkSegmentType; use sqlx::{FromRow, PgConnection}; @@ -65,16 +67,17 @@ pub async fn find_ipv4_for_interface( .map_err(|e| DatabaseError::query(query, e)) } -/// Looks up which machine interface owns an IP, with segment metadata and **allocation type**. +/// Looks up which machine interface owns an IP, with association, segment, role, and allocation +/// metadata. /// -/// `allocation_type` is used by the IP finder to classify operator static assignments -/// (`AllocationType::Static` or addresses on the `static-assignments` segment) as -/// `IpTypeStaticBmcIp` where appropriate. +/// The IP finder uses `interface_type` together with `allocation_type` and segment metadata to +/// distinguish static BMC addresses from static Data addresses. pub async fn find_by_address( txn: impl DbReader<'_>, address: IpAddr, ) -> Result, DatabaseError> { - let query = "SELECT mi.id, mi.machine_id, ns.name, ns.network_segment_type, mia.allocation_type + let query = "SELECT mi.id, mi.machine_id, mi.switch_id, mi.interface_type, + ns.name, ns.network_segment_type, mia.allocation_type FROM machine_interface_addresses mia INNER JOIN machine_interfaces mi ON mi.id = mia.interface_id INNER JOIN network_segments ns ON ns.id = mi.segment_id @@ -291,12 +294,14 @@ pub async fn has_address_for_family( .map_err(|e| DatabaseError::query(query, e)) } -/// Row shape for [`find_by_address`]: interface identity, owning segment, and how the address was -/// assigned (DHCP vs static / operator-configured). +/// Row shape for [`find_by_address`]: interface identity, association, role, owning segment, and +/// how the address was assigned (DHCP vs static / operator-configured). #[derive(Debug, FromRow)] pub struct MachineInterfaceSearchResult { pub id: MachineInterfaceId, pub machine_id: Option, + pub switch_id: Option, + pub interface_type: InterfaceType, pub name: String, pub network_segment_type: NetworkSegmentType, pub allocation_type: AllocationType, From d9041dea3daecf54ba331938b7b3359482d4febd Mon Sep 17 00:00:00 2001 From: Leah Itagaki Date: Fri, 31 Jul 2026 12:31:27 -0400 Subject: [PATCH 09/15] feat(rest-api): Add SKU management endpoints (#3633) Provider Admins can create, partially update, and delete a SKU for a selected Site. The REST API sends the mutation through Temporal and the Site Agent to that Site's real NICo Core service. POST and PATCH return the Core-backed SKU shape; PATCH preserves fields omitted from the request. This adds REST API support for managing SKUs. Resolves #2810 ### What this PR does Adds Site-scoped REST endpoints to create, partially update, and delete Core-backed SKUs. Mutations run through Temporal and the Site Agent to the selected Site's NICo Core. The writable schema includes SKU v5 storage size bounds and PCI patterns; ignored legacy write fields such as `capacityMb` are rejected. --------- Signed-off-by: Leah Itagaki Signed-off-by: Alex Ball --- rest-api/api/pkg/api/handler/instancetype.go | 2 +- rest-api/api/pkg/api/handler/sku.go | 404 ++++++++++ rest-api/api/pkg/api/handler/sku_test.go | 735 ++++++++++++++++++ .../api/pkg/api/model/expectedmachine_test.go | 8 +- rest-api/api/pkg/api/model/sku.go | 414 +++++++--- rest-api/api/pkg/api/model/sku_test.go | 353 ++++++++- rest-api/api/pkg/api/routes.go | 15 + rest-api/api/pkg/api/routes_test.go | 9 +- rest-api/db/pkg/db/model/sku.go | 54 +- rest-api/db/pkg/db/model/sku_test.go | 79 +- .../20260729020000_sku_description.go | 51 ++ rest-api/docs/index.html | 468 ++++++++++- rest-api/openapi/spec.yaml | 250 +++++- rest-api/sdk/standard/api_sku.go | 424 ++++++++++ rest-api/sdk/standard/model_sku.go | 74 ++ rest-api/sdk/standard/model_sku_chassis.go | 37 + .../sdk/standard/model_sku_create_request.go | 302 +++++++ .../sdk/standard/model_sku_ethernet_device.go | 37 + .../standard/model_sku_infiniband_device.go | 37 + rest-api/sdk/standard/model_sku_storage.go | 139 +++- .../sdk/standard/model_sku_update_request.go | 201 +++++ rest-api/workflow/pkg/activity/sku/sku.go | 11 +- .../workflow/pkg/activity/sku/sku_test.go | 17 +- 23 files changed, 3945 insertions(+), 176 deletions(-) create mode 100644 rest-api/db/pkg/migrations/20260729020000_sku_description.go create mode 100644 rest-api/sdk/standard/model_sku_create_request.go create mode 100644 rest-api/sdk/standard/model_sku_update_request.go diff --git a/rest-api/api/pkg/api/handler/instancetype.go b/rest-api/api/pkg/api/handler/instancetype.go index 08d51abed4..339a014e56 100644 --- a/rest-api/api/pkg/api/handler/instancetype.go +++ b/rest-api/api/pkg/api/handler/instancetype.go @@ -96,7 +96,7 @@ func (cith CreateInstanceTypeHandler) Handle(c echo.Context) error { // Validate role, only Provider Admins are allowed to create Instance Types ok = auth.ValidateUserRoles(dbUser, org, nil, auth.ProviderAdminRole) if !ok { - logger.Warn().Msg("user does not have Tenant Admin role, access denied") + logger.Warn().Msg("user does not have Provider Admin role, access denied") return cutil.NewAPIErrorResponse(c, http.StatusForbidden, "User does not have Provider Admin role with org", nil) } diff --git a/rest-api/api/pkg/api/handler/sku.go b/rest-api/api/pkg/api/handler/sku.go index 4666c84550..c5b441ab91 100644 --- a/rest-api/api/pkg/api/handler/sku.go +++ b/rest-api/api/pkg/api/handler/sku.go @@ -8,15 +8,18 @@ import ( "errors" "fmt" "net/http" + "reflect" "github.com/NVIDIA/infra-controller/rest-api/api/internal/config" "github.com/NVIDIA/infra-controller/rest-api/api/pkg/api/handler/util/common" "github.com/NVIDIA/infra-controller/rest-api/api/pkg/api/model" "github.com/NVIDIA/infra-controller/rest-api/api/pkg/api/pagination" + sc "github.com/NVIDIA/infra-controller/rest-api/api/pkg/client/site" cutil "github.com/NVIDIA/infra-controller/rest-api/common/pkg/util" cdb "github.com/NVIDIA/infra-controller/rest-api/db/pkg/db" cdbm "github.com/NVIDIA/infra-controller/rest-api/db/pkg/db/model" "github.com/NVIDIA/infra-controller/rest-api/db/pkg/db/paginator" + corev1 "github.com/NVIDIA/infra-controller/rest-api/proto/core/gen/v1" "github.com/google/uuid" "github.com/labstack/echo/v4" "go.opentelemetry.io/otel/attribute" @@ -277,3 +280,404 @@ func (gsh GetSkuHandler) Handle(c echo.Context) error { logger.Info().Msg("finishing API handler") return c.JSON(http.StatusOK, apiSku) } + +// CreateSkuHandler creates one SKU on a Site's Core service. +type CreateSkuHandler struct { + dbSession *cdb.Session + scp *sc.ClientPool + tracerSpan *cutil.TracerSpan +} + +// NewCreateSkuHandler initializes and returns a new CreateSkuHandler. +func NewCreateSkuHandler(dbSession *cdb.Session, scp *sc.ClientPool) CreateSkuHandler { + return CreateSkuHandler{ + dbSession: dbSession, + scp: scp, + tracerSpan: cutil.NewTracerSpan(), + } +} + +// Handle godoc +// @Summary Create SKU +// @Description Create a SKU on the selected Site's Core service. +// @Tags SKU +// @Accept json +// @Produce json +// @Security ApiKeyAuth +// @Param org path string true "Name of NGC organization" +// @Param request body model.APISkuCreateRequest true "SKU create request" +// @Success 201 {object} model.APISku +// @Router /v2/org/{org}/nico/sku [post] +func (csh CreateSkuHandler) Handle(c echo.Context) error { + org, dbUser, ctx, logger, handlerSpan := common.SetupHandler("SKU", "Create", c, csh.tracerSpan) + if handlerSpan != nil { + defer handlerSpan.End() + } + + // Validate request data + // Bind request data to API model + apiReq := model.APISkuCreateRequest{} + err := c.Bind(&apiReq) + if err != nil { + logger.Warn().Err(err).Msg("error binding request data into API model") + return cutil.NewAPIErrorResponse(c, http.StatusBadRequest, "Failed to parse request data, potentially invalid structure", nil) + } + + // Validate request attributes + verr := apiReq.Validate() + if verr != nil { + logger.Warn().Err(verr).Msg("error validating SKU create request data") + return cutil.NewAPIErrorResponse(c, http.StatusBadRequest, "Error validating SKU create request data", verr) + } + + stc, siteID, apiErr := common.AuthorizeProviderSiteForCore(common.AuthorizeProviderSiteForCoreInput{ + Ctx: ctx, + Logger: logger, + DBSession: csh.dbSession, + SCP: csh.scp, + Org: org, + User: dbUser, + SiteID: apiReq.SiteID, + }) + if apiErr != nil { + return cutil.NewAPIErrorResponse(c, apiErr.Code, apiErr.Message, apiErr.Data) + } + + logger.Info().Str("skuID", apiReq.ID).Str("siteID", siteID).Msg("creating SKU via Core proxy") + var ids corev1.SkuIdList + apiErr = common.ExecuteCoreGRPC(ctx, stc, corev1.Forge_CreateSku_FullMethodName, apiReq.ToProto(), &ids, siteID) + if apiErr != nil { + logAPIError(logger, apiErr, "failed to create SKU via Core proxy") + return cutil.NewAPIErrorResponse(c, apiErr.Code, apiErr.Message, nil) + } + if len(ids.Ids) != 1 { + return cutil.NewAPIErrorResponse(c, http.StatusInternalServerError, "Core returned an unexpected SKU create response", nil) + } + + var response corev1.SkuList + apiErr = common.ExecuteCoreGRPC( + ctx, + stc, + corev1.Forge_FindSkusByIds_FullMethodName, + &corev1.SkusByIdsRequest{Ids: []string{ids.Ids[0]}}, + &response, + siteID, + ) + if apiErr == nil && len(response.Skus) != 1 { + apiErr = cutil.NewAPIError(http.StatusNotFound, "Could not find SKU with the specified ID", nil) + } + + var skuToPersist *corev1.Sku + if apiErr != nil { + logger.Warn().Err(apiErr).Str("skuID", ids.Ids[0]).Str("siteID", siteID). + Msg("SKU created but post-create retrieval failed; returning request-derived response") + createdSKU := apiReq.ToProto().Skus[0] + createdSKU.Id = ids.Ids[0] + skuToPersist = createdSKU + } else { + skuToPersist = response.Skus[0] + } + + if skuToPersist == nil || skuToPersist.Id == "" { + err = errors.New("cannot persist an empty Core SKU") + } else { + projected := &cdbm.SKU{} + projected.FromProto(skuToPersist, uuid.MustParse(siteID)) + skuDAO := cdbm.NewSkuDAO(csh.dbSession) + err = cdb.WithTx(ctx, csh.dbSession, func(tx *cdb.Tx) error { + err := tx.TryAcquireAdvisoryLock(ctx, cdb.GetAdvisoryLockIDFromString(projected.ID), nil) + if err != nil { + return err + } + + _, err = skuDAO.Create(ctx, tx, cdbm.SkuCreateInput{ + SkuID: projected.ID, + SiteID: projected.SiteID, + Description: projected.Description, + SchemaVersion: projected.SchemaVersion, + Components: projected.Components, + DeviceType: projected.DeviceType, + AssociatedMachineIds: projected.AssociatedMachineIds, + }) + return err + }) + } + if err != nil { + errorChecker := cdb.PostgresErrorChecker{} + if errorChecker.IsUniqueConstraintError(err) { + message := fmt.Sprintf( + "SKU %q already exists; inspect it with GET /v2/org/%s/nico/sku/%s and retry changes as an update", + skuToPersist.Id, + org, + skuToPersist.Id, + ) + return cutil.NewAPIErrorResponse(c, http.StatusConflict, message, nil) + } + logger.Error().Err(err).Str("skuID", ids.Ids[0]).Str("siteID", siteID). + Msg("SKU created in Core but failed to update REST DB") + return cutil.NewAPIErrorResponse(c, http.StatusInternalServerError, "SKU was created in Core but failed to update REST DB", nil) + } + if apiErr != nil { + return c.JSON(http.StatusCreated, model.NewAPISkuFromCreateRequest(apiReq, ids.Ids[0], siteID)) + } + return c.JSON(http.StatusCreated, model.NewAPISkuFromProto(skuToPersist, siteID)) +} + +// UpdateSkuHandler partially updates one SKU on a Site's Core service. +type UpdateSkuHandler struct { + dbSession *cdb.Session + scp *sc.ClientPool + tracerSpan *cutil.TracerSpan +} + +// NewUpdateSkuHandler returns a new UpdateSkuHandler. +func NewUpdateSkuHandler(dbSession *cdb.Session, scp *sc.ClientPool) UpdateSkuHandler { + return UpdateSkuHandler{ + dbSession: dbSession, + scp: scp, + tracerSpan: cutil.NewTracerSpan(), + } +} + +// Handle godoc +// @Summary Update SKU +// @Description Update selected mutable fields on a SKU. +// @Tags SKU +// @Accept json +// @Produce json +// @Security ApiKeyAuth +// @Param org path string true "Name of NGC organization" +// @Param id path string true "SKU ID" +// @Param request body model.APISkuUpdateRequest true "SKU update request" +// @Success 200 {object} model.APISku +// @Router /v2/org/{org}/nico/sku/{id} [patch] +func (ush UpdateSkuHandler) Handle(c echo.Context) error { + org, dbUser, ctx, logger, handlerSpan := common.SetupHandler("SKU", "Update", c, ush.tracerSpan) + if handlerSpan != nil { + defer handlerSpan.End() + } + + skuID := c.Param("id") + if skuID == "" { + return cutil.NewAPIErrorResponse(c, http.StatusBadRequest, "SKU ID must be specified", nil) + } + ush.tracerSpan.SetAttribute(handlerSpan, attribute.String("sku_id", skuID), logger) + + apiReq := model.APISkuUpdateRequest{} + err := c.Bind(&apiReq) + if err != nil { + logger.Warn().Err(err).Msg("error binding request data into API model") + return cutil.NewAPIErrorResponse(c, http.StatusBadRequest, "Failed to parse request data, potentially invalid structure", nil) + } + + err = apiReq.Validate() + if err != nil { + logger.Warn().Err(err).Msg("error validating SKU update request data") + return cutil.NewAPIErrorResponse(c, http.StatusBadRequest, "Error validating SKU update request data", err) + } + + skuDAO := cdbm.NewSkuDAO(ush.dbSession) + savedSKU, err := skuDAO.Get(ctx, nil, skuID) + if errors.Is(err, cdb.ErrDoesNotExist) { + return cutil.NewAPIErrorResponse(c, http.StatusNotFound, "Could not find SKU with the specified ID", nil) + } + if err != nil { + logger.Error().Err(err).Str("skuID", skuID).Msg("error retrieving SKU from DB") + return cutil.NewAPIErrorResponse(c, http.StatusInternalServerError, "Failed to retrieve SKU, DB error", nil) + } + + stc, siteID, apiErr := common.AuthorizeProviderSiteForCore(common.AuthorizeProviderSiteForCoreInput{ + Ctx: ctx, + Logger: logger, + DBSession: ush.dbSession, + SCP: ush.scp, + Org: org, + User: dbUser, + SiteID: savedSKU.SiteID.String(), + }) + if apiErr != nil { + return cutil.NewAPIErrorResponse(c, apiErr.Code, apiErr.Message, apiErr.Data) + } + + var response corev1.SkuList + apiErr = common.ExecuteCoreGRPC( + ctx, + stc, + corev1.Forge_FindSkusByIds_FullMethodName, + &corev1.SkusByIdsRequest{Ids: []string{skuID}}, + &response, + siteID, + ) + if apiErr == nil && len(response.Skus) != 1 { + apiErr = cutil.NewAPIError(http.StatusNotFound, "Could not find SKU with the specified ID", nil) + } + if apiErr != nil { + logAPIError(logger, apiErr, "failed to retrieve SKU before update") + return cutil.NewAPIErrorResponse(c, apiErr.Code, apiErr.Message, nil) + } + + apiReq.SkuID = skuID + + updatedSKU := &corev1.Sku{} + if apiReq.Components == nil { + updatedSKU = apiReq.ApplyMetadataToProto(response.Skus[0]) + logger.Info().Str("skuID", skuID).Str("siteID", siteID).Msg("updating SKU metadata via Core proxy") + apiErr = common.ExecuteCoreGRPC(ctx, stc, corev1.Forge_UpdateSkuMetadata_FullMethodName, apiReq.ToMetadataProto(), nil, siteID) + if apiErr != nil { + logAPIError(logger, apiErr, "failed to update SKU metadata via Core proxy") + return cutil.NewAPIErrorResponse(c, apiErr.Code, apiErr.Message, nil) + } + } else { + updatedReq := apiReq.ToReplacementProto(response.Skus[0]) + logger.Info().Str("skuID", skuID).Str("siteID", siteID).Msg("updating SKU via Core proxy") + apiErr = common.ExecuteCoreGRPC(ctx, stc, corev1.Forge_ReplaceSku_FullMethodName, updatedReq, updatedSKU, siteID) + if apiErr != nil { + logAPIError(logger, apiErr, "failed to update SKU via Core proxy") + return cutil.NewAPIErrorResponse(c, apiErr.Code, apiErr.Message, nil) + } + } + + if updatedSKU == nil || updatedSKU.Id == "" { + err = errors.New("cannot persist an empty Core SKU") + } else { + projected := &cdbm.SKU{} + projected.FromProto(updatedSKU, savedSKU.SiteID) + + err = cdb.WithTx(ctx, ush.dbSession, func(tx *cdb.Tx) error { + err := tx.TryAcquireAdvisoryLock(ctx, cdb.GetAdvisoryLockIDFromString(projected.ID), nil) + if err != nil { + return err + } + + existing, err := skuDAO.Get(ctx, tx, projected.ID) + if err != nil { + return err + } + if existing.SiteID != projected.SiteID { + return fmt.Errorf("SKU %q already exists for Site %q", projected.ID, existing.SiteID) + } + + components := projected.Components + if existing.Components != nil && components == nil { + // DAO update uses nil to mean "not supplied", so use an empty value + // to clear stale components when Core reports none. + components = &cdbm.SkuComponents{SkuComponents: &corev1.SkuComponents{}} + } + associatedMachineIDs := projected.AssociatedMachineIds + if associatedMachineIDs == nil { + // DAO update uses nil to mean "not supplied"; Core nil means there + // are no associations in the authoritative projection. + associatedMachineIDs = []string{} + } + if existing.Description == projected.Description && + existing.SchemaVersion == projected.SchemaVersion && + existing.Components.Equal(components) && + reflect.DeepEqual(existing.DeviceType, projected.DeviceType) && + reflect.DeepEqual(existing.AssociatedMachineIds, associatedMachineIDs) { + return nil + } + + _, err = skuDAO.Update(ctx, tx, cdbm.SkuUpdateInput{ + SkuID: projected.ID, + Description: &projected.Description, + SchemaVersion: &projected.SchemaVersion, + Components: components, + DeviceType: projected.DeviceType, + AssociatedMachineIds: associatedMachineIDs, + }) + return err + }) + } + if err != nil { + logger.Error().Err(err).Str("skuID", skuID).Str("siteID", siteID). + Msg("SKU updated in Core but failed to update REST DB") + return cutil.NewAPIErrorResponse(c, http.StatusInternalServerError, "SKU was updated in Core but failed to update REST DB", nil) + } + + return c.JSON(http.StatusOK, model.NewAPISkuFromProto(updatedSKU, siteID)) +} + +// DeleteSkuHandler deletes one unused SKU from a Site's Core service. +type DeleteSkuHandler struct { + dbSession *cdb.Session + scp *sc.ClientPool + tracerSpan *cutil.TracerSpan +} + +// NewDeleteSkuHandler returns a new DeleteSkuHandler. +func NewDeleteSkuHandler(dbSession *cdb.Session, scp *sc.ClientPool) DeleteSkuHandler { + return DeleteSkuHandler{ + dbSession: dbSession, + scp: scp, + tracerSpan: cutil.NewTracerSpan(), + } +} + +// Handle godoc +// @Summary Delete SKU +// @Description Delete an unused SKU. +// @Tags SKU +// @Produce json +// @Security ApiKeyAuth +// @Param org path string true "Name of NGC organization" +// @Param id path string true "SKU ID" +// @Success 204 +// @Router /v2/org/{org}/nico/sku/{id} [delete] +func (dsh DeleteSkuHandler) Handle(c echo.Context) error { + org, dbUser, ctx, logger, handlerSpan := common.SetupHandler("SKU", "Delete", c, dsh.tracerSpan) + if handlerSpan != nil { + defer handlerSpan.End() + } + + skuID := c.Param("id") + if skuID == "" { + return cutil.NewAPIErrorResponse(c, http.StatusBadRequest, "SKU ID must be specified", nil) + } + + dsh.tracerSpan.SetAttribute(handlerSpan, attribute.String("sku_id", skuID), logger) + + skuDAO := cdbm.NewSkuDAO(dsh.dbSession) + savedSKU, err := skuDAO.Get(ctx, nil, skuID) + if errors.Is(err, cdb.ErrDoesNotExist) { + return cutil.NewAPIErrorResponse(c, http.StatusNotFound, "Could not find SKU with the specified ID", nil) + } + if err != nil { + logger.Error().Err(err).Str("skuID", skuID).Msg("error retrieving SKU from DB") + return cutil.NewAPIErrorResponse(c, http.StatusInternalServerError, "Failed to retrieve SKU, DB error", nil) + } + if len(savedSKU.AssociatedMachineIds) > 0 { + logger.Warn().Str("skuID", skuID).Msg("SKU is associated with machines and cannot be deleted") + return cutil.NewAPIErrorResponse(c, http.StatusBadRequest, "SKU is associated with machines and cannot be deleted", nil) + } + + stc, siteID, apiErr := common.AuthorizeProviderSiteForCore(common.AuthorizeProviderSiteForCoreInput{ + Ctx: ctx, + Logger: logger, + DBSession: dsh.dbSession, + SCP: dsh.scp, + Org: org, + User: dbUser, + SiteID: savedSKU.SiteID.String(), + }) + if apiErr != nil { + return cutil.NewAPIErrorResponse(c, apiErr.Code, apiErr.Message, apiErr.Data) + } + + logger.Info().Str("skuID", skuID).Str("siteID", siteID).Msg("deleting SKU via Core proxy") + apiErr = common.ExecuteCoreGRPC(ctx, stc, corev1.Forge_DeleteSku_FullMethodName, &corev1.SkuIdList{Ids: []string{skuID}}, nil, siteID) + if apiErr != nil { + if apiErr.Code != http.StatusNotFound { + logAPIError(logger, apiErr, "failed to delete SKU via Core proxy") + return cutil.NewAPIErrorResponse(c, apiErr.Code, apiErr.Message, nil) + } + logger.Warn().Str("skuID", skuID).Str("siteID", siteID).Msg("SKU not found in Core, removing stale REST DB record") + } + err = skuDAO.Delete(ctx, nil, skuID) + if err != nil { + logger.Error().Err(err).Str("skuID", skuID).Msg("failed to delete SKU from REST DB") + return cutil.NewAPIErrorResponse(c, http.StatusInternalServerError, "Failed to delete SKU from REST DB", nil) + } + + logger.Info().Str("skuID", skuID).Str("siteID", siteID).Msg("finishing API handler") + return c.NoContent(http.StatusNoContent) +} diff --git a/rest-api/api/pkg/api/handler/sku_test.go b/rest-api/api/pkg/api/handler/sku_test.go index b5eb205e66..8157e9a712 100644 --- a/rest-api/api/pkg/api/handler/sku_test.go +++ b/rest-api/api/pkg/api/handler/sku_test.go @@ -6,21 +6,36 @@ package handler import ( "context" "encoding/json" + "errors" "net/http" "net/http/httptest" + "strings" "testing" "github.com/NVIDIA/infra-controller/rest-api/api/internal/config" + "github.com/NVIDIA/infra-controller/rest-api/api/pkg/api/handler/util/common" "github.com/NVIDIA/infra-controller/rest-api/api/pkg/api/model" + sc "github.com/NVIDIA/infra-controller/rest-api/api/pkg/client/site" authz "github.com/NVIDIA/infra-controller/rest-api/auth/pkg/authorization" + "github.com/NVIDIA/infra-controller/rest-api/common/pkg/coreproxy" cutil "github.com/NVIDIA/infra-controller/rest-api/common/pkg/util" cdb "github.com/NVIDIA/infra-controller/rest-api/db/pkg/db" cdbm "github.com/NVIDIA/infra-controller/rest-api/db/pkg/db/model" cdbu "github.com/NVIDIA/infra-controller/rest-api/db/pkg/util" + corev1 "github.com/NVIDIA/infra-controller/rest-api/proto/core/gen/v1" + swe "github.com/NVIDIA/infra-controller/rest-api/site-workflow/pkg/error" "github.com/google/uuid" "github.com/labstack/echo/v4" "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/mock" + "github.com/stretchr/testify/require" "github.com/uptrace/bun/extra/bundebug" + tmocks "go.temporal.io/sdk/mocks" + tp "go.temporal.io/sdk/temporal" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" + "google.golang.org/protobuf/encoding/protojson" + "google.golang.org/protobuf/proto" ) // testSkuInitDB initializes a test database session (pattern from tenant_test.go) @@ -81,6 +96,7 @@ func testSkuSetupTestData(t *testing.T, dbSession *cdb.Session, org string) (*cd sku1 := &cdbm.SKU{ ID: "test-sku-1", SiteID: site.ID, + Description: "First test SKU", DeviceType: &deviceType1, AssociatedMachineIds: []string{"machine-1", "machine-2"}, } @@ -91,6 +107,7 @@ func testSkuSetupTestData(t *testing.T, dbSession *cdb.Session, org string) (*cd sku2 := &cdbm.SKU{ ID: "test-sku-2", SiteID: site.ID, + Description: "Second test SKU", DeviceType: &deviceType2, AssociatedMachineIds: []string{"machine-3"}, } @@ -272,6 +289,7 @@ func TestGetAllSkuHandler_Handle(t *testing.T) { for _, sku := range response { assert.Equal(t, site.ID.String(), sku.SiteID, "All results should be from the specified site") assert.NotEqual(t, unmanagedSku.ID, sku.ID, "Unmanaged SKU should not be in response") + assert.NotEmpty(t, sku.Description) } }, }, @@ -544,6 +562,7 @@ func TestGetSkuHandler_Handle(t *testing.T) { assert.Nil(t, err) assert.Equal(t, sku1.ID, response.ID, "SKU ID should match") assert.Equal(t, site.ID.String(), response.SiteID, "Site ID should match") + assert.Equal(t, sku1.Description, response.Description, "SKU description should match") }, }, { @@ -664,3 +683,719 @@ func TestGetSkuHandler_Handle(t *testing.T) { }) } } + +func TestCreateSkuHandler(t *testing.T) { + t.Run("proxies create and returns created SKU", func(t *testing.T) { + fixture := newSkuManagementFixtureWithOptions(t, []string{authz.ProviderAdminRole}, skuManagementFixtureOptions{ + skipPersistedSKU: true, + }) + req := validSkuCreateRequest(fixture.siteID) + + rec := fixture.request(t, http.MethodPost, "", req, fixture.createHandler.Handle) + require.Equal(t, http.StatusCreated, rec.Code, rec.Body.String()) + require.Len(t, fixture.requests, 2) + assert.Equal(t, corev1.Forge_CreateSku_FullMethodName, fixture.requests[0].FullMethod) + assert.Equal(t, corev1.Forge_FindSkusByIds_FullMethodName, fixture.requests[1].FullMethod) + + var coreReq corev1.SkuList + require.NoError(t, protojson.Unmarshal(fixture.requests[0].RequestJSON, &coreReq)) + require.Len(t, coreReq.Skus, 1) + assert.Equal(t, req.ID, coreReq.Skus[0].Id) + assert.Equal(t, model.CoreSkuSchemaVersion, coreReq.Skus[0].SchemaVersion) + require.Len(t, coreReq.Skus[0].Components.Storage, 1) + assert.Empty(t, coreReq.Skus[0].Components.Storage[0].Vendor) + assert.Zero(t, coreReq.Skus[0].Components.Storage[0].CapacityMb) + assert.Equal(t, uint32(3_600_000), coreReq.Skus[0].Components.Storage[0].GetMinSizeMb()) + assert.Equal(t, uint32(3_900_000), coreReq.Skus[0].Components.Storage[0].GetMaxSizeMb()) + assert.Equal(t, []string{`^/devices/pci.*nvme[0-1]$`}, coreReq.Skus[0].Components.Storage[0].PciPatterns) + + var response model.APISku + require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &response)) + assert.Equal(t, req.ID, response.ID) + assert.Equal(t, fixture.siteID, response.SiteID) + assert.Empty(t, response.AssociatedMachineIds) + + saved, err := cdbm.NewSkuDAO(fixture.createHandler.dbSession).Get(context.Background(), nil, req.ID) + require.NoError(t, err) + assert.Equal(t, uuid.MustParse(fixture.siteID), saved.SiteID) + assert.Equal(t, response.Description, saved.Description) + assert.Equal(t, response.SchemaVersion, saved.SchemaVersion) + assert.Equal(t, response.DeviceType, saved.DeviceType) + require.NotNil(t, saved.Components) + require.NotNil(t, saved.Components.Chassis) + // The post-create Core response is authoritative, even when it differs + // from the create request used by this test fixture. + assert.Equal(t, "existing chassis", saved.Components.Chassis.Model) + }) + + t.Run("returns created when post-create fetch fails", func(t *testing.T) { + fixture := newSkuManagementFixtureWithOptions(t, []string{authz.ProviderAdminRole}, skuManagementFixtureOptions{ + findError: errors.New("post-create fetch failed"), + skipPersistedSKU: true, + }) + req := validSkuCreateRequest(fixture.siteID) + + rec := fixture.request(t, http.MethodPost, "", req, fixture.createHandler.Handle) + require.Equal(t, http.StatusCreated, rec.Code, rec.Body.String()) + require.Len(t, fixture.requests, 2) + assert.Equal(t, corev1.Forge_CreateSku_FullMethodName, fixture.requests[0].FullMethod) + assert.Equal(t, corev1.Forge_FindSkusByIds_FullMethodName, fixture.requests[1].FullMethod) + + var response model.APISku + require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &response)) + assert.Equal(t, req.ID, response.ID) + assert.Equal(t, fixture.siteID, response.SiteID) + assert.Equal(t, *req.Description, response.Description) + assert.Equal(t, model.CoreSkuSchemaVersion, response.SchemaVersion) + assert.Equal(t, req.DeviceType, response.DeviceType) + assert.Equal(t, model.NewAPISkuComponents(req.Components.ToProto()), response.Components) + assert.Empty(t, response.AssociatedMachineIds) + assert.Nil(t, response.Created) + + saved, err := cdbm.NewSkuDAO(fixture.createHandler.dbSession).Get(context.Background(), nil, req.ID) + require.NoError(t, err) + assert.Equal(t, uuid.MustParse(fixture.siteID), saved.SiteID) + assert.Equal(t, *req.Description, saved.Description) + assert.Equal(t, model.CoreSkuSchemaVersion, saved.SchemaVersion) + assert.Equal(t, req.DeviceType, saved.DeviceType) + require.NotNil(t, saved.Components) + require.Len(t, saved.Components.Storage, 1) + assert.Equal(t, uint32(3_600_000), saved.Components.Storage[0].GetMinSizeMb()) + assert.Equal(t, uint32(3_900_000), saved.Components.Storage[0].GetMaxSizeMb()) + }) + + t.Run("uses Core as the authority for duplicate IDs", func(t *testing.T) { + fixture := newSkuManagementFixtureWithOptions(t, []string{authz.ProviderAdminRole}, skuManagementFixtureOptions{ + createError: status.Error(codes.AlreadyExists, "SKU already exists: sku-1"), + }) + + rec := fixture.request(t, http.MethodPost, "", validSkuCreateRequest(fixture.siteID), fixture.createHandler.Handle) + + require.Equal(t, http.StatusConflict, rec.Code, rec.Body.String()) + require.Len(t, fixture.requests, 1) + assert.Equal(t, corev1.Forge_CreateSku_FullMethodName, fixture.requests[0].FullMethod) + }) + + t.Run("returns conflict when the REST projection already exists", func(t *testing.T) { + fixture := newSkuManagementFixture(t, []string{authz.ProviderAdminRole}) + + rec := fixture.request(t, http.MethodPost, "", validSkuCreateRequest(fixture.siteID), fixture.createHandler.Handle) + + require.Equal(t, http.StatusConflict, rec.Code, rec.Body.String()) + require.Len(t, fixture.requests, 2) + assert.Equal(t, corev1.Forge_CreateSku_FullMethodName, fixture.requests[0].FullMethod) + assert.Equal(t, corev1.Forge_FindSkusByIds_FullMethodName, fixture.requests[1].FullMethod) + assert.Contains(t, rec.Body.String(), "sku-1") + assert.Contains(t, rec.Body.String(), "inspect") + assert.Contains(t, rec.Body.String(), "update") + }) + + t.Run("returns error when Core create fails", func(t *testing.T) { + fixture := newSkuManagementFixtureWithOptions(t, []string{authz.ProviderAdminRole}, skuManagementFixtureOptions{ + createError: errors.New("Core unavailable"), + skipPersistedSKU: true, + }) + + rec := fixture.request(t, http.MethodPost, "", validSkuCreateRequest(fixture.siteID), fixture.createHandler.Handle) + + require.Equal(t, http.StatusInternalServerError, rec.Code, rec.Body.String()) + require.Len(t, fixture.requests, 1) + assert.Equal(t, corev1.Forge_CreateSku_FullMethodName, fixture.requests[0].FullMethod) + _, err := cdbm.NewSkuDAO(fixture.createHandler.dbSession).Get(context.Background(), nil, "sku-1") + assert.ErrorIs(t, err, cdb.ErrDoesNotExist) + }) + + t.Run("rejects invalid Site ID", func(t *testing.T) { + fixture := newSkuManagementFixture(t, []string{authz.ProviderAdminRole}) + req := validSkuCreateRequest("invalid-site-id") + + rec := fixture.request(t, http.MethodPost, "", req, fixture.createHandler.Handle) + + require.Equal(t, http.StatusBadRequest, rec.Code, rec.Body.String()) + assert.Contains(t, rec.Body.String(), "siteId") + assert.Empty(t, fixture.requests) + }) + + t.Run("rejects legacy storage mutation fields", func(t *testing.T) { + fixture := newSkuManagementFixture(t, []string{authz.ProviderAdminRole}) + + for _, field := range []string{"vendor", "capacityMb"} { + t.Run(field, func(t *testing.T) { + rec := fixture.request(t, http.MethodPost, "", map[string]any{ + "siteId": fixture.siteID, + "id": "sku-legacy-storage", + "components": map[string]any{ + "storage": []map[string]any{{ + "model": "legacy", + "count": 1, + field: 0, + }}, + }, + }, fixture.createHandler.Handle) + + assert.Equal(t, http.StatusBadRequest, rec.Code, rec.Body.String()) + }) + } + assert.Empty(t, fixture.requests) + }) + + t.Run("rejects inverted storage size range", func(t *testing.T) { + fixture := newSkuManagementFixture(t, []string{authz.ProviderAdminRole}) + req := validSkuCreateRequest(fixture.siteID) + req.Components.Storage[0].MinSizeMiB = cutil.GetPtr(uint32(4_000_000)) + req.Components.Storage[0].MaxSizeMiB = cutil.GetPtr(uint32(3_800_000)) + + rec := fixture.request(t, http.MethodPost, "", req, fixture.createHandler.Handle) + + assert.Equal(t, http.StatusBadRequest, rec.Code, rec.Body.String()) + assert.Contains(t, rec.Body.String(), "minSizeMiB") + assert.Empty(t, fixture.requests) + }) + + t.Run("rejects tenant admin", func(t *testing.T) { + fixture := newSkuManagementFixture(t, []string{authz.TenantAdminRole}) + + rec := fixture.request(t, http.MethodPost, "", validSkuCreateRequest(fixture.siteID), fixture.createHandler.Handle) + assert.Equal(t, http.StatusForbidden, rec.Code) + assert.Empty(t, fixture.requests) + }) +} + +func TestUpdateSkuHandler(t *testing.T) { + t.Run("uses metadata RPC for metadata patch", func(t *testing.T) { + fixture := newSkuManagementFixture(t, []string{authz.ProviderAdminRole}) + deviceType := "cpu-server" + + rec := fixture.request(t, http.MethodPatch, "sku-1", model.APISkuUpdateRequest{ + DeviceType: &deviceType, + }, fixture.updateHandler.Handle) + require.Equal(t, http.StatusOK, rec.Code, rec.Body.String()) + require.Len(t, fixture.requests, 2) + assert.Equal(t, corev1.Forge_FindSkusByIds_FullMethodName, fixture.requests[0].FullMethod) + assert.Equal(t, corev1.Forge_UpdateSkuMetadata_FullMethodName, fixture.requests[1].FullMethod) + + var coreReq corev1.SkuUpdateMetadataRequest + require.NoError(t, protojson.Unmarshal(fixture.requests[1].RequestJSON, &coreReq)) + assert.Equal(t, "sku-1", coreReq.SkuId) + assert.Equal(t, deviceType, coreReq.GetDeviceType()) + + var response model.APISku + require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &response)) + require.NotNil(t, response.DeviceType) + assert.Equal(t, deviceType, *response.DeviceType) + assert.Equal(t, uint32(4), response.SchemaVersion) + + saved, err := cdbm.NewSkuDAO(fixture.updateHandler.dbSession).Get(context.Background(), nil, "sku-1") + require.NoError(t, err) + require.NotNil(t, saved.DeviceType) + assert.Equal(t, deviceType, *saved.DeviceType) + assert.Equal(t, uint32(4), saved.SchemaVersion) + require.NotNil(t, saved.Components) + require.NotNil(t, saved.Components.Chassis) + assert.Equal(t, "existing chassis", saved.Components.Chassis.Model) + }) + + t.Run("preserves projection when Core metadataupdate fails", func(t *testing.T) { + fixture := newSkuManagementFixtureWithOptions(t, []string{authz.ProviderAdminRole}, skuManagementFixtureOptions{ + updateMetadataError: errors.New("Core unavailable"), + }) + deviceType := "cpu-server" + + rec := fixture.request(t, http.MethodPatch, "sku-1", model.APISkuUpdateRequest{ + DeviceType: &deviceType, + }, fixture.updateHandler.Handle) + require.Equal(t, http.StatusInternalServerError, rec.Code, rec.Body.String()) + require.Len(t, fixture.requests, 2) + assert.Equal(t, corev1.Forge_FindSkusByIds_FullMethodName, fixture.requests[0].FullMethod) + assert.Equal(t, corev1.Forge_UpdateSkuMetadata_FullMethodName, fixture.requests[1].FullMethod) + + var coreReq corev1.SkuUpdateMetadataRequest + require.NoError(t, protojson.Unmarshal(fixture.requests[1].RequestJSON, &coreReq)) + assert.Equal(t, deviceType, coreReq.GetDeviceType()) + + saved, err := cdbm.NewSkuDAO(fixture.updateHandler.dbSession).Get(context.Background(), nil, "sku-1") + require.NoError(t, err) + assert.Nil(t, saved.DeviceType) + assert.Nil(t, saved.Components) + assert.Empty(t, saved.AssociatedMachineIds) + }) + + t.Run("does not recreate projection deleted after Core update", func(t *testing.T) { + fixture := newSkuManagementFixtureWithOptions(t, []string{authz.ProviderAdminRole}, skuManagementFixtureOptions{ + afterUpdateMetadata: func(dbSession *cdb.Session) { + skuDAO := cdbm.NewSkuDAO(dbSession) + err := skuDAO.Delete(context.Background(), nil, "sku-1") + require.NoError(t, err) + }, + }) + description := "updated description" + + rec := fixture.request(t, http.MethodPatch, "sku-1", model.APISkuUpdateRequest{ + Description: &description, + }, fixture.updateHandler.Handle) + + require.Equal(t, http.StatusInternalServerError, rec.Code, rec.Body.String()) + assert.Contains(t, rec.Body.String(), "failed to update REST DB") + _, err := cdbm.NewSkuDAO(fixture.updateHandler.dbSession).Get(context.Background(), nil, "sku-1") + assert.ErrorIs(t, err, cdb.ErrDoesNotExist) + }) + + t.Run("replaces version five components", func(t *testing.T) { + existing := existingSkuProto() + existing.SchemaVersion = model.CoreSkuSchemaVersion + deviceType := "cpu-server" + replacementResponse := existingSkuProto() + replacementResponse.SchemaVersion = model.CoreSkuSchemaVersion + replacementResponse.DeviceType = &deviceType + replacementResponse.Components = validSkuCreateRequest("").ToProto().Skus[0].Components + fixture := newSkuManagementFixtureWithOptions(t, []string{authz.ProviderAdminRole}, skuManagementFixtureOptions{ + findResponse: &corev1.SkuList{Skus: []*corev1.Sku{existing}}, + replaceResponse: replacementResponse, + }) + components := validSkuCreateRequest(fixture.siteID).Components + + rec := fixture.request(t, http.MethodPatch, "sku-1", model.APISkuUpdateRequest{ + DeviceType: &deviceType, + Components: components, + }, fixture.updateHandler.Handle) + require.Equal(t, http.StatusOK, rec.Code, rec.Body.String()) + require.Len(t, fixture.requests, 2) + assert.Equal(t, corev1.Forge_FindSkusByIds_FullMethodName, fixture.requests[0].FullMethod) + assert.Equal(t, corev1.Forge_ReplaceSku_FullMethodName, fixture.requests[1].FullMethod) + + var coreReq corev1.Sku + require.NoError(t, protojson.Unmarshal(fixture.requests[1].RequestJSON, &coreReq)) + assert.Equal(t, "sku-1", coreReq.Id) + assert.Equal(t, deviceType, coreReq.GetDeviceType()) + assert.Equal(t, model.CoreSkuSchemaVersion, coreReq.SchemaVersion) + require.NotNil(t, coreReq.Components) + require.Len(t, coreReq.Components.Storage, 1) + + saved, err := cdbm.NewSkuDAO(fixture.updateHandler.dbSession).Get(context.Background(), nil, "sku-1") + require.NoError(t, err) + require.NotNil(t, saved.DeviceType) + assert.Equal(t, deviceType, *saved.DeviceType) + require.NotNil(t, saved.Components) + require.Len(t, saved.Components.Storage, 1) + assert.Equal(t, uint32(3_600_000), saved.Components.Storage[0].GetMinSizeMb()) + assert.Equal(t, uint32(3_900_000), saved.Components.Storage[0].GetMaxSizeMb()) + }) + + t.Run("preserves projection when Core replace fails", func(t *testing.T) { + existing := existingSkuProto() + existing.SchemaVersion = model.CoreSkuSchemaVersion + fixture := newSkuManagementFixtureWithOptions(t, []string{authz.ProviderAdminRole}, skuManagementFixtureOptions{ + findResponse: &corev1.SkuList{Skus: []*corev1.Sku{existing}}, + replaceError: errors.New("Core unavailable"), + }) + deviceType := "cpu-server" + + rec := fixture.request(t, http.MethodPatch, "sku-1", model.APISkuUpdateRequest{ + DeviceType: &deviceType, + Components: validSkuCreateRequest(fixture.siteID).Components, + }, fixture.updateHandler.Handle) + + require.Equal(t, http.StatusInternalServerError, rec.Code, rec.Body.String()) + require.Len(t, fixture.requests, 2) + assert.Equal(t, corev1.Forge_FindSkusByIds_FullMethodName, fixture.requests[0].FullMethod) + assert.Equal(t, corev1.Forge_ReplaceSku_FullMethodName, fixture.requests[1].FullMethod) + + var coreReq corev1.Sku + require.NoError(t, protojson.Unmarshal(fixture.requests[1].RequestJSON, &coreReq)) + assert.Equal(t, deviceType, coreReq.GetDeviceType()) + + saved, err := cdbm.NewSkuDAO(fixture.updateHandler.dbSession).Get(context.Background(), nil, "sku-1") + require.NoError(t, err) + assert.Nil(t, saved.DeviceType) + assert.Nil(t, saved.Components) + }) + + t.Run("replaces components for SKU with associated machines", func(t *testing.T) { + existing := existingSkuProto() + existing.SchemaVersion = model.CoreSkuSchemaVersion + existing.AssociatedMachineIds = []*corev1.MachineId{{Id: "machine-1"}} + replacementResponse := proto.Clone(existing).(*corev1.Sku) + replacementResponse.Components = validSkuCreateRequest("").ToProto().Skus[0].Components + fixture := newSkuManagementFixtureWithOptions(t, []string{authz.ProviderAdminRole}, skuManagementFixtureOptions{ + findResponse: &corev1.SkuList{Skus: []*corev1.Sku{existing}}, + replaceResponse: replacementResponse, + }) + + rec := fixture.request(t, http.MethodPatch, "sku-1", model.APISkuUpdateRequest{ + Components: validSkuCreateRequest(fixture.siteID).Components, + }, fixture.updateHandler.Handle) + + require.Equal(t, http.StatusOK, rec.Code, rec.Body.String()) + require.Len(t, fixture.requests, 2) + assert.Equal(t, corev1.Forge_FindSkusByIds_FullMethodName, fixture.requests[0].FullMethod) + assert.Equal(t, corev1.Forge_ReplaceSku_FullMethodName, fixture.requests[1].FullMethod) + + var coreReq corev1.Sku + require.NoError(t, protojson.Unmarshal(fixture.requests[1].RequestJSON, &coreReq)) + require.Len(t, coreReq.AssociatedMachineIds, 1) + assert.Equal(t, "machine-1", coreReq.AssociatedMachineIds[0].GetId()) + + var response model.APISku + require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &response)) + assert.Equal(t, []string{"machine-1"}, response.AssociatedMachineIds) + + saved, err := cdbm.NewSkuDAO(fixture.updateHandler.dbSession).Get(context.Background(), nil, "sku-1") + require.NoError(t, err) + assert.Equal(t, []string{"machine-1"}, saved.AssociatedMachineIds) + }) + + t.Run("sends version five when replacing legacy SKU components", func(t *testing.T) { + existing := existingSkuProto() + require.Equal(t, uint32(4), existing.SchemaVersion) + replacementResponse := proto.Clone(existing).(*corev1.Sku) + replacementResponse.SchemaVersion = model.CoreSkuSchemaVersion + replacementResponse.Components = validSkuCreateRequest("").ToProto().Skus[0].Components + fixture := newSkuManagementFixtureWithOptions(t, []string{authz.ProviderAdminRole}, skuManagementFixtureOptions{ + findResponse: &corev1.SkuList{Skus: []*corev1.Sku{existing}}, + replaceResponse: replacementResponse, + }) + components := validSkuCreateRequest(fixture.siteID).Components + + rec := fixture.request(t, http.MethodPatch, "sku-1", model.APISkuUpdateRequest{ + Components: components, + }, fixture.updateHandler.Handle) + require.Equal(t, http.StatusOK, rec.Code, rec.Body.String()) + require.Len(t, fixture.requests, 2) + assert.Equal(t, corev1.Forge_FindSkusByIds_FullMethodName, fixture.requests[0].FullMethod) + assert.Equal(t, corev1.Forge_ReplaceSku_FullMethodName, fixture.requests[1].FullMethod) + + var coreReq corev1.Sku + require.NoError(t, protojson.Unmarshal(fixture.requests[1].RequestJSON, &coreReq)) + assert.Equal(t, model.CoreSkuSchemaVersion, coreReq.SchemaVersion) + assert.Equal(t, "sku-1", coreReq.Id) + require.NotNil(t, coreReq.Components) + require.Len(t, coreReq.Components.Storage, 1) + assert.True(t, proto.Equal(components.ToProto(), coreReq.Components)) + }) + + t.Run("rejects inverted storage size range", func(t *testing.T) { + fixture := newSkuManagementFixture(t, []string{authz.ProviderAdminRole}) + components := validSkuCreateRequest(fixture.siteID).Components + components.Storage[0].MinSizeMiB = cutil.GetPtr(uint32(4_000_000)) + components.Storage[0].MaxSizeMiB = cutil.GetPtr(uint32(3_800_000)) + + rec := fixture.request(t, http.MethodPatch, "sku-1", model.APISkuUpdateRequest{ + Components: components, + }, fixture.updateHandler.Handle) + + assert.Equal(t, http.StatusBadRequest, rec.Code, rec.Body.String()) + assert.Contains(t, rec.Body.String(), "minSizeMiB") + assert.Empty(t, fixture.requests) + }) + + t.Run("returns Core not found without completing update", func(t *testing.T) { + fixture := newSkuManagementFixtureWithOptions(t, []string{authz.ProviderAdminRole}, skuManagementFixtureOptions{ + findResponse: &corev1.SkuList{}, + }) + description := "updated description" + + rec := fixture.request(t, http.MethodPatch, "sku-1", model.APISkuUpdateRequest{ + Description: &description, + }, fixture.updateHandler.Handle) + require.Equal(t, http.StatusNotFound, rec.Code, rec.Body.String()) + assert.JSONEq(t, `{"source":"","message":"Could not find SKU with the specified ID","data":null}`, rec.Body.String()) + require.Len(t, fixture.requests, 1) + assert.Equal(t, corev1.Forge_FindSkusByIds_FullMethodName, fixture.requests[0].FullMethod) + }) + + t.Run("returns not found for SKU not present in REST database", func(t *testing.T) { + fixture := newSkuManagementFixture(t, []string{authz.ProviderAdminRole}) + description := "updated description" + + rec := fixture.request(t, http.MethodPatch, "missing-sku", model.APISkuUpdateRequest{ + Description: &description, + }, fixture.updateHandler.Handle) + require.Equal(t, http.StatusNotFound, rec.Code, rec.Body.String()) + assert.JSONEq(t, `{"source":"","message":"Could not find SKU with the specified ID","data":null}`, rec.Body.String()) + assert.Empty(t, fixture.requests) + }) + +} + +func TestDeleteSkuHandler(t *testing.T) { + t.Run("proxies delete", func(t *testing.T) { + fixture := newSkuManagementFixture(t, []string{authz.ProviderAdminRole}) + + rec := fixture.request(t, http.MethodDelete, "sku-1", nil, fixture.deleteHandler.Handle) + require.Equal(t, http.StatusNoContent, rec.Code, rec.Body.String()) + require.Len(t, fixture.requests, 1) + assert.Equal(t, corev1.Forge_DeleteSku_FullMethodName, fixture.requests[0].FullMethod) + + var coreReq corev1.SkuIdList + require.NoError(t, protojson.Unmarshal(fixture.requests[0].RequestJSON, &coreReq)) + assert.Equal(t, []string{"sku-1"}, coreReq.Ids) + _, err := cdbm.NewSkuDAO(fixture.deleteHandler.dbSession).Get(context.Background(), nil, "sku-1") + assert.ErrorIs(t, err, cdb.ErrDoesNotExist) + }) + + t.Run("rejects SKU with associated machines", func(t *testing.T) { + fixture := newSkuManagementFixture(t, []string{authz.ProviderAdminRole}) + skuDAO := cdbm.NewSkuDAO(fixture.deleteHandler.dbSession) + _, err := skuDAO.Update(context.Background(), nil, cdbm.SkuUpdateInput{ + SkuID: "sku-1", + AssociatedMachineIds: []string{"machine-1"}, + }) + require.NoError(t, err) + + rec := fixture.request(t, http.MethodDelete, "sku-1", nil, fixture.deleteHandler.Handle) + require.Equal(t, http.StatusBadRequest, rec.Code, rec.Body.String()) + assert.JSONEq(t, `{"source":"","message":"SKU is associated with machines and cannot be deleted","data":null}`, rec.Body.String()) + assert.Empty(t, fixture.requests) + + saved, err := skuDAO.Get(context.Background(), nil, "sku-1") + require.NoError(t, err) + assert.Equal(t, []string{"machine-1"}, saved.AssociatedMachineIds) + }) + + t.Run("removes stale record when Core returns not found", func(t *testing.T) { + deleteErr := tp.NewApplicationErrorWithCause( + "SKU not found", + swe.ErrTypeNICoObjectNotFound, + status.Error(codes.NotFound, "SKU not found"), + ) + fixture := newSkuManagementFixtureWithOptions(t, []string{authz.ProviderAdminRole}, skuManagementFixtureOptions{ + deleteError: deleteErr, + }) + + rec := fixture.request(t, http.MethodDelete, "sku-1", nil, fixture.deleteHandler.Handle) + require.Equal(t, http.StatusNoContent, rec.Code, rec.Body.String()) + require.Len(t, fixture.requests, 1) + assert.Equal(t, corev1.Forge_DeleteSku_FullMethodName, fixture.requests[0].FullMethod) + _, err := cdbm.NewSkuDAO(fixture.deleteHandler.dbSession).Get(context.Background(), nil, "sku-1") + assert.ErrorIs(t, err, cdb.ErrDoesNotExist) + }) + + t.Run("preserves record when Core delete fails", func(t *testing.T) { + deleteErr := tp.NewApplicationErrorWithCause( + "Core unavailable", + swe.ErrTypeNICoUnavailable, + status.Error(codes.Unavailable, "Core unavailable"), + ) + fixture := newSkuManagementFixtureWithOptions(t, []string{authz.ProviderAdminRole}, skuManagementFixtureOptions{ + deleteError: deleteErr, + }) + + rec := fixture.request(t, http.MethodDelete, "sku-1", nil, fixture.deleteHandler.Handle) + require.Equal(t, http.StatusServiceUnavailable, rec.Code, rec.Body.String()) + assert.Contains(t, rec.Body.String(), "Core unavailable") + require.Len(t, fixture.requests, 1) + assert.Equal(t, corev1.Forge_DeleteSku_FullMethodName, fixture.requests[0].FullMethod) + _, err := cdbm.NewSkuDAO(fixture.deleteHandler.dbSession).Get(context.Background(), nil, "sku-1") + assert.NoError(t, err) + }) + + t.Run("returns not found for unsaved SKU", func(t *testing.T) { + fixture := newSkuManagementFixture(t, []string{authz.ProviderAdminRole}) + + rec := fixture.request(t, http.MethodDelete, "missing-sku", nil, fixture.deleteHandler.Handle) + require.Equal(t, http.StatusNotFound, rec.Code, rec.Body.String()) + assert.JSONEq(t, `{"source":"","message":"Could not find SKU with the specified ID","data":null}`, rec.Body.String()) + assert.Empty(t, fixture.requests) + }) +} + +type skuManagementFixture struct { + org string + siteID string + user *cdbm.User + createHandler CreateSkuHandler + updateHandler UpdateSkuHandler + deleteHandler DeleteSkuHandler + requests []coreproxy.Request +} + +type skuManagementFixtureOptions struct { + findResponse *corev1.SkuList + findError error + createError error + replaceResponse *corev1.Sku + replaceError error + updateMetadataError error + deleteError error + skipPersistedSKU bool + afterUpdateMetadata func(*cdb.Session) +} + +func newSkuManagementFixture(t *testing.T, roles []string) *skuManagementFixture { + return newSkuManagementFixtureWithOptions(t, roles, skuManagementFixtureOptions{}) +} + +func newSkuManagementFixtureWithOptions(t *testing.T, roles []string, options skuManagementFixtureOptions) *skuManagementFixture { + t.Helper() + dbSession := common.TestInitDB(t) + t.Cleanup(dbSession.Close) + common.TestSetupSchema(t, dbSession) + require.NoError(t, dbSession.DB.ResetModel(context.Background(), (*cdbm.SKU)(nil))) + + org := "test-org" + user := common.TestBuildUser(t, dbSession, uuid.NewString(), org, roles) + ip := common.TestBuildInfrastructureProvider(t, dbSession, "Test Provider", org, user) + site := common.TestBuildSite(t, dbSession, ip, "Test Site", user) + sDAO := cdbm.NewSiteDAO(dbSession) + _, err := sDAO.Update(context.Background(), nil, cdbm.SiteUpdateInput{ + SiteID: site.ID, + Status: cutil.GetPtr(cdbm.SiteStatusRegistered), + }) + require.NoError(t, err) + if !options.skipPersistedSKU { + skuDAO := cdbm.NewSkuDAO(dbSession) + _, err = skuDAO.Create(context.Background(), nil, cdbm.SkuCreateInput{ + SkuID: "sku-1", + SiteID: site.ID, + SchemaVersion: 4, + }) + require.NoError(t, err) + } + + fixture := &skuManagementFixture{org: org, siteID: site.ID.String(), user: user} + client := &tmocks.Client{} + existing := existingSkuProto() + if options.createError != nil { + fixture.addWorkflowError(client, corev1.Forge_CreateSku_FullMethodName, options.createError) + } else { + fixture.addWorkflow(t, client, corev1.Forge_CreateSku_FullMethodName, &corev1.SkuIdList{Ids: []string{"sku-1"}}) + } + if options.findError != nil { + fixture.addWorkflowError(client, corev1.Forge_FindSkusByIds_FullMethodName, options.findError) + } else { + findResponse := options.findResponse + if findResponse == nil { + findResponse = &corev1.SkuList{Skus: []*corev1.Sku{existing}} + } + fixture.addWorkflow(t, client, corev1.Forge_FindSkusByIds_FullMethodName, findResponse) + } + replaceResponse := options.replaceResponse + if replaceResponse == nil { + replaceResponse = existing + } + if options.replaceError != nil { + fixture.addWorkflowError(client, corev1.Forge_ReplaceSku_FullMethodName, options.replaceError) + } else { + fixture.addWorkflow(t, client, corev1.Forge_ReplaceSku_FullMethodName, replaceResponse) + } + if options.updateMetadataError != nil { + fixture.addWorkflowError(client, corev1.Forge_UpdateSkuMetadata_FullMethodName, options.updateMetadataError) + } else { + fixture.addWorkflow(t, client, corev1.Forge_UpdateSkuMetadata_FullMethodName, nil, func() { + if options.afterUpdateMetadata != nil { + options.afterUpdateMetadata(dbSession) + } + }) + } + if options.deleteError != nil { + fixture.addWorkflowError(client, corev1.Forge_DeleteSku_FullMethodName, options.deleteError) + } else { + fixture.addWorkflow(t, client, corev1.Forge_DeleteSku_FullMethodName, nil) + } + + scp := sc.NewClientPool(nil) + scp.IDClientMap[site.ID.String()] = client + fixture.createHandler = NewCreateSkuHandler(dbSession, scp) + fixture.updateHandler = NewUpdateSkuHandler(dbSession, scp) + fixture.deleteHandler = NewDeleteSkuHandler(dbSession, scp) + return fixture +} + +func (f *skuManagementFixture) addWorkflowError(client *tmocks.Client, method string, getErr error) { + run := &tmocks.WorkflowRun{} + run.On("Get", mock.Anything, mock.Anything).Return(getErr) + client.On( + "ExecuteWorkflow", + mock.Anything, + mock.Anything, + coreproxy.WorkflowName, + mock.MatchedBy(func(req coreproxy.Request) bool { return req.FullMethod == method }), + ).Run(func(args mock.Arguments) { + f.requests = append(f.requests, args.Get(3).(coreproxy.Request)) + }).Return(run, nil).Maybe() +} + +func (f *skuManagementFixture) addWorkflow(t *testing.T, client *tmocks.Client, method string, response proto.Message, afterGet ...func()) { + t.Helper() + run := &tmocks.WorkflowRun{} + var responseJSON []byte + if response != nil { + var err error + responseJSON, err = protojson.Marshal(response) + require.NoError(t, err) + } + run.On("Get", mock.Anything, mock.Anything).Run(func(args mock.Arguments) { + out, ok := args.Get(1).(*coreproxy.Response) + require.True(t, ok) + out.ResponseJSON = responseJSON + for _, callback := range afterGet { + callback() + } + }).Return(nil) + client.On( + "ExecuteWorkflow", + mock.Anything, + mock.Anything, + coreproxy.WorkflowName, + mock.MatchedBy(func(req coreproxy.Request) bool { return req.FullMethod == method }), + ).Run(func(args mock.Arguments) { + f.requests = append(f.requests, args.Get(3).(coreproxy.Request)) + }).Return(run, nil).Maybe() +} + +func (f *skuManagementFixture) request(t *testing.T, method, skuID string, body any, handler func(echo.Context) error) *httptest.ResponseRecorder { + t.Helper() + requestJSON, err := json.Marshal(body) + require.NoError(t, err) + req := httptest.NewRequest(method, "/", strings.NewReader(string(requestJSON))) + req.Header.Set(echo.HeaderContentType, echo.MIMEApplicationJSON) + rec := httptest.NewRecorder() + ec := echo.New().NewContext(req, rec) + ec.SetParamNames("orgName", "id") + ec.SetParamValues(f.org, skuID) + ec.Set("user", f.user) + require.NoError(t, handler(ec)) + return rec +} + +func validSkuCreateRequest(siteID string) model.APISkuCreateRequest { + deviceType := "gpu-server" + return model.APISkuCreateRequest{ + SiteID: siteID, + ID: "sku-1", + Description: cutil.GetPtr("test SKU"), + DeviceType: &deviceType, + Components: &model.APISkuComponents{ + Chassis: &model.APISkuChassis{ + Vendor: "NVIDIA", + Model: "DGX H100", + Architecture: "x86_64", + }, + Storage: []model.APISkuStorage{{ + Model: "informational-model", + Count: 2, + MinSizeMiB: cutil.GetPtr(uint32(3_600_000)), + MaxSizeMiB: cutil.GetPtr(uint32(3_900_000)), + PciPatterns: []string{`^/devices/pci.*nvme[0-1]$`}, + }}, + }, + } +} + +func existingSkuProto() *corev1.Sku { + description := "old description" + deviceType := "gpu-server" + return &corev1.Sku{ + Id: "sku-1", + Description: &description, + SchemaVersion: 4, + DeviceType: &deviceType, + Components: &corev1.SkuComponents{ + Chassis: &corev1.SkuComponentChassis{ + Vendor: "NVIDIA", + Model: "existing chassis", + Architecture: "x86_64", + }, + }, + } +} diff --git a/rest-api/api/pkg/api/model/expectedmachine_test.go b/rest-api/api/pkg/api/model/expectedmachine_test.go index c004875c79..e5f0153e6a 100644 --- a/rest-api/api/pkg/api/model/expectedmachine_test.go +++ b/rest-api/api/pkg/api/model/expectedmachine_test.go @@ -1145,9 +1145,9 @@ func TestNewAPIExpectedMachineWithSkuComponents(t *testing.T) { // Validate Storage components assert.Len(t, apiEM.Sku.Components.Storage, 1) - assert.Equal(t, "Samsung", apiEM.Sku.Components.Storage[0].Vendor) + assert.Equal(t, cutil.GetPtr("Samsung"), apiEM.Sku.Components.Storage[0].Vendor) assert.Equal(t, "PM9A3", apiEM.Sku.Components.Storage[0].Model) - assert.Equal(t, uint32(3840000), apiEM.Sku.Components.Storage[0].CapacityMb) + assert.Equal(t, cutil.GetPtr(uint32(3840000)), apiEM.Sku.Components.Storage[0].CapacityMb) assert.Equal(t, uint32(4), apiEM.Sku.Components.Storage[0].Count) // Validate Chassis component @@ -1291,8 +1291,8 @@ func TestNewAPIExpectedMachineWithSkuComponents(t *testing.T) { // Validate multiple Storage components assert.Len(t, apiEM.Sku.Components.Storage, 2) - assert.Equal(t, "Samsung", apiEM.Sku.Components.Storage[0].Vendor) - assert.Equal(t, "Intel", apiEM.Sku.Components.Storage[1].Vendor) + assert.Equal(t, cutil.GetPtr("Samsung"), apiEM.Sku.Components.Storage[0].Vendor) + assert.Equal(t, cutil.GetPtr("Intel"), apiEM.Sku.Components.Storage[1].Vendor) }, }, { diff --git a/rest-api/api/pkg/api/model/sku.go b/rest-api/api/pkg/api/model/sku.go index 7491c07dec..c4445fb746 100644 --- a/rest-api/api/pkg/api/model/sku.go +++ b/rest-api/api/pkg/api/model/sku.go @@ -4,10 +4,14 @@ package model import ( + "fmt" "time" cdbm "github.com/NVIDIA/infra-controller/rest-api/db/pkg/db/model" corev1 "github.com/NVIDIA/infra-controller/rest-api/proto/core/gen/v1" + validation "github.com/go-ozzo/ozzo-validation/v4" + validationis "github.com/go-ozzo/ozzo-validation/v4/is" + "google.golang.org/protobuf/proto" ) // APISku is the data structure to capture API representation of a SKU @@ -16,6 +20,12 @@ type APISku struct { ID string `json:"id"` // SiteID is the ID of the Site this SKU belongs to SiteID string `json:"siteId"` + // Description is the human-readable SKU description + Description string `json:"description"` + // SchemaVersion is the Core SKU schema version when known. + // When creating a new SKU or updating the components of an existing SKU, + // the schema version must be the current schema version. + SchemaVersion uint32 `json:"schemaVersion"` // DeviceType is the optional device type identifier DeviceType *string `json:"deviceType"` // AssociatedMachineIds is the list of machine IDs associated with this SKU @@ -23,9 +33,147 @@ type APISku struct { // Components contains the hardware components of this SKU Components *APISkuComponents `json:"components"` // Created is the date and time the entity was created - Created time.Time `json:"created"` + Created *time.Time `json:"created"` // Updated is the date and time the entity was last updated - Updated time.Time `json:"updated"` + Updated *time.Time `json:"updated"` +} + +// APISkuCreateRequest is the POST /sku request body. +type APISkuCreateRequest struct { + // SiteID is the Site whose Core service will own the SKU. + SiteID string `json:"siteId"` + // ID is the unique SKU identifier. + ID string `json:"id"` + // Description is the human-readable SKU description. + Description *string `json:"description"` + // DeviceType is the optional device type identifier. + DeviceType *string `json:"deviceType,omitempty"` + // Components is the expected hardware configuration. + Components *APISkuComponents `json:"components"` +} + +// APISkuUpdateRequest is the PATCH /sku/:id request body. +type APISkuUpdateRequest struct { + // SkuID is populated from the request path before proto conversion. + SkuID string `json:"-"` + // Description replaces the description when provided. + Description *string `json:"description,omitempty"` + // DeviceType replaces the device type when provided. + DeviceType *string `json:"deviceType,omitempty"` + // Components replaces the hardware configuration when provided. + Components *APISkuComponents `json:"components,omitempty"` +} + +// CoreSkuSchemaVersion is the Core wire format emitted by REST SKU mutations. +const CoreSkuSchemaVersion uint32 = 5 + +// Validate checks the create request before conversion to Core protobufs. +func (ascr APISkuCreateRequest) Validate() error { + return validation.ValidateStruct(&ascr, + validation.Field(&ascr.SiteID, + validation.Required.Error(validationErrorValueRequired), + validationis.UUID.Error(validationErrorInvalidUUID)), + validation.Field(&ascr.ID, validation.Required.Error(validationErrorValueRequired)), + validation.Field(&ascr.Components, validation.Required.Error(validationErrorValueRequired)), + ) +} + +// ToProto converts a validated create request into Core's single-item SkuList. +func (ascr APISkuCreateRequest) ToProto() *corev1.SkuList { + return &corev1.SkuList{Skus: []*corev1.Sku{{ + Id: ascr.ID, + Description: ascr.Description, + SchemaVersion: CoreSkuSchemaVersion, + DeviceType: ascr.DeviceType, + Components: ascr.Components.ToProto(), + }}} +} + +// Validate checks the update request and requires at least one mutable field. +func (asur APISkuUpdateRequest) Validate() error { + if asur.Description == nil && asur.DeviceType == nil && asur.Components == nil { + return validation.Errors{"request": validation.NewError("validation_required", "at least one mutable field is required")} + } + return validation.ValidateStruct(&asur, + validation.Field(&asur.Components), + ) +} + +// ToMetadataProto converts a validated metadata-only PATCH request. +func (asur APISkuUpdateRequest) ToMetadataProto() *corev1.SkuUpdateMetadataRequest { + return &corev1.SkuUpdateMetadataRequest{ + SkuId: asur.SkuID, + Description: asur.Description, + DeviceType: asur.DeviceType, + } +} + +// ApplyMetadataToProto merges metadata fields into a copy of the current Core SKU. +func (asur APISkuUpdateRequest) ApplyMetadataToProto(current *corev1.Sku) *corev1.Sku { + updated := proto.Clone(current).(*corev1.Sku) + updated.Id = asur.SkuID + if asur.Description != nil { + updated.Description = asur.Description + } + if asur.DeviceType != nil { + updated.DeviceType = asur.DeviceType + } + return updated +} + +// ToReplacementProto converts a component PATCH into Core's current SKU format. +func (asur APISkuUpdateRequest) ToReplacementProto(current *corev1.Sku) *corev1.Sku { + updated := asur.ApplyMetadataToProto(current) + updated.SchemaVersion = CoreSkuSchemaVersion + if asur.Components != nil { + updated.Components = asur.Components.ToProto() + } + return updated +} + +// NewAPISkuFromProto converts a Core SKU into the REST API representation. +func NewAPISkuFromProto(sku *corev1.Sku, siteID string) *APISku { + if sku == nil { + return nil + } + response := &APISku{ + ID: sku.Id, + SiteID: siteID, + Description: sku.GetDescription(), + SchemaVersion: sku.SchemaVersion, + DeviceType: sku.DeviceType, + AssociatedMachineIds: []string{}, + Components: NewAPISkuComponents(sku.Components), + } + for _, machineID := range sku.AssociatedMachineIds { + id := machineID.GetId() + if id != "" { + response.AssociatedMachineIds = append(response.AssociatedMachineIds, id) + } + } + if sku.Created != nil { + created := sku.Created.AsTime() + response.Created = &created + } + return response +} + +// NewAPISkuFromCreateRequest builds the best-known response +// after Core accepted a create request but the post-create read failed. +func NewAPISkuFromCreateRequest(req APISkuCreateRequest, skuID, siteID string) *APISku { + description := "" + if req.Description != nil { + description = *req.Description + } + return &APISku{ + ID: skuID, + SiteID: siteID, + Description: description, + SchemaVersion: CoreSkuSchemaVersion, + DeviceType: req.DeviceType, + AssociatedMachineIds: []string{}, + Components: NewAPISkuComponents(req.Components.ToProto()), + } } // NewAPISku accepts a DB layer SKU object and returns an API layer object @@ -37,10 +185,12 @@ func NewAPISku(dbSku *cdbm.SKU) *APISku { apiSku := &APISku{ ID: dbSku.ID, SiteID: dbSku.SiteID.String(), + Description: dbSku.Description, + SchemaVersion: dbSku.SchemaVersion, DeviceType: dbSku.DeviceType, AssociatedMachineIds: dbSku.AssociatedMachineIds, - Created: dbSku.Created, - Updated: dbSku.Updated, + Created: &dbSku.Created, + Updated: &dbSku.Updated, } // Map SKU Components if available @@ -51,7 +201,6 @@ func NewAPISku(dbSku *cdbm.SKU) *APISku { return apiSku } -// APISkuComponents is the data structure to capture API representation of SKU Components type APISkuComponents struct { // Cpus describes CPU components Cpus []APISkuCpu `json:"cpus"` @@ -71,6 +220,11 @@ type APISkuComponents struct { Tpm *APISkuTpm `json:"tpm"` } +// Validate checks every storage component. +func (c APISkuComponents) Validate() error { + return validation.Validate(c.Storage, validation.Each()) +} + // APISkuCpu represents a CPU component in the SKU type APISkuCpu struct { // Vendor describes the vendor of the CPU @@ -107,14 +261,45 @@ type APISkuMemory struct { // APISkuStorage represents a storage component in the SKU type APISkuStorage struct { - // Vendor describes the vendor of the storage device - Vendor string `json:"vendor"` - // Model describes the model of the storage device + // Vendor participates in storage matching for schema version 4 SKUs. It is + // read-only in REST mutation requests because component mutations use the + // current schema version. + Vendor *string `json:"vendor"` + // Model is informational starting with the 2.1 release. Model string `json:"model"` - // CapacityMb describes the capacity in megabytes - CapacityMb uint32 `json:"capacityMb"` + // CapacityMb participates in storage matching for schema version 4 SKUs. It + // is read-only in REST mutation requests because component mutations use the + // current schema version. + CapacityMb *uint32 `json:"capacityMb"` // Count describes the number of storage devices present Count uint32 `json:"count"` + // MinSizeMiB is the inclusive minimum capacity in mebibytes for each storage device. + // It is only used for matching for SKUs of schema version 5 and onwards. + MinSizeMiB *uint32 `json:"minSizeMiB"` + // MaxSizeMiB is the inclusive maximum capacity in mebibytes for each storage device. + // It is only used for matching for SKUs of schema version 5 and onwards. + MaxSizeMiB *uint32 `json:"maxSizeMiB"` + // PciPatterns contains regular expressions matched against storage PCI paths. + // It is only used for matching for SKUs of schema version 5 and onwards. + PciPatterns []string `json:"pciPatterns"` +} + +// Validate rejects read-only fields and invalid size bounds. +func (s APISkuStorage) Validate() error { + errs := validation.Errors{} + if s.Vendor != nil { + errs["vendor"] = fmt.Errorf("is a read-only SKU storage field") + } + if s.CapacityMb != nil { + errs["capacityMb"] = fmt.Errorf("is a read-only SKU storage field") + } + if s.MinSizeMiB != nil && s.MaxSizeMiB != nil && *s.MinSizeMiB > *s.MaxSizeMiB { + errs["minSizeMiB"] = fmt.Errorf("must be less than or equal to maxSizeMiB") + } + if len(errs) == 0 { + return nil + } + return errs } // APISkuChassis represents the chassis component in the SKU @@ -123,6 +308,8 @@ type APISkuChassis struct { Vendor string `json:"vendor"` // Model describes the model of the chassis Model string `json:"model"` + // Architecture describes the chassis architecture. + Architecture string `json:"architecture"` } // APISkuEthernetDevice represents an ethernet device component in the SKU @@ -133,6 +320,8 @@ type APISkuEthernetDevice struct { Model string `json:"model"` // Count describes the number of ethernet devices present Count uint32 `json:"count"` + // IsConnected reports whether the Ethernet device is connected. + IsConnected bool `json:"isConnected"` } // APISkuInfinibandDevice represents an infiniband device component in the SKU @@ -143,6 +332,8 @@ type APISkuInfinibandDevice struct { Model string `json:"model"` // Count describes the number of infiniband devices present Count uint32 `json:"count"` + // InactiveDevices contains zero-based indexes of inactive devices. + InactiveDevices []uint32 `json:"inactiveDevices"` } // APISkuTpm represents a TPM component in the SKU @@ -153,106 +344,149 @@ type APISkuTpm struct { Version string `json:"version"` } -// NewAPISkuComponents converts proto SkuComponents to API SkuComponents +// NewAPISkuComponents converts proto SkuComponents to API SkuComponents. func NewAPISkuComponents(protoComponents *corev1.SkuComponents) *APISkuComponents { if protoComponents == nil { return nil } - apiComponents := &APISkuComponents{} - - // Map CPU components - if len(protoComponents.Cpus) > 0 { - apiComponents.Cpus = []APISkuCpu{} - for _, cpu := range protoComponents.Cpus { - apiComponents.Cpus = append(apiComponents.Cpus, APISkuCpu{ - Vendor: cpu.Vendor, - Model: cpu.Model, - ThreadCount: cpu.ThreadCount, - Count: cpu.Count, - }) - } + components := &APISkuComponents{} + for _, cpu := range protoComponents.Cpus { + components.Cpus = append(components.Cpus, APISkuCpu{ + Vendor: cpu.Vendor, + Model: cpu.Model, + ThreadCount: cpu.ThreadCount, + Count: cpu.Count, + }) } - - // Map GPU components - if len(protoComponents.Gpus) > 0 { - apiComponents.Gpus = []APISkuGpu{} - for _, gpu := range protoComponents.Gpus { - apiComponents.Gpus = append(apiComponents.Gpus, APISkuGpu{ - Vendor: gpu.Vendor, - Model: gpu.Model, - TotalMemory: gpu.TotalMemory, - Count: gpu.Count, - }) - } + for _, gpu := range protoComponents.Gpus { + components.Gpus = append(components.Gpus, APISkuGpu{ + Vendor: gpu.Vendor, + Model: gpu.Model, + TotalMemory: gpu.TotalMemory, + Count: gpu.Count, + }) } - - // Map Memory components - if len(protoComponents.Memory) > 0 { - apiComponents.Memory = []APISkuMemory{} - for _, mem := range protoComponents.Memory { - apiComponents.Memory = append(apiComponents.Memory, APISkuMemory{ - CapacityMb: mem.CapacityMb, - MemoryType: mem.MemoryType, - Count: mem.Count, - }) - } + for _, memory := range protoComponents.Memory { + components.Memory = append(components.Memory, APISkuMemory{ + CapacityMb: memory.CapacityMb, + MemoryType: memory.MemoryType, + Count: memory.Count, + }) } - - // Map Storage components - if len(protoComponents.Storage) > 0 { - apiComponents.Storage = []APISkuStorage{} - for _, storage := range protoComponents.Storage { - apiComponents.Storage = append(apiComponents.Storage, APISkuStorage{ - Vendor: storage.Vendor, - Model: storage.Model, - CapacityMb: storage.CapacityMb, - Count: storage.Count, - }) - } + for _, storage := range protoComponents.Storage { + vendor := storage.Vendor + capacityMb := storage.CapacityMb + components.Storage = append(components.Storage, APISkuStorage{ + Vendor: &vendor, + Model: storage.Model, + CapacityMb: &capacityMb, + Count: storage.Count, + MinSizeMiB: storage.MinSizeMb, + MaxSizeMiB: storage.MaxSizeMb, + PciPatterns: storage.PciPatterns, + }) } - - // Map Chassis component (single object) if protoComponents.Chassis != nil { - apiComponents.Chassis = &APISkuChassis{ - Vendor: protoComponents.Chassis.Vendor, - Model: protoComponents.Chassis.Model, + components.Chassis = &APISkuChassis{ + Vendor: protoComponents.Chassis.Vendor, + Model: protoComponents.Chassis.Model, + Architecture: protoComponents.Chassis.Architecture, } } - - // Map EthernetDevices components - if len(protoComponents.EthernetDevices) > 0 { - apiComponents.EthernetDevices = []APISkuEthernetDevice{} - for _, ethDev := range protoComponents.EthernetDevices { - apiComponents.EthernetDevices = append(apiComponents.EthernetDevices, APISkuEthernetDevice{ - Vendor: ethDev.Vendor, - Model: ethDev.Model, - Count: ethDev.Count, - }) - } + for _, ethernet := range protoComponents.EthernetDevices { + components.EthernetDevices = append(components.EthernetDevices, APISkuEthernetDevice{ + Vendor: ethernet.Vendor, + Model: ethernet.Model, + Count: ethernet.Count, + IsConnected: ethernet.IsConnected, + }) } - - // Map InfinibandDevices components - if len(protoComponents.InfinibandDevices) > 0 { - apiComponents.InfinibandDevices = []APISkuInfinibandDevice{} - for _, ibDev := range protoComponents.InfinibandDevices { - apiComponents.InfinibandDevices = append(apiComponents.InfinibandDevices, APISkuInfinibandDevice{ - Vendor: ibDev.Vendor, - Model: ibDev.Model, - Count: ibDev.Count, - }) - } + for _, infiniband := range protoComponents.InfinibandDevices { + components.InfinibandDevices = append(components.InfinibandDevices, APISkuInfinibandDevice{ + Vendor: infiniband.Vendor, + Model: infiniband.Model, + Count: infiniband.Count, + InactiveDevices: infiniband.InactiveDevices, + }) } - - // Map Tpm components if protoComponents.Tpm != nil { - apiComponents.Tpm = &APISkuTpm{ + components.Tpm = &APISkuTpm{ Vendor: protoComponents.Tpm.Vendor, Version: protoComponents.Tpm.Version, } } + return components +} - return apiComponents +// ToProto converts API SKU components into the Core protobuf shape. +func (c *APISkuComponents) ToProto() *corev1.SkuComponents { + if c == nil { + return nil + } + components := &corev1.SkuComponents{} + if c.Chassis != nil { + components.Chassis = &corev1.SkuComponentChassis{ + Vendor: c.Chassis.Vendor, + Model: c.Chassis.Model, + Architecture: c.Chassis.Architecture, + } + } + for _, cpu := range c.Cpus { + components.Cpus = append(components.Cpus, &corev1.SkuComponentCpu{ + Vendor: cpu.Vendor, + Model: cpu.Model, + ThreadCount: cpu.ThreadCount, + Count: cpu.Count, + }) + } + for _, gpu := range c.Gpus { + components.Gpus = append(components.Gpus, &corev1.SkuComponentGpu{ + Vendor: gpu.Vendor, + Model: gpu.Model, + TotalMemory: gpu.TotalMemory, + Count: gpu.Count, + }) + } + for _, memory := range c.Memory { + components.Memory = append(components.Memory, &corev1.SkuComponentMemory{ + CapacityMb: memory.CapacityMb, + MemoryType: memory.MemoryType, + Count: memory.Count, + }) + } + for _, storage := range c.Storage { + components.Storage = append(components.Storage, &corev1.SkuComponentStorage{ + Model: storage.Model, + Count: storage.Count, + MinSizeMb: storage.MinSizeMiB, + MaxSizeMb: storage.MaxSizeMiB, + PciPatterns: storage.PciPatterns, + }) + } + for _, ethernet := range c.EthernetDevices { + components.EthernetDevices = append(components.EthernetDevices, &corev1.SkuComponentEthernetDevices{ + Vendor: ethernet.Vendor, + Model: ethernet.Model, + Count: ethernet.Count, + IsConnected: ethernet.IsConnected, + }) + } + for _, infiniband := range c.InfinibandDevices { + components.InfinibandDevices = append(components.InfinibandDevices, &corev1.SkuComponentInfinibandDevices{ + Vendor: infiniband.Vendor, + Model: infiniband.Model, + Count: infiniband.Count, + InactiveDevices: infiniband.InactiveDevices, + }) + } + if c.Tpm != nil { + components.Tpm = &corev1.SkuComponentTpm{ + Vendor: c.Tpm.Vendor, + Version: c.Tpm.Version, + } + } + return components } // APISkuSummary is the data structure to capture summary of a SKU diff --git a/rest-api/api/pkg/api/model/sku_test.go b/rest-api/api/pkg/api/model/sku_test.go index 24bfaa7346..31f6106796 100644 --- a/rest-api/api/pkg/api/model/sku_test.go +++ b/rest-api/api/pkg/api/model/sku_test.go @@ -4,15 +4,18 @@ package model import ( + "encoding/json" "fmt" "reflect" "testing" "time" + cutil "github.com/NVIDIA/infra-controller/rest-api/common/pkg/util" cdbm "github.com/NVIDIA/infra-controller/rest-api/db/pkg/db/model" corev1 "github.com/NVIDIA/infra-controller/rest-api/proto/core/gen/v1" "github.com/google/uuid" "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" ) func TestNewAPISku(t *testing.T) { @@ -21,6 +24,7 @@ func TestNewAPISku(t *testing.T) { } siteID := uuid.New() + description := "Test SKU description" deviceType := "test-device-type" associatedMachineIds := []string{"machine-1", "machine-2"} createdTime := time.Now() @@ -30,6 +34,8 @@ func TestNewAPISku(t *testing.T) { dbSku := &cdbm.SKU{ ID: "test-sku-id", SiteID: siteID, + Description: description, + SchemaVersion: 5, DeviceType: &deviceType, AssociatedMachineIds: associatedMachineIds, Created: createdTime, @@ -52,10 +58,12 @@ func TestNewAPISku(t *testing.T) { want: &APISku{ ID: dbSku.ID, SiteID: siteID.String(), + Description: description, + SchemaVersion: 5, DeviceType: &deviceType, AssociatedMachineIds: associatedMachineIds, - Created: createdTime, - Updated: updatedTime, + Created: &createdTime, + Updated: &updatedTime, }, }, { @@ -83,6 +91,8 @@ func TestNewAPISku(t *testing.T) { // Compare basic fields assert.Equal(t, tt.want.ID, got.ID) assert.Equal(t, tt.want.SiteID, got.SiteID) + assert.Equal(t, tt.want.Description, got.Description) + assert.Equal(t, tt.want.SchemaVersion, got.SchemaVersion) assert.Equal(t, tt.want.DeviceType, got.DeviceType) assert.Equal(t, tt.want.AssociatedMachineIds, got.AssociatedMachineIds) assert.Equal(t, tt.want.Created, got.Created) @@ -114,6 +124,9 @@ func TestNewAPISkuWithFullComponents(t *testing.T) { deviceType := "gpu-server" createdTime := time.Now() updatedTime := time.Now() + minStorageSizeMb := uint32(7_600_000) + maxStorageSizeMb := uint32(7_800_000) + pciPattern := `^/devices/pci.*nvme[0-3]$` t.Run("complete GPU server with all component types", func(t *testing.T) { dbSku := &cdbm.SKU{ @@ -148,10 +161,13 @@ func TestNewAPISkuWithFullComponents(t *testing.T) { }, Storage: []*corev1.SkuComponentStorage{ { - Vendor: "Samsung", - Model: "PM9A3", - CapacityMb: 7680000, - Count: 4, + Vendor: "Samsung", + Model: "PM9A3", + CapacityMb: 7680000, + Count: 4, + MinSizeMb: &minStorageSizeMb, + MaxSizeMb: &maxStorageSizeMb, + PciPatterns: []string{pciPattern}, }, }, Chassis: &corev1.SkuComponentChassis{ @@ -199,10 +215,13 @@ func TestNewAPISkuWithFullComponents(t *testing.T) { // Validate Storage assert.Len(t, result.Components.Storage, 1) - assert.Equal(t, "Samsung", result.Components.Storage[0].Vendor) + assert.Equal(t, cutil.GetPtr("Samsung"), result.Components.Storage[0].Vendor) assert.Equal(t, "PM9A3", result.Components.Storage[0].Model) - assert.Equal(t, uint32(7680000), result.Components.Storage[0].CapacityMb) + assert.Equal(t, cutil.GetPtr(uint32(7680000)), result.Components.Storage[0].CapacityMb) assert.Equal(t, uint32(4), result.Components.Storage[0].Count) + assert.Equal(t, &minStorageSizeMb, result.Components.Storage[0].MinSizeMiB) + assert.Equal(t, &maxStorageSizeMb, result.Components.Storage[0].MaxSizeMiB) + assert.Equal(t, []string{pciPattern}, result.Components.Storage[0].PciPatterns) // Validate Chassis assert.NotNil(t, result.Components.Chassis) @@ -328,15 +347,15 @@ func TestNewAPISkuWithFullComponents(t *testing.T) { assert.Len(t, result.Components.Storage, 2) // Validate first storage type - assert.Equal(t, "Samsung", result.Components.Storage[0].Vendor) + assert.Equal(t, cutil.GetPtr("Samsung"), result.Components.Storage[0].Vendor) assert.Equal(t, "PM1733", result.Components.Storage[0].Model) - assert.Equal(t, uint32(15360000), result.Components.Storage[0].CapacityMb) + assert.Equal(t, cutil.GetPtr(uint32(15360000)), result.Components.Storage[0].CapacityMb) assert.Equal(t, uint32(24), result.Components.Storage[0].Count) // Validate second storage type - assert.Equal(t, "Intel", result.Components.Storage[1].Vendor) + assert.Equal(t, cutil.GetPtr("Intel"), result.Components.Storage[1].Vendor) assert.Equal(t, "P5520", result.Components.Storage[1].Model) - assert.Equal(t, uint32(7680000), result.Components.Storage[1].CapacityMb) + assert.Equal(t, cutil.GetPtr(uint32(7680000)), result.Components.Storage[1].CapacityMb) assert.Equal(t, uint32(4), result.Components.Storage[1].Count) // Validate memory configuration @@ -483,7 +502,7 @@ func TestNewAPISkuWithFullComponents(t *testing.T) { // Validate compact storage assert.Len(t, result.Components.Storage, 1) - assert.Equal(t, uint32(960000), result.Components.Storage[0].CapacityMb) + assert.Equal(t, cutil.GetPtr(uint32(960000)), result.Components.Storage[0].CapacityMb) }) } @@ -823,7 +842,7 @@ func TestAPISkuComponentsWithSpecialValues(t *testing.T) { assert.Equal(t, uint32(32768), result.Components.Memory[0].CapacityMb) assert.Len(t, result.Components.Storage, 3) - assert.Equal(t, "Samsung", result.Components.Storage[0].Vendor) + assert.Equal(t, cutil.GetPtr("Samsung"), result.Components.Storage[0].Vendor) assert.NotNil(t, result.Components.Tpm) assert.Equal(t, "Infineon", result.Components.Tpm.Vendor) @@ -876,3 +895,309 @@ func TestAPISkuSummaryEdgeCases(t *testing.T) { assert.Equal(t, specialType, *result.DeviceType) }) } + +func TestAPISkuCreateRequest(t *testing.T) { + t.Run("converts to proto", func(t *testing.T) { + deviceType := "gpu-server" + req := APISkuCreateRequest{ + SiteID: uuid.NewString(), + ID: "dgx-h100", + Description: cutil.GetPtr("DGX H100"), + DeviceType: &deviceType, + Components: testAPISkuComponents(), + } + + require.NoError(t, req.Validate()) + proto := req.ToProto() + require.Len(t, proto.Skus, 1) + sku := proto.Skus[0] + assert.Equal(t, "dgx-h100", sku.Id) + assert.Equal(t, "DGX H100", sku.GetDescription()) + assert.Equal(t, CoreSkuSchemaVersion, sku.SchemaVersion) + assert.Equal(t, "gpu-server", sku.GetDeviceType()) + require.NotNil(t, sku.Components) + require.NotNil(t, sku.Components.Chassis) + assert.Equal(t, "x86_64", sku.Components.Chassis.Architecture) + require.Len(t, sku.Components.InfinibandDevices, 1) + assert.Equal(t, []uint32{1}, sku.Components.InfinibandDevices[0].InactiveDevices) + require.Len(t, sku.Components.Storage, 1) + assert.Empty(t, sku.Components.Storage[0].Vendor) + assert.Zero(t, sku.Components.Storage[0].CapacityMb) + assert.Equal(t, "informational-model", sku.Components.Storage[0].Model) + assert.Equal(t, cutil.GetPtr(uint32(3_600_000)), sku.Components.Storage[0].MinSizeMb) + assert.Equal(t, cutil.GetPtr(uint32(3_900_000)), sku.Components.Storage[0].MaxSizeMb) + assert.Equal(t, []string{`^/devices/pci.*nvme0$`}, sku.Components.Storage[0].PciPatterns) + }) + + t.Run("preserves omitted description in proto", func(t *testing.T) { + req := APISkuCreateRequest{ + SiteID: uuid.NewString(), + ID: "dgx-h100", + Components: testAPISkuComponents(), + } + + require.NoError(t, req.Validate()) + proto := req.ToProto() + require.Len(t, proto.Skus, 1) + assert.Nil(t, proto.Skus[0].Description) + }) + + t.Run("validates required fields", func(t *testing.T) { + req := APISkuCreateRequest{SiteID: "not-a-uuid", ID: ""} + assert.Error(t, req.Validate()) + }) + + t.Run("rejects inverted storage size range", func(t *testing.T) { + components := testAPISkuComponents() + components.Storage[0].MinSizeMiB = cutil.GetPtr(uint32(4_000_000)) + components.Storage[0].MaxSizeMiB = cutil.GetPtr(uint32(3_800_000)) + req := APISkuCreateRequest{ + SiteID: uuid.NewString(), + ID: "dgx-h100", + Components: components, + } + + err := req.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "minSizeMiB: must be less than or equal to maxSizeMiB") + }) + + for _, test := range []struct { + name string + min *uint32 + max *uint32 + }{ + {name: "accepts equal storage size bounds", min: cutil.GetPtr(uint32(3_800_000)), max: cutil.GetPtr(uint32(3_800_000))}, + {name: "accepts only minimum size", min: cutil.GetPtr(uint32(3_800_000))}, + {name: "accepts only maximum size", max: cutil.GetPtr(uint32(4_000_000))}, + } { + t.Run(test.name, func(t *testing.T) { + components := testAPISkuComponents() + components.Storage[0].MinSizeMiB = test.min + components.Storage[0].MaxSizeMiB = test.max + req := APISkuCreateRequest{ + SiteID: uuid.NewString(), + ID: "dgx-h100", + Components: components, + } + + assert.NoError(t, req.Validate()) + }) + } +} + +func TestNewAPISkuFromCreateRequest_OmittedDescription(t *testing.T) { + response := NewAPISkuFromCreateRequest(APISkuCreateRequest{}, "dgx-h100", uuid.NewString()) + + assert.Empty(t, response.Description) +} + +func TestAPISku_MarshalJSON(t *testing.T) { + response := APISku{ + Components: &APISkuComponents{ + Storage: []APISkuStorage{{}}, + }, + } + + encoded, err := json.Marshal(response) + require.NoError(t, err) + assert.JSONEq(t, `{ + "id":"", + "siteId":"", + "description":"", + "schemaVersion":0, + "deviceType":null, + "associatedMachineIds":null, + "components":{ + "cpus":null, + "gpus":null, + "memory":null, + "storage":[{ + "vendor":null, + "model":"", + "capacityMb":null, + "count":0, + "minSizeMiB":null, + "maxSizeMiB":null, + "pciPatterns":null + }], + "chassis":null, + "ethernetDevices":null, + "infinibandDevices":null, + "tpm":null + }, + "created":null, + "updated":null + }`, string(encoded)) +} + +func TestAPISkuStorage_Validate(t *testing.T) { + t.Run("accepts schema version 5 fields", func(t *testing.T) { + var storage APISkuStorage + err := json.Unmarshal([]byte(`{ + "model":"informational-model", + "count":2, + "minSizeMiB":3600000, + "maxSizeMiB":3900000, + "pciPatterns":["^/devices/pci.*nvme[0-1]$"] + }`), &storage) + require.NoError(t, err) + require.NoError(t, storage.Validate()) + assert.Equal(t, "informational-model", storage.Model) + assert.Equal(t, uint32(2), storage.Count) + assert.Equal(t, cutil.GetPtr(uint32(3_600_000)), storage.MinSizeMiB) + assert.Equal(t, cutil.GetPtr(uint32(3_900_000)), storage.MaxSizeMiB) + assert.Equal(t, []string{`^/devices/pci.*nvme[0-1]$`}, storage.PciPatterns) + }) + + for name, body := range map[string]string{ + "rejects read-only vendor": `{"model":"legacy","count":1,"vendor":""}`, + "rejects read-only capacityMb": `{"model":"legacy","count":1,"capacityMb":0}`, + } { + t.Run(name, func(t *testing.T) { + var storage APISkuStorage + err := json.Unmarshal([]byte(body), &storage) + require.NoError(t, err) + + err = storage.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "read-only") + }) + } +} + +func TestAPISkuRequests_UnmarshalJSON(t *testing.T) { + t.Run("create", func(t *testing.T) { + var request APISkuCreateRequest + err := json.Unmarshal([]byte(`{ + "siteId":"60189e9c-7d12-438c-b9ca-6998d9c364b1", + "id":"sku-1", + "description":"description", + "deviceType":"gpu", + "components":{} + }`), &request) + + require.NoError(t, err) + assert.Equal(t, "60189e9c-7d12-438c-b9ca-6998d9c364b1", request.SiteID) + assert.Equal(t, "sku-1", request.ID) + assert.Equal(t, "description", *request.Description) + assert.Equal(t, "gpu", *request.DeviceType) + require.NotNil(t, request.Components) + }) + + t.Run("update", func(t *testing.T) { + var request APISkuUpdateRequest + err := json.Unmarshal([]byte(`{ + "description":"description", + "deviceType":"gpu", + "components":{} + }`), &request) + + require.NoError(t, err) + assert.Equal(t, "description", *request.Description) + assert.Equal(t, "gpu", *request.DeviceType) + require.NotNil(t, request.Components) + }) + +} + +func TestAPISkuUpdateRequest(t *testing.T) { + t.Run("converts metadata update to proto", func(t *testing.T) { + description := "updated description" + req := APISkuUpdateRequest{ + SkuID: "dgx-h100", + Description: &description, + } + existing := &corev1.Sku{ + Id: "dgx-h100", + Description: cutil.GetPtr("old description"), + SchemaVersion: 4, + DeviceType: cutil.GetPtr("gpu-server"), + Components: testAPISkuComponents().ToProto(), + AssociatedMachineIds: []*corev1.MachineId{{Id: "machine-1"}}, + } + + require.NoError(t, req.Validate()) + metadata := req.ToMetadataProto() + assert.Equal(t, "dgx-h100", metadata.SkuId) + assert.Equal(t, "updated description", metadata.GetDescription()) + + updated := req.ApplyMetadataToProto(existing) + assert.Equal(t, "updated description", updated.GetDescription()) + assert.Equal(t, uint32(4), updated.SchemaVersion) + assert.Equal(t, "gpu-server", updated.GetDeviceType()) + assert.Equal(t, existing.Components, updated.Components) + assert.Equal(t, existing.AssociatedMachineIds, updated.AssociatedMachineIds) + assert.NotSame(t, existing, updated) + }) + + t.Run("replaces components using current schema version", func(t *testing.T) { + description := "updated description" + req := APISkuUpdateRequest{ + SkuID: "dgx-h100", + Description: &description, + Components: testAPISkuComponents(), + } + existing := &corev1.Sku{ + Id: "dgx-h100", + Description: cutil.GetPtr("old description"), + SchemaVersion: CoreSkuSchemaVersion, + DeviceType: cutil.GetPtr("gpu-server"), + Components: testAPISkuComponents().ToProto(), + AssociatedMachineIds: []*corev1.MachineId{{Id: "machine-1"}}, + } + + require.NoError(t, req.Validate()) + updated := req.ToReplacementProto(existing) + assert.Equal(t, "updated description", updated.GetDescription()) + assert.Equal(t, CoreSkuSchemaVersion, updated.SchemaVersion) + assert.Equal(t, req.Components.ToProto(), updated.Components) + assert.Equal(t, existing.AssociatedMachineIds, updated.AssociatedMachineIds) + assert.NotSame(t, existing, updated) + }) + + t.Run("requires a mutable field", func(t *testing.T) { + req := APISkuUpdateRequest{} + assert.Error(t, req.Validate()) + }) + + t.Run("rejects inverted storage size range", func(t *testing.T) { + components := testAPISkuComponents() + components.Storage[0].MinSizeMiB = cutil.GetPtr(uint32(4_000_000)) + components.Storage[0].MaxSizeMiB = cutil.GetPtr(uint32(3_800_000)) + req := APISkuUpdateRequest{Components: components} + + err := req.Validate() + require.Error(t, err) + assert.Contains(t, err.Error(), "minSizeMiB: must be less than or equal to maxSizeMiB") + }) +} + +func testAPISkuComponents() *APISkuComponents { + return &APISkuComponents{ + Chassis: &APISkuChassis{ + Vendor: "NVIDIA", + Model: "DGX H100", + Architecture: "x86_64", + }, + Cpus: []APISkuCpu{{ + Vendor: "Intel", + Model: "Xeon", + ThreadCount: 112, + Count: 2, + }}, + Storage: []APISkuStorage{{ + Model: "informational-model", + Count: 2, + MinSizeMiB: cutil.GetPtr(uint32(3_600_000)), + MaxSizeMiB: cutil.GetPtr(uint32(3_900_000)), + PciPatterns: []string{`^/devices/pci.*nvme0$`}, + }}, + InfinibandDevices: []APISkuInfinibandDevice{{ + Vendor: "NVIDIA", + Model: "ConnectX-7", + Count: 2, + InactiveDevices: []uint32{1}, + }}, + } +} diff --git a/rest-api/api/pkg/api/routes.go b/rest-api/api/pkg/api/routes.go index c9c5017a54..870c88e5e2 100644 --- a/rest-api/api/pkg/api/routes.go +++ b/rest-api/api/pkg/api/routes.go @@ -979,6 +979,11 @@ func NewAPIRoutes(dbSession *cdb.Session, tc tClient.Client, tnc tClient.Namespa Handler: apiHandler.NewDeleteDpuExtensionServiceVersionHandler(dbSession, tc, scp, cfg), }, // SKU endpoints + { + Path: apiPathPrefix + "/sku", + Method: http.MethodPost, + Handler: apiHandler.NewCreateSkuHandler(dbSession, scp), + }, { Path: apiPathPrefix + "/sku", Method: http.MethodGet, @@ -989,6 +994,16 @@ func NewAPIRoutes(dbSession *cdb.Session, tc tClient.Client, tnc tClient.Namespa Method: http.MethodGet, Handler: apiHandler.NewGetSkuHandler(dbSession, tc, cfg), }, + { + Path: apiPathPrefix + "/sku/:id", + Method: http.MethodPatch, + Handler: apiHandler.NewUpdateSkuHandler(dbSession, scp), + }, + { + Path: apiPathPrefix + "/sku/:id", + Method: http.MethodDelete, + Handler: apiHandler.NewDeleteSkuHandler(dbSession, scp), + }, // Task endpoints (Flow). /rack/task/* and /task/* share get/cancel // handlers; list operations are exposed under /rack/{id}/task and // /tray/{id}/task. diff --git a/rest-api/api/pkg/api/routes_test.go b/rest-api/api/pkg/api/routes_test.go index 31d119691e..e4004d7679 100644 --- a/rest-api/api/pkg/api/routes_test.go +++ b/rest-api/api/pkg/api/routes_test.go @@ -72,7 +72,7 @@ func TestNewAPIRoutes(t *testing.T) { "network-security-group": 5, "machine-validation": 11, "dpu-extension-service": 7, - "sku": 2, + "sku": 5, "task": 2, "rule": 5, "run": 8, @@ -148,7 +148,14 @@ func TestNewAPIRoutes(t *testing.T) { assertRouteExists(t, got, http.MethodGet, ipxeTemplatePath) assertRouteExists(t, got, http.MethodGet, ipxeTemplatePath+"/:id") + skuPath := "/org/:orgName/" + cfg.GetAPIName() + "/sku" + assertRouteExists(t, got, http.MethodPost, skuPath) + assertRouteExists(t, got, http.MethodGet, skuPath) + assertRouteExists(t, got, http.MethodGet, skuPath+"/:id") + assertRouteExists(t, got, http.MethodPatch, skuPath+"/:id") + assertRouteExists(t, got, http.MethodDelete, skuPath+"/:id") runPath := "/org/:orgName/" + cfg.GetAPIName() + "/task/run" + assertRouteExists(t, got, http.MethodPost, runPath) assertRouteExists(t, got, http.MethodGet, runPath) assertRouteExists(t, got, http.MethodGet, runPath+"/:id") diff --git a/rest-api/db/pkg/db/model/sku.go b/rest-api/db/pkg/db/model/sku.go index 5ff5226177..976f363f81 100644 --- a/rest-api/db/pkg/db/model/sku.go +++ b/rest-api/db/pkg/db/model/sku.go @@ -80,29 +80,26 @@ type SKU struct { ID string `bun:"id,pk"` SiteID uuid.UUID `bun:"site_id,type:uuid,notnull"` Site *Site `bun:"rel:belongs-to,join:site_id=id"` - DeviceType *string `bun:"device_type"` // NOTE: can be added once available in nico.proto + Description string `bun:"description,notnull,default:''"` + SchemaVersion uint32 `bun:"schema_version,notnull,default:0"` + DeviceType *string `bun:"device_type"` Components *SkuComponents `bun:"components,type:jsonb"` AssociatedMachineIds []string `bun:"associated_machines,type:text[],default:'{}'"` Created time.Time `bun:"created,nullzero,notnull,default:current_timestamp"` Updated time.Time `bun:"updated,nullzero,notnull,default:current_timestamp"` } -// ToProto converts this SKU into its workflow proto representation. -// Used as the canonical entity-to-proto conversion; SKU has no API -// Create/Update request shapes (the Site is the source of truth for -// SKU data, so the cloud API exposes read-only handlers), so this -// receiver is the only `ToProto` the model carries. +// ToProto converts the REST database projection into its Core SKU representation. // -// Fields that exist on the proto but not on the DB row -// (`Description`, the proto-level `Created` timestamp, `SchemaVersion`) -// are intentionally omitted — the DB does not carry the data to fill -// them, and no current caller depends on them. `SiteID` is on the -// model but not on the proto, so it is also dropped on the wire (the -// receiving side reconstructs it from context, mirroring `FromProto`). +// The Core-level Created timestamp is omitted because the REST projection does +// not store it. SiteID is also omitted because it is not carried by the Core SKU +// message; callers supply it separately to FromProto. func (sk *SKU) ToProto() *corev1.Sku { proto := &corev1.Sku{ - Id: sk.ID, - DeviceType: sk.DeviceType, + Id: sk.ID, + Description: &sk.Description, + SchemaVersion: sk.SchemaVersion, + DeviceType: sk.DeviceType, } if sk.Components != nil { proto.Components = sk.Components.SkuComponents @@ -117,13 +114,16 @@ func (sk *SKU) ToProto() *corev1.Sku { return proto } -// FromProto populates this SKU from a workflow proto reported by a Site. -// A nil proto is a no-op. This is the inverse of `ToProto`; `siteID` -// is supplied by the caller because it isn't carried on the proto. +// FromProto populates the REST database projection from a Core SKU returned by +// inventory synchronization or an immediate REST mutation. A nil proto is a +// no-op. This is the inverse of ToProto; siteID is supplied by the caller +// because it is not carried on the Core SKU message. // // Field-level contract: // - `sk.ID` is overwritten with `proto.Id` (callers pre-validate // non-empty IDs at the activity layer). +// - `Description` uses the protobuf default when absent, so a nil Core +// description clears any stale REST projection value. // - `Components` mirrors the proto: stays nil when `proto.Components` // is nil, otherwise wraps it, so the activity layer can distinguish // "not provided" from "explicitly set". @@ -137,6 +137,8 @@ func (sk *SKU) FromProto(proto *corev1.Sku, siteID uuid.UUID) { } sk.ID = proto.Id sk.SiteID = siteID + sk.Description = proto.GetDescription() + sk.SchemaVersion = proto.SchemaVersion sk.DeviceType = proto.DeviceType if proto.Components != nil { sk.Components = &SkuComponents{SkuComponents: proto.Components} @@ -158,8 +160,10 @@ func (sk *SKU) FromProto(proto *corev1.Sku, siteID uuid.UUID) { // SkuCreateInput input parameters for Create method type SkuCreateInput struct { - SkuID string // NICo is the source of truth: id must always be provided on creation. + SkuID string // Core is authoritative, so the ID must be provided when creating its REST projection. SiteID uuid.UUID + Description string + SchemaVersion uint32 Components *SkuComponents DeviceType *string AssociatedMachineIds []string @@ -168,6 +172,8 @@ type SkuCreateInput struct { // SkuUpdateInput input parameters for Update method type SkuUpdateInput struct { SkuID string + Description *string + SchemaVersion *uint32 Components *SkuComponents DeviceType *string AssociatedMachineIds []string @@ -236,6 +242,8 @@ func (ssd SkuSQLDAO) Create(ctx context.Context, tx *db.Tx, input SkuCreateInput sk := &SKU{ ID: input.SkuID, SiteID: input.SiteID, + Description: input.Description, + SchemaVersion: input.SchemaVersion, DeviceType: input.DeviceType, Components: input.Components, AssociatedMachineIds: input.AssociatedMachineIds, @@ -354,6 +362,16 @@ func (ssd SkuSQLDAO) Update(ctx context.Context, tx *db.Tx, input SkuUpdateInput sk := &SKU{ID: input.SkuID} updatedFields := []string{} + if input.Description != nil { + sk.Description = *input.Description + updatedFields = append(updatedFields, "description") + } + + if input.SchemaVersion != nil { + sk.SchemaVersion = *input.SchemaVersion + updatedFields = append(updatedFields, "schema_version") + } + if input.Components != nil { sk.Components = input.Components updatedFields = append(updatedFields, "components") diff --git a/rest-api/db/pkg/db/model/sku_test.go b/rest-api/db/pkg/db/model/sku_test.go index 74f4660e0c..6669f6a1f2 100644 --- a/rest-api/db/pkg/db/model/sku_test.go +++ b/rest-api/db/pkg/db/model/sku_test.go @@ -54,6 +54,8 @@ func TestSKU_ToProto(t *testing.T) { sk := &SKU{ ID: "sku-1", SiteID: siteID, + Description: "GPU compute SKU", + SchemaVersion: 5, DeviceType: &deviceType, Components: &SkuComponents{SkuComponents: &corev1.SkuComponents{}}, AssociatedMachineIds: []string{"m-1", "m-2"}, @@ -61,6 +63,8 @@ func TestSKU_ToProto(t *testing.T) { proto := sk.ToProto() require.NotNil(t, proto) assert.Equal(t, "sku-1", proto.Id) + assert.Equal(t, "GPU compute SKU", proto.GetDescription()) + assert.Equal(t, uint32(5), proto.SchemaVersion) assert.Equal(t, &deviceType, proto.DeviceType) require.NotNil(t, proto.Components) require.Len(t, proto.AssociatedMachineIds, 2) @@ -97,9 +101,11 @@ func TestSKU_FromProto(t *testing.T) { t.Run("populates fields from proto", func(t *testing.T) { sk := &SKU{} sk.FromProto(&corev1.Sku{ - Id: "sku-1", - DeviceType: &deviceType, - Components: &corev1.SkuComponents{}, + Id: "sku-1", + Description: cutil.GetPtr("GPU compute SKU"), + SchemaVersion: 4, + DeviceType: &deviceType, + Components: &corev1.SkuComponents{}, AssociatedMachineIds: []*corev1.MachineId{ {Id: "m-1"}, {Id: ""}, // skipped @@ -108,11 +114,19 @@ func TestSKU_FromProto(t *testing.T) { }, siteID) assert.Equal(t, "sku-1", sk.ID) assert.Equal(t, siteID, sk.SiteID) + assert.Equal(t, "GPU compute SKU", sk.Description) + assert.Equal(t, uint32(4), sk.SchemaVersion) assert.Equal(t, &deviceType, sk.DeviceType) assert.Equal(t, []string{"m-1", "m-2"}, sk.AssociatedMachineIds) require.NotNil(t, sk.Components) }) + t.Run("nil description clears existing description", func(t *testing.T) { + sk := &SKU{Description: "stale description"} + sk.FromProto(&corev1.Sku{Id: "sku-1"}, siteID) + assert.Empty(t, sk.Description) + }) + t.Run("nil Components yields nil wrapper", func(t *testing.T) { sk := &SKU{Components: &SkuComponents{SkuComponents: &corev1.SkuComponents{}}} sk.FromProto(&corev1.Sku{Id: "sku-1"}, siteID) @@ -153,7 +167,12 @@ func testSkuCreateSkus(ctx context.Context, t *testing.T, dbSession *db.Session, ids := []string{"sku-1", "sku-2", "sku-3"} for _, id := range ids { protoSku := &corev1.SkuComponents{} - sk, err := ssd.Create(ctx, nil, SkuCreateInput{SkuID: id, Components: &SkuComponents{SkuComponents: protoSku}, SiteID: siteId}) + sk, err := ssd.Create(ctx, nil, SkuCreateInput{ + SkuID: id, + SiteID: siteId, + Description: id + " description", + Components: &SkuComponents{SkuComponents: protoSku}, + }) require.NoError(t, err) require.NotNil(t, sk) created = append(created, *sk) @@ -185,13 +204,13 @@ func TestSkuSQLDAO_Create(t *testing.T) { }{ { desc: "create one", - inputs: []SkuCreateInput{{SkuID: "sku-1", Components: &SkuComponents{SkuComponents: &corev1.SkuComponents{}}, SiteID: site.ID}}, + inputs: []SkuCreateInput{{SkuID: "sku-1", SiteID: site.ID, Description: "first SKU", SchemaVersion: 5, Components: &SkuComponents{SkuComponents: &corev1.SkuComponents{}}}}, expectError: false, verifyChildSpanner: true, }, { desc: "create multiple", - inputs: []SkuCreateInput{{SkuID: "sku-2", Components: &SkuComponents{SkuComponents: &corev1.SkuComponents{}}, SiteID: site.ID}, {SkuID: "sku-3", Components: &SkuComponents{SkuComponents: &corev1.SkuComponents{}}, SiteID: site.ID}}, + inputs: []SkuCreateInput{{SkuID: "sku-2", SiteID: site.ID, Description: "second SKU", Components: &SkuComponents{SkuComponents: &corev1.SkuComponents{}}}, {SkuID: "sku-3", SiteID: site.ID, Description: "third SKU", Components: &SkuComponents{SkuComponents: &corev1.SkuComponents{}}}}, expectError: false, }, } @@ -203,6 +222,8 @@ func TestSkuSQLDAO_Create(t *testing.T) { if !tc.expectError { assert.NotNil(t, got) assert.Equal(t, input.SkuID, got.ID) + assert.Equal(t, input.SchemaVersion, got.SchemaVersion) + assert.Equal(t, input.Description, got.Description) if input.Components != nil { assert.NotNil(t, got.Components) } @@ -333,19 +354,51 @@ func TestSkuSQLDAO_Update(t *testing.T) { _, _, ctx = testCommonTraceProviderSetup(t, ctx) tests := []struct { - desc string - input SkuUpdateInput - check bool + desc string + input SkuUpdateInput + expectedDescription string + expectedVersion uint32 }{ - {desc: "update sku data", input: SkuUpdateInput{SkuID: created[0].ID, Components: &SkuComponents{SkuComponents: &corev1.SkuComponents{}}}, check: true}, + { + desc: "update sku data", + input: SkuUpdateInput{ + SkuID: created[0].ID, + Description: cutil.GetPtr("updated description"), + SchemaVersion: cutil.GetPtr(uint32(5)), + Components: &SkuComponents{SkuComponents: &corev1.SkuComponents{}}, + }, + expectedDescription: "updated description", + expectedVersion: 5, + }, + { + desc: "nil description preserves stored description", + input: SkuUpdateInput{ + SkuID: created[1].ID, + Description: nil, + Components: &SkuComponents{SkuComponents: &corev1.SkuComponents{}}, + }, + expectedDescription: created[1].Description, + expectedVersion: created[1].SchemaVersion, + }, + { + desc: "empty description clears stored description", + input: SkuUpdateInput{ + SkuID: created[2].ID, + Description: cutil.GetPtr(""), + }, + expectedDescription: "", + expectedVersion: created[2].SchemaVersion, + }, } for _, tc := range tests { t.Run(tc.desc, func(t *testing.T) { got, err := ssd.Update(ctx, nil, tc.input) require.NoError(t, err) - if tc.check { - assert.NotNil(t, got) - assert.Equal(t, tc.input.SkuID, got.ID) + require.NotNil(t, got) + assert.Equal(t, tc.input.SkuID, got.ID) + assert.Equal(t, tc.expectedDescription, got.Description) + assert.Equal(t, tc.expectedVersion, got.SchemaVersion) + if tc.input.Components != nil { assert.NotNil(t, got.Components) } // tracer diff --git a/rest-api/db/pkg/migrations/20260729020000_sku_description.go b/rest-api/db/pkg/migrations/20260729020000_sku_description.go new file mode 100644 index 0000000000..7b4dd21022 --- /dev/null +++ b/rest-api/db/pkg/migrations/20260729020000_sku_description.go @@ -0,0 +1,51 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package migrations + +import ( + "context" + "database/sql" + "fmt" + + "github.com/NVIDIA/infra-controller/rest-api/db/pkg/db/model" + "github.com/uptrace/bun" +) + +func init() { + Migrations.MustRegister(func(ctx context.Context, db *bun.DB) error { + tx, terr := db.BeginTx(ctx, &sql.TxOptions{}) + if terr != nil { + handlePanic(terr, "failed to begin transaction") + } + + _, err := tx.NewAddColumn(). + Model((*model.SKU)(nil)). + IfNotExists(). + ColumnExpr("description TEXT NOT NULL DEFAULT ''"). + Exec(ctx) + handleError(tx, err) + + _, err = tx.NewAddColumn(). + Model((*model.SKU)(nil)). + IfNotExists(). + ColumnExpr("schema_version INTEGER NOT NULL DEFAULT 0"). + Exec(ctx) + handleError(tx, err) + + terr = tx.Commit() + if terr != nil { + handlePanic(terr, "failed to commit transaction") + } + + fmt.Print(" [up migration] Added 'description' and 'schema_version' columns to 'sku' table successfully. ") + return nil + }, func(ctx context.Context, db *bun.DB) error { + _, err := db.ExecContext(ctx, `ALTER TABLE sku DROP COLUMN IF EXISTS description, DROP COLUMN IF EXISTS schema_version`) + if err != nil { + return err + } + fmt.Print(" [down migration] Dropped 'description' and 'schema_version' columns from 'sku' table successfully. ") + return nil + }) +} diff --git a/rest-api/docs/index.html b/rest-api/docs/index.html index 93c3c9b032..d4ef437b14 100644 --- a/rest-api/docs/index.html +++ b/rest-api/docs/index.html @@ -464,7 +464,7 @@ -
API docs by Redocly
API docs by Redocly
Typical API Call Flow for Tenant " class="sc-iJSMbW sc-cBEgGa fiNpIH bAoMjv">

Unique identifier for the SKU

siteId
string <uuid>

ID of the Site this SKU belongs to

+
description
string

Human-readable SKU description

deviceType
string or null

Optional device type identifier (e.g. "gpu", "cpu", "storage")

associatedMachineIds
Array of strings
Typical API Call Flow for Tenant " class="sc-iJSMbW sc-cBEgGa fiNpIH bAoMjv">

Unique identifier for the SKU

siteId
string <uuid>

ID of the Site this SKU belongs to

+
description
string

Human-readable SKU description

deviceType
string or null

Optional device type identifier (e.g. "gpu", "cpu", "storage")

associatedMachineIds
Array of strings
Typical API Call Flow for Tenant " class="sc-iJSMbW sc-cBEgGa fiNpIH bAoMjv">

Unique identifier for the SKU

siteId
string <uuid>

ID of the Site this SKU belongs to

+
description
string

Human-readable SKU description

deviceType
string or null

Optional device type identifier (e.g. "gpu", "cpu", "storage")

associatedMachineIds
Array of strings
Typical API Call Flow for Tenant " class="sc-iJSMbW sc-cBEgGa fiNpIH bAoMjv">

Unique identifier for the SKU

siteId
string <uuid>

ID of the Site this SKU belongs to

+
description
string

Human-readable SKU description

deviceType
string or null

Optional device type identifier (e.g. "gpu", "cpu", "storage")

associatedMachineIds
Array of strings
Typical API Call Flow for Tenant " class="sc-iJSMbW sc-cBEgGa fiNpIH bAoMjv">

Unique identifier for the SKU

siteId
string <uuid>

ID of the Site this SKU belongs to

+
description
string

Human-readable SKU description

deviceType
string or null

Optional device type identifier (e.g. "gpu", "cpu", "storage")

associatedMachineIds
Array of strings
Typical API Call Flow for Tenant " class="sc-iJSMbW sc-cBEgGa fiNpIH bAoMjv">

Unique identifier for the SKU

siteId
string <uuid>

ID of the Site this SKU belongs to

+
description
string

Human-readable SKU description

deviceType
string or null

Optional device type identifier (e.g. "gpu", "cpu", "storage")

associatedMachineIds
Array of strings
Typical API Call Flow for Tenant

Response samples

Content type
application/json
{
  • "source": "nico",
  • "message": "Error validating request data",
  • "data": {
    }
}

SKU

SKU (Stock Keeping Unit) defines one or more hardware configurations or Machine Bill of Materials (BOM).

-

SKUs are automatically derived from machine hardware characteristics and used to group similar machines. SKUs are read-only and managed by the system.

+

SKUs group similar machines by their hardware characteristics. Most SKUs are managed by the system, but Provider Admins can create, update, and delete SKUs for a Site.

Retrieve all SKUs

Typical API Call Flow for Tenant " class="sc-iJSMbW sc-cBEgGa fiNpIH bAoMjv">

Unique identifier for the SKU

siteId
string <uuid>

ID of the Site this SKU belongs to

+
description
string

Human-readable SKU description

deviceType
string or null

Optional device type identifier (e.g. "gpu", "cpu", "storage")

associatedMachineIds
Array of strings
Typical API Call Flow for Tenant " class="sc-iJSMbW sc-cBEgGa sc-ciCrSJ fiNpIH dNfUH dDDioG">

Error response when user is not authorized to call an endpoint or retrieve/modify objects

Response samples

Content type
application/json
[
  • {
    }
]

Retrieve SKU

https://nico-rest-api.nico.svc.cluster.local/v2/org/{org}/nico/sku

Response samples

Content type
application/json
[
  • {
    }
]

Create SKU

Create a SKU.

+

The Org must have an Infrastructure Provider that owns the Site. The user must have an authorization role with the PROVIDER_ADMIN suffix.

+
Authorizations:
JWTBearerToken
path Parameters
org
required
string

Name of the Org

+
Request Body schema: application/json
required
siteId
required
string <uuid>

ID of the Site where the SKU will be created

+
id
required
string non-empty

Unique SKU identifier

+
description
string

Human-readable SKU description

+
deviceType
string or null

Optional device type identifier

+
required
object (SkuMutationComponents)

Hardware components accepted and returned by SKU mutation endpoints

+
Array of objects (SkuCpu)

CPU components

+
Array
vendor
string

Vendor of the CPU

+
model
string

Model of the CPU

+
threadCount
integer <uint32>

Number of threads for the CPU

+
count
integer <uint32>

Number of CPUs present

+
Array of objects (SkuGpu)

GPU components

+
Array
vendor
string

Vendor of the GPU

+
model
string

Model of the GPU

+
totalMemory
string

Total memory of the GPU (e.g. "80GB HBM3")

+
count
integer <uint32>

Number of GPUs present

+
Array of objects (SkuMemory)

Memory components

+
Array
capacityMb
integer <uint32>

Capacity in megabytes

+
memoryType
string

Type of memory (e.g. "DDR4", "DDR5")

+
count
integer <uint32>

Number of memory modules present

+
Array of objects (SkuStorageMutation)

Storage components

+
Array
model
string

Informational storage model. Starting with the 2.1 release, NICo does not use this field for storage matching or validation.

+
count
integer <uint32>

Number of storage devices required

+
minSizeMiB
integer or null <uint32>

Inclusive minimum size in MiB for each storage device. Null or omission means no lower bound. Used for SKU schema version 5 and later.

+
maxSizeMiB
integer or null <uint32>

Inclusive maximum size in MiB for each storage device. Null or omission means no upper bound. Used for SKU schema version 5 and later.

+
pciPatterns
Array of strings

Regular expressions matched against storage sysfs PCI paths. An empty or omitted list disables PCI location matching. Used for SKU schema version 5 and later.

+
object (SkuChassis)

Chassis component

+
vendor
string

Vendor of the chassis

+
model
string

Model of the chassis

+
architecture
string

Architecture of the chassis

+
Array of objects (SkuEthernetDevice)

Ethernet device components

+
Array
vendor
string

Vendor of the ethernet device

+
model
string

Model of the ethernet device

+
count
integer <uint32>

Number of ethernet devices present

+
isConnected
boolean

Whether the ethernet device is connected

+
Array of objects (SkuInfinibandDevice)

Infiniband device components

+
Array
vendor
string

Vendor of the infiniband device

+
model
string

Model of the infiniband device

+
count
integer <uint32>

Number of infiniband devices present

+
inactiveDevices
Array of integers <uint32> [ items <uint32 > ]

Zero-based indexes of inactive devices

+
object (SkuTpm)

TPM component

+
vendor
string

Vendor of the TPM

+
version
string

Version of the TPM

+

Responses

Response Schema: application/json
id
string

Unique SKU identifier

+
siteId
string <uuid>

ID of the Site that owns the SKU

+
description
string

Human-readable SKU description

+
schemaVersion
integer <uint32>

Core SKU schema version

+
deviceType
string or null

Optional device type identifier

+
associatedMachineIds
Array of strings

Machine IDs currently associated with the SKU

+
object (SkuComponents)

Hardware components that make up a SKU

+
Array of objects (SkuCpu)

CPU components

+
Array
vendor
string

Vendor of the CPU

+
model
string

Model of the CPU

+
threadCount
integer <uint32>

Number of threads for the CPU

+
count
integer <uint32>

Number of CPUs present

+
Array of objects (SkuGpu)

GPU components

+
Array
vendor
string

Vendor of the GPU

+
model
string

Model of the GPU

+
totalMemory
string

Total memory of the GPU (e.g. "80GB HBM3")

+
count
integer <uint32>

Number of GPUs present

+
Array of objects (SkuMemory)

Memory components

+
Array
capacityMb
integer <uint32>

Capacity in megabytes

+
memoryType
string

Type of memory (e.g. "DDR4", "DDR5")

+
count
integer <uint32>

Number of memory modules present

+
Array of objects (SkuStorage)

Storage components

+
Array
vendor
string
Deprecated

Deprecated legacy field retained for response compatibility. Core returns an empty string and does not use this field for storage matching.

+
model
string

Informational storage model. Starting with the 2.1 release, NICo does not use this field for storage matching or validation.

+
capacityMb
integer <uint32>
Deprecated

Deprecated legacy field retained for response compatibility. Core returns zero; use minSizeMiB and maxSizeMiB for storage size constraints.

+
count
integer <uint32>

Number of storage devices present

+
minSizeMiB
integer or null <uint32>

Inclusive minimum size in MiB for each storage device. Null or omission means no lower bound. Used for SKU schema version 5 and later.

+
maxSizeMiB
integer or null <uint32>

Inclusive maximum size in MiB for each storage device. Null or omission means no upper bound. Used for SKU schema version 5 and later.

+
pciPatterns
Array of strings

Regular expressions matched against storage sysfs PCI paths. An empty or omitted list disables PCI location matching. Used for SKU schema version 5 and later.

+
object (SkuChassis)

Chassis component

+
vendor
string

Vendor of the chassis

+
model
string

Model of the chassis

+
architecture
string

Architecture of the chassis

+
Array of objects (SkuEthernetDevice)

Ethernet device components

+
Array
vendor
string

Vendor of the ethernet device

+
model
string

Model of the ethernet device

+
count
integer <uint32>

Number of ethernet devices present

+
isConnected
boolean

Whether the ethernet device is connected

+
Array of objects (SkuInfinibandDevice)

Infiniband device components

+
Array
vendor
string

Vendor of the infiniband device

+
model
string

Model of the infiniband device

+
count
integer <uint32>

Number of infiniband devices present

+
inactiveDevices
Array of integers <uint32> [ items <uint32 > ]

Zero-based indexes of inactive devices

+
Array of objects (SkuTpm)

TPM components

+
Array
vendor
string

Vendor of the TPM

+
version
string

Version of the TPM

+
created
string or null <date-time>

Core creation timestamp when available

+

Request samples

Content type
application/json
{
  • "siteId": "60189e9c-7d12-438c-b9ca-6998d9c364b1",
  • "id": "dgx-h100",
  • "description": "DGX H100 compute node",
  • "deviceType": "gpu-server",
  • "components": {
    }
}

Response samples

Content type
application/json
{
  • "id": "dgx-gb200",
  • "siteId": "60189e9c-7d12-438c-b9ca-6998d9c364b1",
  • "description": "DGX GB200 GPU node",
  • "schemaVersion": 5,
  • "deviceType": "gpu-server",
  • "associatedMachineIds": [ ],
  • "components": {
    },
  • "created": "2026-07-20T14:15:22Z"
}

Retrieve SKU

Typical API Call Flow for Tenant " class="sc-iJSMbW sc-cBEgGa fiNpIH bAoMjv">

Unique identifier for the SKU

siteId
string <uuid>

ID of the Site this SKU belongs to

+
description
string

Human-readable SKU description

deviceType
string or null

Optional device type identifier (e.g. "gpu", "cpu", "storage")

associatedMachineIds
Array of strings
Typical API Call Flow for Tenant " class="sc-iJSMbW sc-cBEgGa fiNpIH bAoMjv">

Number of memory modules present

Array of objects (SkuStorage)

Storage components

-
Array
vendor
string

Vendor of the storage device

-
model
string

Model of the storage device

-
capacityMb
integer <uint32>

Capacity in megabytes

-
count
integer <uint32>
Array
vendor
string
Deprecated

Deprecated legacy field retained for response compatibility. Core returns an empty string and does not use this field for storage matching.

+
model
string

Informational storage model. Starting with the 2.1 release, NICo does not use this field for storage matching or validation.

+
capacityMb
integer <uint32>
Deprecated

Deprecated legacy field retained for response compatibility. Core returns zero; use minSizeMiB and maxSizeMiB for storage size constraints.

+
count
integer <uint32>

Number of storage devices present

+
minSizeMiB
integer or null <uint32>

Inclusive minimum size in MiB for each storage device. Null or omission means no lower bound. Used for SKU schema version 5 and later.

+
maxSizeMiB
integer or null <uint32>

Inclusive maximum size in MiB for each storage device. Null or omission means no upper bound. Used for SKU schema version 5 and later.

+
pciPatterns
Array of strings

Regular expressions matched against storage sysfs PCI paths. An empty or omitted list disables PCI location matching. Used for SKU schema version 5 and later.

object (SkuChassis)

Chassis component

vendor
string

Vendor of the chassis

-
model
string
model
string

Model of the chassis

+
architecture
string

Architecture of the chassis

Array of objects (SkuEthernetDevice)

Ethernet device components

Array
vendor
string

Vendor of the ethernet device

model
string

Model of the ethernet device

-
count
integer <uint32>
count
integer <uint32>

Number of ethernet devices present

+
isConnected
boolean

Whether the ethernet device is connected

Array of objects (SkuInfinibandDevice)

Infiniband device components

Array
vendor
string

Vendor of the infiniband device

model
string

Model of the infiniband device

-
count
integer <uint32>
count
integer <uint32>

Number of infiniband devices present

+
inactiveDevices
Array of integers <uint32> [ items <uint32 > ]

Zero-based indexes of inactive devices

Array of objects (SkuTpm)

TPM components

Array
vendor
string
Typical API Call Flow for Tenant " class="sc-iJSMbW sc-cBEgGa sc-ciCrSJ fiNpIH dNfUH dDDioG">

Error response when requested object is not found

Response samples

Content type
application/json
{
  • "id": "lenovo.sr650v2.cpu.1",
  • "siteId": "60189e9c-7d12-438c-b9ca-6998d9c364b1",
  • "deviceType": "gpu",
  • "associatedMachineIds": [
    ],
  • "components": {
    },
  • "created": "2019-08-24T14:15:22Z",
  • "updated": "2019-08-24T14:15:22Z"
}

InfiniBand Partition

https://nico-rest-api.nico.svc.cluster.local/v2/org/{org}/nico/sku/{skuId}

Response samples

Content type
application/json
{
  • "id": "lenovo.sr650v2.cpu.1",
  • "siteId": "60189e9c-7d12-438c-b9ca-6998d9c364b1",
  • "deviceType": "gpu",
  • "associatedMachineIds": [
    ],
  • "components": {
    },
  • "created": "2019-08-24T14:15:22Z",
  • "updated": "2019-08-24T14:15:22Z"
}

Delete SKU

Delete an unused SKU. The Site is determined from the saved SKU record.

+

The Org must have an Infrastructure Provider that owns the SKU's Site. The user must have an authorization role with the PROVIDER_ADMIN suffix.

+
Authorizations:
JWTBearerToken
path Parameters
org
required
string

Name of the Org

+
skuId
required
string

ID of the SKU

+

Responses

Response samples

Content type
application/json
{
  • "source": "nico",
  • "message": "Error validating request data",
  • "data": {
    }
}

Update SKU

Update selected mutable fields on a SKU. The Site is determined from the saved SKU record.

+

Fields omitted from the request retain their current values. If a user updates the components field, +the schema version will be updated to the current version (5).

+

The Org must have an Infrastructure Provider that owns the SKU's Site. The user must have an authorization role with the PROVIDER_ADMIN suffix.

+
Authorizations:
JWTBearerToken
path Parameters
org
required
string

Name of the Org

+
skuId
required
string

ID of the SKU

+
Request Body schema: application/json
required
description
string

Replacement SKU description

+
deviceType
string

Replacement device type identifier

+
object (SkuMutationComponents)

Hardware components accepted and returned by SKU mutation endpoints

+
Array of objects (SkuCpu)

CPU components

+
Array
vendor
string

Vendor of the CPU

+
model
string

Model of the CPU

+
threadCount
integer <uint32>

Number of threads for the CPU

+
count
integer <uint32>

Number of CPUs present

+
Array of objects (SkuGpu)

GPU components

+
Array
vendor
string

Vendor of the GPU

+
model
string

Model of the GPU

+
totalMemory
string

Total memory of the GPU (e.g. "80GB HBM3")

+
count
integer <uint32>

Number of GPUs present

+
Array of objects (SkuMemory)

Memory components

+
Array
capacityMb
integer <uint32>

Capacity in megabytes

+
memoryType
string

Type of memory (e.g. "DDR4", "DDR5")

+
count
integer <uint32>

Number of memory modules present

+
Array of objects (SkuStorageMutation)

Storage components

+
Array
model
string

Informational storage model. Starting with the 2.1 release, NICo does not use this field for storage matching or validation.

+
count
integer <uint32>

Number of storage devices required

+
minSizeMiB
integer or null <uint32>

Inclusive minimum size in MiB for each storage device. Null or omission means no lower bound. Used for SKU schema version 5 and later.

+
maxSizeMiB
integer or null <uint32>

Inclusive maximum size in MiB for each storage device. Null or omission means no upper bound. Used for SKU schema version 5 and later.

+
pciPatterns
Array of strings

Regular expressions matched against storage sysfs PCI paths. An empty or omitted list disables PCI location matching. Used for SKU schema version 5 and later.

+
object (SkuChassis)

Chassis component

+
vendor
string

Vendor of the chassis

+
model
string

Model of the chassis

+
architecture
string

Architecture of the chassis

+
Array of objects (SkuEthernetDevice)

Ethernet device components

+
Array
vendor
string

Vendor of the ethernet device

+
model
string

Model of the ethernet device

+
count
integer <uint32>

Number of ethernet devices present

+
isConnected
boolean

Whether the ethernet device is connected

+
Array of objects (SkuInfinibandDevice)

Infiniband device components

+
Array
vendor
string

Vendor of the infiniband device

+
model
string

Model of the infiniband device

+
count
integer <uint32>

Number of infiniband devices present

+
inactiveDevices
Array of integers <uint32> [ items <uint32 > ]

Zero-based indexes of inactive devices

+
object (SkuTpm)

TPM component

+
vendor
string

Vendor of the TPM

+
version
string

Version of the TPM

+

Responses

Response Schema: application/json
id
string

Unique SKU identifier

+
siteId
string <uuid>

ID of the Site that owns the SKU

+
description
string

Human-readable SKU description

+
schemaVersion
integer <uint32>

Core SKU schema version

+
deviceType
string or null

Optional device type identifier

+
associatedMachineIds
Array of strings

Machine IDs currently associated with the SKU

+
object (SkuComponents)

Hardware components that make up a SKU

+
Array of objects (SkuCpu)

CPU components

+
Array
vendor
string

Vendor of the CPU

+
model
string

Model of the CPU

+
threadCount
integer <uint32>

Number of threads for the CPU

+
count
integer <uint32>

Number of CPUs present

+
Array of objects (SkuGpu)

GPU components

+
Array
vendor
string

Vendor of the GPU

+
model
string

Model of the GPU

+
totalMemory
string

Total memory of the GPU (e.g. "80GB HBM3")

+
count
integer <uint32>

Number of GPUs present

+
Array of objects (SkuMemory)

Memory components

+
Array
capacityMb
integer <uint32>

Capacity in megabytes

+
memoryType
string

Type of memory (e.g. "DDR4", "DDR5")

+
count
integer <uint32>

Number of memory modules present

+
Array of objects (SkuStorage)

Storage components

+
Array
vendor
string
Deprecated

Deprecated legacy field retained for response compatibility. Core returns an empty string and does not use this field for storage matching.

+
model
string

Informational storage model. Starting with the 2.1 release, NICo does not use this field for storage matching or validation.

+
capacityMb
integer <uint32>
Deprecated

Deprecated legacy field retained for response compatibility. Core returns zero; use minSizeMiB and maxSizeMiB for storage size constraints.

+
count
integer <uint32>

Number of storage devices present

+
minSizeMiB
integer or null <uint32>

Inclusive minimum size in MiB for each storage device. Null or omission means no lower bound. Used for SKU schema version 5 and later.

+
maxSizeMiB
integer or null <uint32>

Inclusive maximum size in MiB for each storage device. Null or omission means no upper bound. Used for SKU schema version 5 and later.

+
pciPatterns
Array of strings

Regular expressions matched against storage sysfs PCI paths. An empty or omitted list disables PCI location matching. Used for SKU schema version 5 and later.

+
object (SkuChassis)

Chassis component

+
vendor
string

Vendor of the chassis

+
model
string

Model of the chassis

+
architecture
string

Architecture of the chassis

+
Array of objects (SkuEthernetDevice)

Ethernet device components

+
Array
vendor
string

Vendor of the ethernet device

+
model
string

Model of the ethernet device

+
count
integer <uint32>

Number of ethernet devices present

+
isConnected
boolean

Whether the ethernet device is connected

+
Array of objects (SkuInfinibandDevice)

Infiniband device components

+
Array
vendor
string

Vendor of the infiniband device

+
model
string

Model of the infiniband device

+
count
integer <uint32>

Number of infiniband devices present

+
inactiveDevices
Array of integers <uint32> [ items <uint32 > ]

Zero-based indexes of inactive devices

+
Array of objects (SkuTpm)

TPM components

+
Array
vendor
string

Vendor of the TPM

+
version
string

Version of the TPM

+
created
string or null <date-time>

Core creation timestamp when available

+

Request samples

Content type
application/json
{
  • "description": "DGX H100 compute node with InfiniBand",
  • "components": {
    }
}

Response samples

Content type
application/json
{
  • "id": "dgx-gb200",
  • "siteId": "60189e9c-7d12-438c-b9ca-6998d9c364b1",
  • "description": "DGX GB200 GPU node",
  • "schemaVersion": 5,
  • "deviceType": "gpu-server",
  • "associatedMachineIds": [ ],
  • "components": {
    },
  • "created": "2026-07-20T14:15:22Z"
}

InfiniBand Partition

InfiniBand (IB) is a high-performance, low-latency networking standard designed for interconnecting servers and storage in HPC (High-Performance Computing) and AI systems, utilizing RDMA (Remote Direct Memory Access) to reduce CPU overhead. InfiniBand Partitions are used to group Machines into logical partitions for network isolation and load distribution.

@@ -16212,7 +16646,11 @@

Typical API Call Flow for Tenant