fix(talos): make flannel ownership singular

argocd/applicationsets/platform.yaml

@@ -76,13 +76,6 @@ spec:
                   argocd.argoproj.io/sync-wave: "-1"
                 automation: auto
                 enabled: "true"
-              - name: flannel-cni
-                path: argocd/applications/flannel-cni
-                namespace: kube-system
-                annotations:
-                  argocd.argoproj.io/sync-wave: "-1"
-                automation: auto
-                enabled: "true"
               - name: volume-snapshot-controller
                 path: argocd/applications/volume-snapshot-controller
                 namespace: kube-system

devices/galactic/docs/troubleshooting-networking.md

@@ -1,30 +1,64 @@
-# Troubleshooting: pod networking (Talos Flannel VXLAN with MTU guard)
+# Troubleshooting: Talos-owned Flannel VXLAN
 
-Talos owns the Flannel DaemonSet, service account, and RBAC resources. The `flannel-cni` Argo application deliberately
-guards only `kube-flannel-cfg` so the CNI delegate keeps MTU 1400 while the backend remains Talos' default VXLAN on
-UDP 4789.
+Talos owns the complete Flannel manifest through `cluster.network.cni.name: custom`. The manifest URL is pinned to an
+immutable Git commit by `devices/galactic/manifests/custom-flannel-cni.patch.yaml`; the bundle owns the ClusterRole,
+ClusterRoleBinding, ServiceAccount, ConfigMap, and DaemonSet together. Argo CD does not manage any Flannel resource.
 
-This is a temporary compatibility exception for stable Talos 1.13, which does not expose a Flannel MTU setting. Fresh
-ARC runners at pod MTU 1450 repeatedly timed out fetching GitHub pipeline connection data; changing the same pod to
-MTU 1400 made it register and claim work immediately. Do not reduce the physical underlay MTU to force Flannel's
-derived value: adding an otherwise-incomplete interface entry can displace platform-provided addressing and remove a
-control-plane node from the network.
+The custom bundle is required because stable Talos 1.13 does not expose a native Flannel MTU setting. Fresh ARC runners
+at pod MTU 1450 repeatedly timed out fetching GitHub pipeline connection data; changing the same pod to MTU 1400 made
+it register and claim work immediately. Do not reduce the physical underlay MTU to force Flannel's derived value:
+adding an otherwise-incomplete interface entry can displace platform-provided addressing and remove a control-plane
+node from the network.
 
-The long-term fix is either a stable Talos release with native Flannel MTU configuration or a complete custom CNI
-manifest with one owner. Until then, preserve this narrowly-scoped guard and treat its reconciliation as part of every
-Kubernetes upgrade.
+Before changing the CNI source or upgrading Kubernetes, confirm the pinned manifest is reachable and valid:
+
+```bash
+curl -fsS "$(yq -r '.cluster.network.cni.urls[0]' devices/galactic/manifests/custom-flannel-cni.patch.yaml)" \
+  | kubeconform -strict -summary -
+```
+
+## Ownership cutover
+
+Do not start the cutover unless all control-plane nodes are reachable, etcd has a leader and quorum, and the Kubernetes
+API is ready. Apply the checked-in patch to one control-plane node at a time without rebooting:
+
+```bash
+talosctl -e <healthy-control-plane-ip> -n <node-ip> patch machineconfig --mode=no-reboot \
+  --patch @devices/galactic/manifests/custom-flannel-cni.patch.yaml
+```
+
+After every node is patched, verify its safe CNI subtree and the Talos manifest inventory:
+
+```bash
+talosctl -e <healthy-control-plane-ip> -n <node-ip> get machineconfig v1alpha1 -o yaml \
+  | yq -r '.spec' | yq -e '.cluster.network.cni.name == "custom" and (.cluster.network.cni.urls | length == 1)'
+talosctl -e <healthy-control-plane-ip> -n <node-ip> get manifests
+talosctl -e <healthy-control-plane-ip> -n <node-ip> get manifeststatuses
+```
+
+Keep the Argo application in place until Talos has downloaded and applied the custom manifest and the network checks
+below pass. After Git removes the application, confirm `Application/flannel-cni` is absent, then remove its stale
+preservation annotation from the now Talos-owned ConfigMap:
+
+```bash
+kubectl -n argocd get application flannel-cni
+kubectl -n kube-system annotate configmap kube-flannel-cfg argocd.argoproj.io/sync-options-
+```
+
+Before the Argo application is removed, rollback is a machine-config patch back to `name: flannel` with an empty URL
+list; the Argo MTU guard remains the safety net. After Argo removal, restore the guard in Git before making that rollback
+so MTU 1400 is never left without an owner.
 
 Before every Kubernetes upgrade, inspect the Talos manifest plan:
 
 ```bash
 talosctl -e <control-plane-ip> -n <control-plane-ip> upgrade-k8s --to <target-version> --dry-run
 ```
 
-The dry run must keep the VXLAN backend on UDP 4789. Immediately after the upgrade, sync `flannel-cni`, verify the
-guarded ConfigMap, restart the Flannel DaemonSet, and validate a newly-created pod on every node:
+The dry run must retain the custom CNI URL and keep the VXLAN backend on UDP 4789. Immediately after the upgrade,
+verify the Talos-owned ConfigMap, restart the Flannel DaemonSet, and validate a newly-created pod on every node:
 
 ```bash
-argocd app sync flannel-cni
 kubectl -n kube-system get configmap kube-flannel-cfg \
   -o jsonpath='{.data.cni-conf\.json}' | jq -e '.plugins[0].delegate.mtu == 1400'
 kubectl -n kube-system rollout restart daemonset/kube-flannel
@@ -33,9 +67,9 @@ kubectl -n <workload-namespace> exec <fresh-pod-on-each-node> -- \
   sh -c "ip -o link show eth0 | grep -q 'mtu 1400'"
 ```
 
-The guard does not set Flannel's backend MTU, and `/run/flannel/subnet.env` may still report `FLANNEL_MTU=1450` on a
-normal 1500-byte underlay. Do not continue workload validation unless the guarded CNI value is 1400 and fresh pod
-interfaces on every node report MTU 1400.
+The custom CNI delegate does not change Flannel's backend MTU, and `/run/flannel/subnet.env` may still report
+`FLANNEL_MTU=1450` on a normal 1500-byte underlay. Do not continue workload validation unless the ConfigMap value is
+1400 and fresh pod interfaces on every node report MTU 1400.
 
 If VXLAN forwarding breaks, pods on one node cannot reach pods on another node, even though nodes may still appear
 `Ready`.

devices/galactic/manifests/README.md

@@ -10,7 +10,11 @@ The upstream template is pinned by the Talos `v1.13.6` tag in
 [`flannel.go`](https://github.com/siderolabs/talos/blob/v1.13.6/internal/app/machined/pkg/controllers/k8s/internal/k8stemplates/flannel.go),
 and Talos v1.13.6 pins Flannel `v0.28.5` in its machinery constants.
 
-The file is staged separately so the cluster configuration can later reference an immutable Git commit URL. It does
-not become active merely by being present in this repository. Activation must set `cluster.network.cni.name` to
-`custom`, set `cluster.network.cni.urls` to the immutable raw URL, and remove the partial Argo `flannel-cni` owner in
-the same guarded handoff.
+`custom-flannel-cni.patch.yaml` switches Talos from its generated Flannel bundle to this complete custom bundle. The
+URL is pinned to merge commit `15f59fd037dd45ecd79cf69031215d1e79b6b479`, so node boot and Kubernetes upgrades do
+not depend on mutable `main` content.
+
+Apply that patch to every control-plane node only after etcd, the Kubernetes API, and the raw manifest URL are healthy.
+Keep the existing Flannel resources running throughout the change; Talos replaces its generated manifest source with
+the custom source at the same priority. The Argo `flannel-cni` application must be removed only after all nodes report
+the custom CNI configuration and Talos reports the custom manifest as applied.

devices/galactic/manifests/custom-flannel-cni.patch.yaml

@@ -0,0 +1,6 @@
+cluster:
+  network:
+    cni:
+      name: custom
+      urls:
+        - https://raw.githubusercontent.com/proompteng/lab/15f59fd037dd45ecd79cf69031215d1e79b6b479/devices/galactic/manifests/flannel-v0.28.5-mtu1400.yaml

packages/scripts/src/docs/__tests__/flannel-mtu-runbook.test.ts

@@ -8,10 +8,12 @@ const runbookPath = 'devices/galactic/docs/troubleshooting-networking.md'
 const runbook = readFileSync(join(repoRoot, runbookPath), 'utf8')
 const scriptsWorkflow = readFileSync(join(repoRoot, '.github/workflows/scripts-ci.yml'), 'utf8')
 
-it('validates the MTU value controlled by the Flannel CNI guard', () => {
+it('validates the MTU value controlled by the Talos custom Flannel CNI', () => {
+  expect(runbook).toContain('cluster.network.cni.name: custom')
   expect(runbook).toContain("jq -e '.plugins[0].delegate.mtu == 1400'")
   expect(runbook).toContain("ip -o link show eth0 | grep -q 'mtu 1400'")
-  expect(runbook).toContain('may still report `FLANNEL_MTU=1450`')
+  expect(runbook).toMatch(/may still report\s+`FLANNEL_MTU=1450`/)
+  expect(runbook).not.toContain('argocd app sync flannel-cni')
   expect(runbook).not.toContain('unless every node reports `FLANNEL_MTU=1400`')
 })
 

packages/scripts/src/shared/__tests__/enabled-apps.test.ts

@@ -21,11 +21,6 @@ const productApplicationSet = YAML.parse(readFileSync('argocd/applicationsets/pr
     templatePatch?: string
   }
 }
-const flannelConfigMap = YAML.parse(readFileSync('argocd/applications/flannel-cni/kube-flannel-cfg.yaml', 'utf8')) as {
-  metadata?: { annotations?: Record<string, string> }
-  data?: Record<string, string>
-}
-
 const entry = (name: string) => {
   const found = inventory.entries.find((candidate) => candidate.name === name)
   if (!found) throw new Error(`Missing enabled app inventory entry for ${name}`)
@@ -34,18 +29,15 @@ const entry = (name: string) => {
 
 describe('enabled app inventory', () => {
   it('loads only root-enabled ApplicationSet entries plus direct root-managed Applications', () => {
-    expect(inventory.applicationSetEntryCount).toBe(68)
+    expect(inventory.applicationSetEntryCount).toBe(67)
     expect(inventory.directApplicationCount).toBe(1)
-    expect(inventory.entries).toHaveLength(69)
+    expect(inventory.entries).toHaveLength(68)
     expect(inventory.entries.some((candidate) => candidate.name === 'facteur')).toBe(false)
     expect(inventory.entries.some((candidate) => candidate.name === 'bonjour')).toBe(false)
     expect(inventory.entries.some((candidate) => candidate.name === 'olden')).toBe(false)
     expect(inventory.entries.some((candidate) => candidate.name === 'posthog')).toBe(false)
     expect(inventory.entries.some((candidate) => candidate.name === 'sag')).toBe(false)
-    expect(entry('flannel-cni')).toMatchObject({
-      class: 'vendor-manifest',
-      path: 'argocd/applications/flannel-cni',
-    })
+    expect(inventory.entries.some((candidate) => candidate.name === 'flannel-cni')).toBe(false)
   })
 
   it('records preservation intent when a product app is disabled', () => {
@@ -54,19 +46,6 @@ describe('enabled app inventory', () => {
     expect(productElements.find((candidate) => candidate.name === 'sag')?.cascadeResourcesOnDeletion).not.toBe(true)
   })
 
-  it('guards the required pod MTU without changing the Talos VXLAN backend', () => {
-    const cni = JSON.parse(flannelConfigMap.data?.['cni-conf.json'] ?? '{}') as {
-      plugins?: Array<{ delegate?: { mtu?: number } }>
-    }
-    const network = JSON.parse(flannelConfigMap.data?.['net-conf.json'] ?? '{}') as {
-      Backend?: { Type?: string; Port?: number }
-    }
-
-    expect(flannelConfigMap.metadata?.annotations?.['argocd.argoproj.io/sync-options']).toBe('Delete=false')
-    expect(cni.plugins?.[0]?.delegate?.mtu).toBe(1400)
-    expect(network.Backend).toEqual({ Type: 'vxlan', Port: 4789 })
-  })
-
   it('cascades resources for generated Applications that are disabled destructively', () => {
     const productElements = productApplicationSet.spec?.generators?.[0]?.matrix?.generators?.[1]?.list?.elements ?? []
     expect(productElements.find((candidate) => candidate.name === 'olden')?.cascadeResourcesOnDeletion).toBe(true)

packages/scripts/src/shared/__tests__/talos-flannel.test.ts

@@ -4,6 +4,10 @@ import { describe, expect, it } from 'bun:test'
 import YAML from 'yaml'
 
 const manifestPath = 'devices/galactic/manifests/flannel-v0.28.5-mtu1400.yaml'
+const cniPatchPath = 'devices/galactic/manifests/custom-flannel-cni.patch.yaml'
+const cniPatch = YAML.parse(readFileSync(cniPatchPath, 'utf8')) as {
+  cluster?: { network?: { cni?: { name?: string; urls?: string[] } } }
+}
 const resources = YAML.parseAllDocuments(readFileSync(manifestPath, 'utf8')).map((document) =>
   document.toJSON(),
 ) as Array<{
@@ -28,6 +32,16 @@ const resource = (kind: string, name: string) => {
 }
 
 describe('Talos-owned custom Flannel manifest', () => {
+  it('pins Talos custom CNI ownership to the immutable manifest commit', () => {
+    const cni = cniPatch.cluster?.network?.cni
+
+    expect(cni?.name).toBe('custom')
+    expect(cni?.urls).toEqual([
+      'https://raw.githubusercontent.com/proompteng/lab/15f59fd037dd45ecd79cf69031215d1e79b6b479/devices/galactic/manifests/flannel-v0.28.5-mtu1400.yaml',
+    ])
+    expect(cni?.urls?.[0]).toMatch(/\/proompteng\/lab\/[0-9a-f]{40}\//)
+  })
+
   it('contains the complete Talos Flannel resource set without a Namespace', () => {
     expect(resources.map(({ kind, metadata }) => `${kind}/${metadata?.name}`)).toEqual([
       'ClusterRole/flannel',