fix(ci): repair broken test-e2e, cluster-init, and sdk pipelines - #418
Merged
Conversation
Trailing newline only, no content changes. Touches sdk/** and infra/cluster-init/** so the PR exercises turing.yaml, sdk.yaml, and cluster-init.yaml (gated by paths/paths-ignore filters) for a CI test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- infra/cluster-init/Dockerfile: bitnami/kubectl:1.22.16 is no longer published on Docker Hub (Bitnami removed legacy versioned tags). Fetch the same kubectl version directly from the official Kubernetes release binaries instead of depending on a third-party image. - sdk/Makefile: pip's isolated build environment (used to compile pandas from sdist, since no cp312 wheel exists for pandas<2) resolves setuptools per pandas' own unbounded pyproject.toml build requirement, ignoring whatever setuptools is pinned in the outer install. That fetched setuptools>=80, which dropped pkg_resources, breaking the legacy sdist build with ModuleNotFoundError: No module named 'pkg_resources'. Fix: pre-install setuptools<75 (still ships pkg_resources), Cython<3, and numpy<2 in the outer environment, then install requirements with --no-build-isolation so the sdist build reuses those instead of fetching its own unconstrained toolchain. pandas stays unpinned. Verified locally on Python 3.12.13 (matching the CI matrix): `make setup` completes without error and `make test` passes (383 passed). Also drops the now-redundant no-op newline from the prior dummy commit since these fixes already touch the required paths. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #418 +/- ##
=======================================
Coverage 96.38% 96.38%
=======================================
Files 26 26
Lines 2101 2101
=======================================
Hits 2025 2025
Misses 76 76
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
test-e2e fails during "Setup local k8s cluster" with: curl: (22) The requested URL returned error: 404 Failed to install k3d Root cause: AbsaOSS/k3d-action@v2.3.0 defaults to k3d v5.4.1. That release's k3d-linux-amd64 binary still exists on GitHub, but its checksums.txt release asset does not (k3d only started publishing checksums.txt from v5.5.0 onward). k3d's own install.sh unconditionally fetches checksums.txt and verifies against it, so under `set -e` the missing file aborts the install with curl exit code 22. Fix: explicitly set k3d-version: v5.5.0 via the action's k3d-version input, the earliest release with a complete asset set. Verified locally by running k3d-io/k3d's install.sh with TAG=v5.5.0 - it downloads, checksum-verifies, and installs cleanly (k3d v5.5.0, default k3s v1.26.4-k3s1). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test-e2e hung after the k3d-version pin (c0cbb84) because k3d v5.5.0's default k3s (v1.26.4-k3s1) no longer accepts --no-deploy, causing the server container to fatal on startup ("flag provided but not defined: -no-deploy") and never report ready. Verified locally: reproduced the hang with --k3s-arg "--no-deploy=...", then confirmed cluster creation succeeds (all nodes Ready in ~20s) using the exact workflow args with --disable=traefik,metrics-server instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test-e2e kept failing at the "Install Infrastructure" step (job/ turing-init-init timing out) even after the earlier k3d/k3s-arg fixes. Root cause: k3d v5.5.0's default k3s (v1.26.4-k3s1, i.e. k8s 1.26) no longer serves policy/v1beta1 (PodDisruptionBudget) or autoscaling/ v2beta1 (HorizontalPodAutoscaler) - both removed from k8s in 1.25 - but Istio 1.12.5 (the version turing-init's chart installs by default) hardcodes those APIs in its install manifests, so istioctl install fails and the init job never completes. Verified locally: - Confirmed k3s v1.24.17-k3s1 still serves both policy/v1beta1 and autoscaling/v2beta1 (kubectl api-versions / raw API check). - Ran the actual Istio 1.12.5 istioctl install (with the repo's IstioOperator config from infra/charts/turing-init/values.yaml) against a k3d cluster pinned to k3s v1.24.17-k3s1 - istiod installed successfully with no API errors. - Re-created the cluster with the exact combined workflow args (1 server, 3 agents, port mapping, registry-create, --disable k3s-arg, and this --image pin) - all 4 nodes reach Ready on v1.24.17+k3s1 in under 20s. Fix: add --image rancher/k3s:v1.24.17-k3s1 to the k3d-action args so the cluster's Kubernetes version stays compatible with the pinned Istio version, independent of whatever k3s version k3d itself defaults to. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test-e2e's turing-postgresql-0/-mlp-0/-merlin-0 pods were stuck in ImagePullBackOff: Failed to pull image "docker.io/bitnami/postgresql:12.13.0": ... not found Bitnami stopped retaining versioned tags on the free docker.io/bitnami/* repos (only "latest" remains there now); historical tags were moved to docker.io/bitnamilegacy/*. Since turing-postgresql/mlp/merlin all pin image.tag: 12.13.0 (in infra/charts/turing/values.yaml and the vendored mlp/merlin subchart defaults), every instance failed to pull. infra/e2e/turing.values.yaml already centralizes postgresql config via a shared YAML anchor reused by turing/mlp/merlin, so pointing the image at bitnamilegacy there fixes all three in one place without touching the vendored chart tarballs. Verified locally: - `docker pull docker.io/bitnami/postgresql:12.13.0` reproduces the exact "not found" error from the CI logs. - `docker pull docker.io/bitnamilegacy/postgresql:12.13.0` succeeds. - `helm template` with this change shows all three postgres statefulsets (turing, turing-mlp, turing-merlin) now resolving to docker.io/bitnamilegacy/postgresql:12.13.0. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
anantadwi13
changed the title
…ro bug sdk/publish (make build -> python setup.py sdist bdist_wheel) failed: TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero' setuptools 71.0.0 through at least 74.1.2 call packaging.utils.canonicalize_version(..., strip_trailing_zero=...) from _core_metadata.py during sdist/egg_info, but that kwarg only exists in packaging>=22. requirements.txt pins mlflow>=1.26.1,<2.0.0, which resolves to mlflow 1.30.1 and hard-requires packaging<22, so the two constraints can't be satisfied together - whatever setuptools in that range is installed, egg_info breaks. setuptools 70.3.0 (last release below 71) doesn't have this call, so capping below 71 avoids the conflict entirely while keeping pkg_resources available (still only removed at 80, per the setuptools cap fixed in e97dd1a). Verified locally with the exact CI toolchain: pip install "setuptools>=64,<71" ... then -r requirements.txt -r requirements.dev.txt (resolves packaging==21.3, mlflow==1.30.1, setuptools==70.3.0) - `make build` produces the sdist/wheel successfully, and `make test` still passes (383 passed), matching the result from the prior sdk fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CI across
test-e2e,cluster-init, andsdkwas broken by externaldependencies that changed out from under us (Bitnami Docker Hub tag
retention, k3d/k3s release assets, setuptools defaults). This chases
each failure down with a real repro and fixes them one at a time:
bitnami/kubectl:1.22.16is gone fromDocker Hub (Bitnami dropped legacy versioned tags) — fetch kubectl
directly from the official Kubernetes release binaries instead.
unconstrained
setuptools>=80, which droppedpkg_resourcesandbroke the legacy build (
ModuleNotFoundError: pkg_resources) — pinsetuptools<75,Cython<3,numpy<2and build with--no-build-isolationso the outer pins are respected.run-cluster-initaction):AbsaOSS/k3d-action@v2.3.0's default k3d (v5.4.1) is missing itschecksums.txtrelease asset, so k3d's own install script 404sunder
set -e→ pink3d-version: v5.5.0.--no-deployflag → switched to--disable.policy/v1beta1andautoscaling/v2beta1, which the pinned Istio 1.12.5 installmanifests require → pin the cluster's k3s image to
v1.24.17-k3s1, keeping those APIs available.bitnami/postgresql:12.13.0is also gone fromDocker Hub (only
latestremains); the same tags are still publishedunder
bitnamilegacy/postgresql— repointed the e2e values there forturing/mlp/merlin's Postgres instances (all share one YAML anchor).
Test plan
make setup && make testpasses locally forsdkon Python3.12.13 (matching the CI matrix) — 383 passed.
install.shwithTAG=v5.5.0locally.--no-deployserver crash by creating alocal k3d cluster with the exact workflow args.
policy/v1beta1/autoscaling/v2beta1are served onk3s v1.24.17 but not v1.26.4, and ran the real Istio 1.12.5
install against a k3s v1.24.17 cluster successfully.
ImagePullBackOffviadocker pull,confirmed
bitnamilegacy/postgresql:12.13.0pulls, and verifiedvia
helm templatethat all three Postgres statefulsets resolveto the working image.
test-e2e(both matrix jobs),cluster-init, andsdkworkflows for this PR.🤖 Generated with Claude Code