[WIP] chore(backend): Storage-layer refactor & Postgres enablement roadmap by kaikaila · Pull Request #12063 · kubeflow/pipelines

kaikaila · 2025-07-17T19:33:12Z

Superseded by #12379.
Further development and review will continue there.

Background

Kubeflow Pipelines currently hardcodes MySQL assumptions in its storage layer. To enable PostgreSQL adoption (via pgx driver) and improve long-term maintainability, we need to decouple SQL dialect logic and centralize it into a reusable abstraction.

⸻

Changes

This PR introduces a major refactor of the backend storage layer and related test workflows:
Dialect abstraction
• Introduced DBDialect interface and refactored pipeline/experiment/run/job stores to use it.
• Removed legacy SQLDialect, centralized initialization in InitDBClient.
Storage refactor
• Updated resource_reference, db_status_store, and cache/storage clients to use the new dialect.
• Moved filter logic into dialect package, cleaned up list package.
• Adapted db_fake to support dialect abstraction.
Tests & dependencies
• Updated unit tests across storage, list, and filter packages.
• Updated go.mod dependencies and tidied modules.
• Added PostgreSQL GitHub Actions workflow with kustomize deployment and port-forward.

⸻

CI Status

In the upstream CI, the failures are infra-related: tests share the same database and pollute each other.
Running the tests in parallel jobs resolves this issue and results in all tests passing, as shown in this PR's CI results.
Per maintainer feedback, PR#12251 was closed to make room for the ongoing test overhaul PR#12222.
This PR includes the fixes from PR#12251, so CI results here are green.

…flow#11924) * Minimize the Kubernetes version range in the CI This reduces the matrix to only include the low and high versions. See the KFP community call notes for more context: https://docs.google.com/document/d/1cHAdK1FoGEbuQ-Rl6adBDL5W2YpDiUbnMLIwmoXBoAU/edit?tab=t.0#heading=h.fovolzywu84d Signed-off-by: mprahl <mprahl@users.noreply.github.com> * Reduce the testing matrix for Python versions This reduces the Python versions being tested to the low and high versions to reduce GitHub CI consumption. See the KFP community call discussion for more context: https://docs.google.com/document/d/1cHAdK1FoGEbuQ-Rl6adBDL5W2YpDiUbnMLIwmoXBoAU/edit?tab=t.0#heading=h.fovolzywu84d Signed-off-by: mprahl <mprahl@users.noreply.github.com> * Add an option to run integration tests locally Signed-off-by: mprahl <mprahl@users.noreply.github.com> * Stop using whalesay in the tests The container image is over 10 years old and is in a format that is deprecated on newer Kubernetes versions. Signed-off-by: mprahl <mprahl@users.noreply.github.com> --------- Signed-off-by: mprahl <mprahl@users.noreply.github.com>

…11908) Signed-off-by: agoins <alyssacgoins@gmail.com>

…ubeflow#11936) Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.4.2 to 6.5.1. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](tornadoweb/tornado@v6.4.2...v6.5.1) --- updated-dependencies: - dependency-name: tornado dependency-version: 6.5.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Fixes flaky test behavior in kfp-samples. Signed-off-by: agoins <alyssacgoins@gmail.com>

Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>

Signed-off-by: rsun19 <robertssun1234@gmail.com>

* Nest SDK readthedocs documentation Signed-off-by: Anish Asthana <anishasthana1@gmail.com> * Merge sdk OWNERS with KFP website approvers Signed-off-by: Anish Asthana <anishasthana1@gmail.com> --------- Signed-off-by: Anish Asthana <anishasthana1@gmail.com>

* chore: Adding Devin's DeepWiki to README Signed-off-by: Francisco Javier Arceo <farceo@redhat.com> using Devin's badge instead Signed-off-by: Francisco Javier Arceo <farceo@redhat.com> Adding badge and section: Signed-off-by: Francisco Javier Arceo <farceo@redhat.com> * adding warning and moved below documentation Signed-off-by: Francisco Javier Arceo <farceo@redhat.com> --------- Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>

) Signed-off-by: rsun19 <robertssun1234@gmail.com>

Signed-off-by: Michael <m.zazula@gmail.com>

…anges (kubeflow#11934) Signed-off-by: VaniHaripriya <vmudadla@redhat.com>

… versions (kubeflow#11952) * Add the name field for pipelines and pipeline versions The API only accepted display_name, which accepted arbitrary characters but this does not work well in Kubernetes native API mode (Kubernetes pipeline store) since object names must be a valid DNS subdomain. This commit adds the name field which is required, but for backwards compatibility, the API will accept just name or display_name and duplicate the set value to the unset field. For existing users, they will only see that there is an extra field of "name" on pipelines and pipeline versions that is the same value as "display_name". The database migration logic adds a new DisplayName column and sets the values to what was in the Name column. Confusingly, the API field "display_name" used to map the a database column of "Name". The UI now handles both name and display name but for a simpler user experience, the display name form fields when uploading a pipeline are hidden when not in Kubernetes native API mode (Kubernetes pipeline store) since there is no need to have a display name since there is no naming restrictions on the name. Signed-off-by: mprahl <mprahl@users.noreply.github.com> * Add a better API error when a Kubernetes object name is invalid Signed-off-by: mprahl <mprahl@users.noreply.github.com> --------- Signed-off-by: mprahl <mprahl@users.noreply.github.com>

Signed-off-by: Laurent Goderre <laurent.goderre@docker.com>

…amples/ffdl-seldon/source/seldon-pytorch-serving-image (kubeflow#11959) Bumps [torch](https://github.com/pytorch/pytorch) from 2.6.0 to 2.7.1. - [Release notes](https://github.com/pytorch/pytorch/releases) - [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md) - [Commits](pytorch/pytorch@v2.6.0...v2.7.1) --- updated-dependencies: - dependency-name: torch dependency-version: 2.7.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…beflow#11920) * fix(cache): Optimize cache expiration query, add index to db column, rm duplicate cache row insertions Signed-off-by: cbartram <cbartram3@gmail.com> * fix(cache): Use composite index for cache started_at_in_sec and id fields Signed-off-by: cbartram <cbartram3@gmail.com> Signed-off-by: cbartram <cbartram3@gmail.com> * chore(format): Fix formatting, rm commented code Signed-off-by: cbartram <cbartram3@gmail.com> Signed-off-by: cbartram <cbartram3@gmail.com> --------- Signed-off-by: cbartram <cbartram3@gmail.com>

) Add support for exit handlers to take input PipelineTaskFinalStatus. Signed-off-by: agoins <alyssacgoins@gmail.com>

…kubeflow#11960) Signed-off-by: Laurent Goderre <laurent.goderre@docker.com>

Signed-off-by: John Thompson <jpthompson23@gmail.com>

…ib/arena/docker (kubeflow#11963) Bumps [requests](https://github.com/psf/requests) from 2.18.4 to 2.32.4. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.18.4...v2.32.4) --- updated-dependencies: - dependency-name: requests dependency-version: 2.32.4 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

After making name and display_name separate fields in the API, the pipeline sample upload code needed to accomodate that, otherwise it would try to upload the pipeline without a display name, which is a required field. The backwards compatibility was handled at the API server level but this code bypassed the API server endpoints and used the resource manager directly. Signed-off-by: mprahl <mprahl@users.noreply.github.com>

…nal-test (kubeflow#11966) Bumps [requests](https://github.com/psf/requests) from 2.32.2 to 2.32.4. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.32.2...v2.32.4) --- updated-dependencies: - dependency-name: requests dependency-version: 2.32.4 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: mprahl <mprahl@users.noreply.github.com>

…ubeflow#11971) Bumps and [brace-expansion](https://github.com/juliangruber/brace-expansion). These dependencies needed to be updated together. Updates `brace-expansion` from 2.0.1 to 2.0.2 - [Release notes](https://github.com/juliangruber/brace-expansion/releases) - [Commits](juliangruber/brace-expansion@v2.0.1...v2.0.2) Updates `brace-expansion` from 1.1.11 to 2.0.2 - [Release notes](https://github.com/juliangruber/brace-expansion/releases) - [Commits](juliangruber/brace-expansion@v2.0.1...v2.0.2) --- updated-dependencies: - dependency-name: brace-expansion dependency-version: 2.0.2 dependency-type: indirect - dependency-name: brace-expansion dependency-version: 2.0.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ubeflow#11972) Bumps [brace-expansion](https://github.com/juliangruber/brace-expansion) from 1.1.11 to 1.1.12. - [Release notes](https://github.com/juliangruber/brace-expansion/releases) - [Commits](juliangruber/brace-expansion@1.1.11...v1.1.12) --- updated-dependencies: - dependency-name: brace-expansion dependency-version: 1.1.12 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…server (kubeflow#11978) Bumps [brace-expansion](https://github.com/juliangruber/brace-expansion) from 1.1.11 to 1.1.12. - [Release notes](https://github.com/juliangruber/brace-expansion/releases) - [Commits](juliangruber/brace-expansion@1.1.11...v1.1.12) --- updated-dependencies: - dependency-name: brace-expansion dependency-version: 1.1.12 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…nal-test (kubeflow#11985) Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 4.25.3 to 4.25.8. - [Release notes](https://github.com/protocolbuffers/protobuf/releases) - [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/protobuf_release.bzl) - [Commits](protocolbuffers/protobuf@v4.25.3...v4.25.8) --- updated-dependencies: - dependency-name: protobuf dependency-version: 4.25.8 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>

…kubeflow#11986) Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 4.25.4 to 4.25.8. - [Release notes](https://github.com/protocolbuffers/protobuf/releases) - [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/protobuf_release.bzl) - [Commits](protocolbuffers/protobuf@v4.25.4...v4.25.8) --- updated-dependencies: - dependency-name: protobuf dependency-version: 4.25.8 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…gemaker (kubeflow#11993) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.15 to 2.5.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.15...2.5.0) --- updated-dependencies: - dependency-name: urllib3 dependency-version: 2.5.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: zazulam <m.zazula@gmail.com>

…ckage Signed-off-by: kaikaila <lyk2772@126.com>

Signed-off-by: kaikaila <lyk2772@126.com>

apiserver launched successfully in kind cluster Signed-off-by: kaikaila <lyk2772@126.com>

Signed-off-by: kaikaila <lyk2772@126.com>

Signed-off-by: kaikaila <lyk2772@126.com> Signed-off-by: kaikaila <lyk2772@126.com>

Signed-off-by: kaikaila <lyk2772@126.com>

- Add os.Getenv("TLS_CERT_DIR") to allow custom certificate directory - Add startup success log messages for better debugging - Improve logging with cache server ready message This change enables local debugging without modifying /etc/webhook/certs path. Signed-off-by: kaikaila <lyk2772@126.com>

- Add TLS certificate generation script - Add local setup verification script - Add manual webhook testing script - Add deployment verification script - Add VSCode launch configuration for cache-server - Add test data files for webhook testing These tools enable local cache-server development and testing. Signed-off-by: kaikaila <lyk2772@126.com>

- Add CACHE_WORKFLOW.md explaining cache mechanism - Add LOCAL_DEV.md for local development setup - Add MANUAL_TESTING.md with detailed testing guide - Add QUICK_START.md for quick reference These docs help developers understand and test cache-server locally. Signed-off-by: kaikaila <lyk2772@126.com>

Signed-off-by: kaikaila <lyk2772@126.com>

Prevent certificate files from being accidentally committed to the repository. Signed-off-by: kaikaila <lyk2772@126.com>

google-oss-prow · 2025-10-19T03:37:13Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chensun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

backend/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mprahl and others added 30 commits May 28, 2025 13:37

feat(backend) implement retryStrategy for nested pipelines (kubeflow#…

c3b1638

…11908) Signed-off-by: agoins <alyssacgoins@gmail.com>

fix(tests): free up space in kfp samples test workflow (kubeflow#11942)

5364e1e

Fixes flaky test behavior in kfp-samples. Signed-off-by: agoins <alyssacgoins@gmail.com>

chore: pipeline api spec minor release (kubeflow#11939)

d49f9aa

Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>

docs(frontend): added build instruction to readme (kubeflow#11943)

5890ef0

Signed-off-by: rsun19 <robertssun1234@gmail.com>

docs(frontend): Added clean up section to openshift docs (kubeflow#11951

9900e2c

) Signed-off-by: rsun19 <robertssun1234@gmail.com>

chore: add c1 to ADOPTERS.md (kubeflow#11954)

7910c5e

Signed-off-by: Michael <m.zazula@gmail.com>

feat(sdk): update PipelineConfig to reflect new workspace Protobuf ch…

cb3b8dc

…anges (kubeflow#11934) Signed-off-by: VaniHaripriya <vmudadla@redhat.com>

feat(frontend) Use native k8s probes for pipeline-ui (kubeflow#11955)

d0cd470

Signed-off-by: Laurent Goderre <laurent.goderre@docker.com>

feat(backend/sdk): support PipelineTaskFinalStatus input (kubeflow#11953

e2f70a4

) Add support for exit handlers to take input PipelineTaskFinalStatus. Signed-off-by: agoins <alyssacgoins@gmail.com>

feat(backend): Use native k8s probes for api-server and visualization (…

18250e9

…kubeflow#11960) Signed-off-by: Laurent Goderre <laurent.goderre@docker.com>

test(backend): Add unit test for NewLauncherV2 (kubeflow#11854)

2364066

Signed-off-by: John Thompson <jpthompson23@gmail.com>

fix(test): Fix the frontend sample test (kubeflow#11968)

fc78e8a

Signed-off-by: mprahl <mprahl@users.noreply.github.com>

user master for dev manifests (kubeflow#11977)

7455870

Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>

chore: update workflows to test proto changes in ci (kubeflow#11995)

39ce42b

Signed-off-by: zazulam <m.zazula@gmail.com>

kaikaila added 22 commits October 17, 2025 16:36

refactor(storage): move Filter logic into dialect and cleanup list pa…

a0b2ef9

…ckage Signed-off-by: kaikaila <lyk2772@126.com>

refactor(storage): update resource_reference and db_status_store

f93e22b

Signed-off-by: kaikaila <lyk2772@126.com>

test(storage): update unit tests for storage, list, filter

3b057b1

Signed-off-by: kaikaila <lyk2772@126.com>

refactor(cache): adapt cache client and storage to new dialect

6f19f9f

Signed-off-by: kaikaila <lyk2772@126.com>

refactor(storage): adapt db_fake with dialect.DBDialect

3ed44e3

Signed-off-by: kaikaila <lyk2772@126.com>

chore(deps): update go.mod and tidy dependencies

51547bb

Signed-off-by: kaikaila <lyk2772@126.com>

ci(pg): add PostgreSQL workflow with kustomize deploy & port-forward

b9691f2

Signed-off-by: kaikaila <lyk2772@126.com>

wip:integrate pgx into deploy-kfp.sh;

1ba568e

apiserver launched successfully in kind cluster Signed-off-by: kaikaila <lyk2772@126.com>

add logging to show dirty runs and dump run_details in mysql pod

ea7d1a9

Signed-off-by: kaikaila <lyk2772@126.com>

wip: debug TestCache/TestCacheSingleRunWithPVC_SameName_Caches

8cceac9

Signed-off-by: kaikaila <lyk2772@126.com>

fixup workflows

ab4574a

Signed-off-by: kaikaila <lyk2772@126.com>

fix TestGetExperiment_JsonOmitEmpty

074ae3d

Signed-off-by: kaikaila <lyk2772@126.com>

group by 保留run_details.UUID,舍弃longtext字段。

9259d87

Signed-off-by: kaikaila <lyk2772@126.com>

update pre-commit yapf

91d52bd

Signed-off-by: kaikaila <lyk2772@126.com>

update generated files

c7b0f29

Signed-off-by: kaikaila <lyk2772@126.com>

c 开头

f1286ad

Signed-off-by: kaikaila <lyk2772@126.com>

debug json serialization

37d378b

Signed-off-by: kaikaila <lyk2772@126.com>

not ginkgo, but testify/suite

d7d9bce

Signed-off-by: kaikaila <lyk2772@126.com>

fix(backend): update test expectations for Job Mode field

477d8a8

Signed-off-by: kaikaila <lyk2772@126.com>

rename workflows for postgres and apply Ginkgo

f24764a

Signed-off-by: kaikaila <lyk2772@126.com> Signed-off-by: kaikaila <lyk2772@126.com>

feat(ci): Add cache-server to image build and deployment pipeline

a2501ad

Signed-off-by: kaikaila <lyk2772@126.com>

本地用的确认cache连上postgres

9ca4190

Signed-off-by: kaikaila <lyk2772@126.com>

github-advanced-security AI found potential problems Oct 19, 2025

View reviewed changes

Comment thread backend/src/cache/certs/key.pem Fixed

kaikaila added 5 commits October 18, 2025 20:10

fixup: cache server source code

03b1fb7

Signed-off-by: kaikaila <lyk2772@126.com>

chore(cache): Add .gitignore for certificate files

5306cfd

Prevent certificate files from being accidentally committed to the repository. Signed-off-by: kaikaila <lyk2772@126.com>

kaikaila mentioned this pull request Oct 24, 2025

feat(backend): postgres integration #12379

Merged

Navigation Menu

Repository navigation

[WIP] chore(backend): Storage-layer refactor & Postgres enablement roadmap - #12063

[WIP] chore(backend): Storage-layer refactor & Postgres enablement roadmap#12063
kaikaila wants to merge 6581 commits into
kubeflow:masterfrom
kaikaila:feature/replace-str-with-largetext-WIP

kaikaila commented Jul 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

google-oss-prow Bot commented Oct 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

kaikaila commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Changes

CI Status

Uh oh!

Uh oh!

google-oss-prow Bot commented Oct 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

kaikaila commented Jul 17, 2025 •

edited

Loading