[jeffspahr]

Problem

The recurring-run lookup index added in #14679 also changed run_details.JobUUID from varchar(255) to varchar(191). MySQL cannot shrink that column with ALGORITHM=INPLACE; API startup can therefore copy a large run table and interrupt writes just to add a lookup index.

Change

Preserve the existing JobUUID type and width during schema reconciliation, including installations that already have varchar(191) and historical text columns. Keep the lookup index, but prepare it separately:

• MySQL explicitly requires ALGORITHM=INPLACE, LOCK=NONE and uses a nonunique 191-character prefix. 191 is a conservative index-prefix compatibility choice, not a stored-column limit or a modern database requirement. It supports historical text columns and older row formats without column conversion. Ordinary schedule UUIDs fit entirely within the prefix; longer identifiers remain intact and queries compare the full value. Equal prefixes can require extra comparisons. Keep this uniform approach rather than add schema-dependent index selection with no meaningful benefit for ordinary UUIDs.
• PostgreSQL creates a full-column index on "JobUUID" with CREATE INDEX CONCURRENTLY outside a transaction. It does not use the MySQL prefix or narrow existing columns. PostgreSQL's normal index-entry size limits still apply to unusually large historical text values.
• Reuse compatible existing indexes. Coordinate preparation across replicas on a dedicated session, bound metadata-lock waits to five seconds, and reject incompatible indexes or unsupported online DDL without a copying/blocking fallback.
• Log preparation and completion; preserve ordinary schema reconciliation for every other column. Fresh JobUUID columns use varchar(255).

This targets release-2.18: the narrowing/index tag was introduced on that branch. Updates the changelog and database development guidance; #14703 carries the operator/agent guides. Tracking: #14421.

Validation

• PostgreSQL cancellation fixtures now retain their blocking transaction until the original builder releases its advisory lock or its captured backend PID exits. All three fixtures passed 10 normal repetitions and 5 repetitions with a 500ms cancellation-delivery delay; invalid-index recovery, concurrent-write, conflicting-index preservation and idempotence assertions remain intact. Go lint and applicable pre-commit hooks passed. Updated to the release base containing fix(backend): automate retention index preparation on 2.18 #14877; fresh hosted CI is pending on 3320bdf99.
• CI prerequisite follow-up: backport master’s checksum-pinned GitHub CLI bootstrap before deployment artifact polling, with an immediate error if the CLI is missing. This fixes the SDK runner setup failure before tests started. All 7 bootstrap and 12 polling regressions plus applicable pre-commit hooks pass; prior SDK validation ran on 7cb1042fb.
• Full client-manager package passed with MySQL 8.0.46 and PostgreSQL 16 integrations enabled; full storage package passed.
• A populated MySQL test with 262,144 rows completed reconciliation/index preparation in 17.61 seconds, with 1,539 writes completing while ADD INDEX was still active. Maximum observed write latency was 1.21 seconds; InnoDB TABLE_ID remained unchanged. The removed narrowing fails the explicit online-algorithm negative control.
• Regression coverage includes existing varchar(255), varchar(191), and text definitions; long stored values; same-prefix/different-suffix equality; repeated startup; competing index preparers; bounded metadata-lock failure followed by successful retry; existing full-width indexes; PostgreSQL interrupted/invalid indexes and schema-scoped locking. Database tests use the existing backend CI service environment variables.
• Fresh warning-as-error Sphinx builds passed for implementation documentation and the companion guide updates.

Operational limits

This removes the JobUUID table-copy hazard, not every possible startup availability impact. Online index construction still consumes I/O/disk and takes brief metadata locks; waiting metadata locks can delay writes until the five-second timeout. Preparation is synchronous, with a 30-minute deadline, and failed attempts stop that API process's startup with an actionable error. MySQL advisory-lock acquisition waits up to ten seconds; PostgreSQL reports another preparer immediately. Existing replicas are not stopped by this helper.

An interrupted PostgreSQL concurrent build can leave an invalid index requiring inspection/repair before startup succeeds; it is never dropped automatically. Other startup schema changes/reference backfills remain separate, and separate retention-index automation and prepare-only staging are covered by #14877 (backport of #14876).

[github-actions]

Contributor Report for @jeffspahr

Metric	Value
Kubeflow org member	Yes
Issues opened in kubeflow/pipelines	339
Merged PRs in kubeflow/pipelines	281
GitHub account age	3463 days (created 2017-04-21)
PR comments in kubeflow/pipelines	1472

[droctothorpe]

AA review summary for #14874: fix(backend): preserve JobUUID and create its index online on 2.18

• Reviewed head SHA: eacd6a974f74bdafdc41f7cb7535497490eaaebc
• Eyeballs verdict: eyeballs required (ER)
• AA review result: passed
• Correctness / risk: Full static review found no introduced defect. JobUUID preservation is scoped, online DDL has no blocking/copy fallback, and lock/settings/DDL share one physical connection. Still synchronous startup: size probe budgets and rehearse representative tables; invalid PostgreSQL index recovery is operator-led. Does not certify unrelated migrations nonblocking.
• Standards / tests: Replacement tests preserve migration and query-plan coverage and prove writes overlap active DDL. Exact-head backend/docs/format checks passed; saved aggregate failed in SDK-client Kubernetes 1.36.1 job 115924791646. Diagnose and obtain required green CI. Optional: synchronize concurrent-preparer test overlap explicitly. No local engine tests or upgrade run.
• Blocking findings: 0
• Human review still required: yes

[droctothorpe]

/approve
/lgtm

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: droctothorpe

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [droctothorpe]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[google-oss-prow]

New changes are detected. LGTM label has been removed.