[jeffspahr]

Problem

Enabling run retention currently requires operators to create two database indexes manually. On large run histories, that makes index scheduling and recovery an operator task, while simply moving the SQL into API startup would risk blocking readiness and triggering retention catch-up as soon as a lengthy build finishes.

Changes

The elected GC worker now prepares the required indexes sequentially in the background. API startup does not await this DDL. Existing compatible, optimizer-visible indexes are reused; MySQL requires ALGORITHM=INPLACE, LOCK=NONE, and PostgreSQL uses CREATE INDEX CONCURRENTLY. No column narrowing, copying fallback or destructive index replacement is allowed.

• RUNS_GC_INDEX_BUILD_TIMEOUT: positive duration per index attempt, default 12h, separate from the five-second metadata-lock wait.
• RUNS_GC_INDEX_PREPARE_ONLY=true: prepare indexes without archiving/deleting, even with retention windows disabled. Apply consistently across replicas/installations; it cannot pause an old leader before that leader receives the configuration.
• New leaders, completed builds and recovery from failed preparation wait a fresh RUNS_GC_INTERVAL before collecting. GC remains gated on both valid indexes. This separates preparation from catch-up; prepare-only provides an explicit activation step for large installations.
• A dedicated session owns the database/schema advisory lock and DDL. Session settings are restored or the connection discarded. Failed preparation pauses GC and retries with exponential backoff/jitter capped at 15 minutes.
• Reject invisible MySQL indexes as unusable instead of allowing unindexed retention scans.

Retention remains disabled by default. Already-configured retention that was paused by missing indexes can now begin archival/deletion automatically after preparation and the interval. The release notes explain prepare-only staging and the remaining operational limits. These settings apply to retention indexes, independently of the JobUUID lookup-index work in #14874.

Verification

• Full common, gc, client_manager and API-server package tests passed, including configuration validation, preparation-only safety, first-leader/build/recovery separation, retries and cancellation.
• Real MySQL 8.0.46 and PostgreSQL16 tests passed: sequential preparation/reuse, conflicting indexes, build deadline/retry, independent ownership contention, invalid PostgreSQL index recovery and MySQL invisible-index refusal. MySQL metadata contention failed within the five-second lock bound despite a longer build deadline.
• Populated MySQL fixture: 262,144 rows, both index builds completed in 27.71s, 772 and 228 application read/write cycles completed while the respective ALTER was still active, maximum cycle latency 1.51s, and unchanged InnoDB table identity. This is evidence for the online path, not a performance estimate for a 100 GB deployment.
• Required hooks and fresh warning-as-error documentation build passed. Database regressions use the existing CI service environment variables.

Operational limits

Online creation consumes I/O, CPU and disk and still takes brief metadata locks. The configurable deadline bounds the client attempt; MySQL server cleanup may outlast cancellation, retaining the session advisory lock so another preparer cannot overlap it. PostgreSQL cancellation can leave an invalid index. The preparer now automatically recovers abandoned indexes matching the reserved name and exact expected definition using concurrent drop/rebuild within the same per-index deadline. Active or waiting database maintenance defers recovery. Conflicting definitions, custom storage settings and nondefault tablespaces remain untouched. Manual DDL on these managed names must be coordinated by stopping preparers or taking their advisory lock; activity checks cannot serialize arbitrary administrator DDL. Failures pause retention, not API startup.

Prepare-only can be enabled with retention disabled to build ahead of activation. Catch-up is a separate batched workload and can still be substantial. Existing valid indexes remain on disable/rollback. This is the release-2.18 backport of #14876; merge the master PR first. Tracking: #14421.

Backport validation: all four affected Go packages passed on this branch, as did MySQL/PostgreSQL recovery, ownership, visibility and metadata-lock tests, required hooks and the warning-as-error docs build. The populated performance evidence above comes from the identical preparation/GC implementation on the master candidate; it is not a separate 100 GB acceptance run. Adaptation is limited to release-note placement and the release-specific documentation link.

Interrupted PostgreSQL build recovery

Real PostgreSQL 16 tests interrupt a concurrent build, verify that it leaves an invalid index, and recover without manual SQL. Ten application read/write cycles complete while the replacement concurrent build is active. Tests also preserve an active external build and invalid indexes with wrong columns, descending order, expressions, nondefault operator classes/collations and custom storage settings. Retention remains paused until both indexes are ready, then waits the separate collection interval.

MySQL interruption and database-crash validation

Added an opt-in Docker harness, python3 test/retention-index-mysql-recovery.py, that creates and destroys only its own database. On 262,144 rows with 1 KiB payloads, it interrupts measured InnoDB scan/sort work after 25% of the current stage estimate, first by client cancellation and then by SIGKILL/restart with the same database volume. The production preparer retries without index-cleanup SQL. Tests require successful read/write cycles during recovery builds, usable indexes, the acknowledged pre-crash write, all rows, and unchanged InnoDB table identity. The crash case requires writes during reconstruction of the interrupted lifecycle index itself. Database downtime interrupts writes; the continuity claim applies to online builds before/after restart.

MySQL 8.0.46 validation passed. Crash recovery reconnected in 11.49 seconds, with 590 lifecycle and 593 archived read/write cycles completing during the subsequent builds; the largest observed cycle was 213 ms. These fixture measurements are not a 100 GB latency or recovery-time guarantee.

The identical fixture also passed on MySQL 8.4.11 against the 2.18 backport: crash reconnect 4.65 seconds, 236 lifecycle and 247 archived read/write cycles completed during rebuilds, maximum observed cycle 229 ms. Ordinary package tests, repository hooks and documentation build passed.

Release-base reconciliation

Rebased onto release-2.18 90dcc99a9. Resolved the release-note overlap by retaining both the Python support update and the retention rollout guidance. The retention implementation and regression tests are unchanged. Release-note hooks, whitespace checks, and fresh go test ./backend/src/common ./backend/src/apiserver/gc ./backend/src/apiserver/client_manager ./backend/src/apiserver -count=1 passed.

[github-actions]

Contributor Report for @jeffspahr

Metric	Value
Kubeflow org member	Yes
Issues opened in kubeflow/pipelines	336
Merged PRs in kubeflow/pipelines	279
GitHub account age	3463 days (created 2017-04-21)
PR comments in kubeflow/pipelines	1468

[droctothorpe]

AA review summary for #14877: fix(backend): automate retention index preparation on 2.18

• Reviewed head SHA: 69b143030aec4047fe10c86279f9ae65f8811e9a
• Eyeballs verdict: eyeballs required (ER)
• AA review result: passed
• Correctness / risk: Full static review found no backport defect. Shared retention helper/tests/runtime verified by pinned blob IDs; release-specific integration, configuration and docs independently inspected. Separate JobUUID migration is not claimed fixed here. ER for production data behavior; consistent prepare-only rollout and capacity rehearsal remain necessary.
• Standards / tests: Release backport preserves substantive engine/GC regressions and branch-appropriate guidance. Saved exact-head backend/docs/format/aggregate checks succeeded. Honor feat(backend): prepare retention indexes online before collection #14876-first prerequisite, then final-base verification and human approval. Opt-in crash fixtures and live upgrade acceptance were not executed locally.
• Blocking findings: 0
• Human review still required: yes

[droctothorpe]

/approve
/lgtm

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: droctothorpe

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [droctothorpe]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[google-oss-prow]

New changes are detected. LGTM label has been removed.