feat(backend): add MLMD-to-native data migration and startup gate #14029

Chore description

Kubeflow Pipelines needs a supported upgrade path that migrates historical runtime
metadata from ML Metadata (MLMD) into the native KFP task, artifact, metric, and
relationship storage introduced by the MLMD-removal work.

The release also needs an API-server startup gate that detects an existing
MLMD-backed installation whose migration has not completed. In that state, the
server should refuse to start with an actionable error instead of becoming healthy
while historical runs, tasks, artifacts, metrics, relationships, and cache records
are silently unavailable through the new APIs.

This can be implemented separately from #13986, but it is a release blocker for
upgrades from MLMD-backed KFP versions unless the project explicitly declares those
upgrades unsupported and documents an alternative path.

Background

#11760 tracks removing MLMD as a KFP dependency. PR #13986 replaces MLMD-backed
runtime reads and writes with native run-scoped task and artifact APIs and removes
the MLMD deployment components.

The MLMD-removal design's Migration section
already defines the upgrade contract:

• provide a migration tool that converts MLMD executions into native tasks;
• recreate historical artifacts in native artifact storage;
• route metrics into the native metrics representation;
• preserve valid cache fingerprints;
• validate the migrated data; and
• require operators to opt in by running the migration, with the API server
  refusing to start when migration has not completed.

The corresponding migration and post-migration test cases
are currently marked as not automated.

Original implementation behavior (historical baseline)

There is no MLMD-to-native data migration or startup-completion gate in the current
MLMD-removal implementation.

On upgrade:

1. Historical execution, event, artifact, metric, and cache metadata remains in
   MLMD.
2. The new native Task and Artifact APIs have no corresponding rows.
3. The API server's normal
   AutoMigrate
   creates/reconciles the new tables, but it neither imports the MLMD data nor records
   that a validated data migration occurred.
4. The API server can start and report healthy even though historical runtime
   metadata is absent from the APIs now used by the UI, driver, launcher, and cache
   path.

This creates an additional artifact-access failure mode. Authenticated artifact
requests perform an exact URI lookup through the native Artifact API. If no row is
found, the frontend server's
artifact-then-prefix fallback
can establish ownership only when the object key encodes
private-artifacts/<namespace>/. Historical artifacts using an unnamespaced root,
including the shipped minio://mlpipeline/v2/artifacts
default or another non-canonical layout, cannot use that fallback. In an auth-enabled
deployment the request is correctly denied because ownership cannot be established,
even though the artifact and its ownership evidence still exist in MLMD.

Migrating trusted ownership metadata populates historical native records; it does
not authorize arbitrary object reads. The current master validator also requires
a namespace prefix, but the replacement content boundary in #14031 must preserve
already-authorized roots without requiring object moves or native registration.
It may authorize a constrained historical locator using trusted storage authority
without an Artifact API row. Neither a metadata miss nor an unnamespaced URI
is itself an access grant.

Reproduction

1. Install a KFP version that still uses MLMD for runtime metadata.
2. Run V2 pipelines that collectively include:
   • ordinary input and output artifacts;
   • metrics and visualization artifacts;
   • a cacheable component;
   • nested DAGs, loops/iterations, conditions, and an exit handler; and
   • at least two namespaces for multi-user validation.
3. Include an artifact stored below a historical URI such as
   minio://mlpipeline/v2/artifacts/... that does not encode
   private-artifacts/<namespace>/.
4. Confirm the completed executions, events, artifacts, and cache metadata exist in
   MLMD.
5. Upgrade to the MLMD-removal build without running a data migration.
6. Observe that the API server starts successfully.
7. Through the native APIs and UI, inspect the historical run graph, task details,
   inputs/outputs, metrics, relationships, visualizations, and artifact downloads.
8. Submit an equivalent pipeline that should reuse the historical completed cache
   entry.

Expected behavior: the upgrade is blocked until migration completes, or all
historical metadata remains available and correctly authorized after a supported
migration.

Current behavior: native records are absent; historical views and cache behavior can
be incomplete; artifacts with namespace-less roots remain rejected even with a migrated row,
unless a separately supported storage-ownership solution is delivered.

User and operator impact

• Historical run graphs may appear empty or incomplete.
• Task state, hierarchy, parameters, inputs, and outputs may be missing.
• Artifacts and metrics may disappear from run, artifact, visualization, and compare
  views.
• Artifact preview, visualization, and download may fail.
• Producer/consumer and input/output relationships may be lost.
• Previously valid completed cache entries may stop producing cache hits.
• An installation may accept new native records while leaving its old runtime history
  stranded in MLMD.
• Operators may discover the incompatible state only after the upgraded deployment is
  serving traffic because table creation alone does not prove migration completion.

This is primarily a metadata availability and upgrade-integrity problem. Object-store
payloads may still exist, but users lose the supported KFP metadata and authorization
path required to locate, interpret, visualize, or safely retrieve them.

Required implementation

1. Deliver an explicit migration mechanism

Provide either:

• a documented operator-run command/job/script; or
• an explicitly enabled, one-time API-server migration mode.

The migration must:

• support the database and deployment configurations covered by the KFP upgrade
  policy;
• document required backups and prerequisites, including the availability of MLMD;
• be idempotent or safely resumable after interruption;
• avoid duplicate tasks, artifacts, metrics, and relationships when rerun;
• report progress, source/destination counts, skipped records, and validation errors;
• avoid modifying artifact object data unless explicitly necessary; and
• write a durable migration-version/completion marker only after validation succeeds.

A dry-run or validation-only mode is strongly recommended.

2. Migrate task and run-time execution metadata

Convert relevant MLMD execution/context data into native tasks while preserving, as
applicable:

• run and namespace association;
• task identity, name, display name, and parent-child hierarchy;
• task type and type-specific attributes;
• creation, start, and finish timestamps;
• state, status metadata, and state history;
• input/output parameters, pod references, and scope information; and
• the stable identities needed by the native APIs and cache path.

Cover the execution/task forms described by the removal design: root DAG, generic
DAG, runtime/container, loop, loop iteration, condition, condition branch, and exit
handler. Include the documented exit-handler-* detection case where MLMD does not
otherwise identify that type declaratively.

3. Preserve cache correctness

Migrate cache fingerprints with their owning task metadata. As required by the
design, retain a ContainerExecution fingerprint only when the source execution is
complete. Failed, cancelled, or incomplete executions must not become valid cache
entries.

An equivalent post-upgrade pipeline should reuse an eligible migrated cache entry,
while ineligible source executions must remain cache misses.

4. Migrate artifacts, metrics, and relationships

For relevant MLMD artifacts, preserve:

• namespace and run ownership;
• URI and stable identity, or an auditable old-to-new identity mapping;
• name, schema/type, lifecycle state, timestamps, and custom metadata; and
• information required for current artifact providers and visualization behavior.

Route metric artifact types according to the native metrics storage/API contract
rather than treating them as ordinary downloadable artifacts.

Convert MLMD events into native artifact-task relationships while preserving:

• producer/consumer task and run identity;
• artifact key;
• input versus output direction; and
• iterator, OneOf, task-final-status, and other relationship semantics supported by
  the native API.

Every migrated relationship must reference an existing migrated run, task, and
artifact/metric in the same authorized namespace.

5. Validate before declaring completion

Before writing the completion marker, validate at least:

• source and destination counts, with documented exclusions;
• task-to-run and parent-task referential integrity;
• artifact/metric-to-task relationship integrity;
• namespace consistency and absence of cross-namespace relationships;
• cache-fingerprint eligibility;
• metric-versus-artifact routing;
• stable identity/deduplication invariants; and
• orphaned or unsupported MLMD records.

Validation failures must be actionable, leave the migration incomplete, and not
silently discard unsupported data.

6. Add an API-server startup gate

Startup must distinguish these states:

1. Fresh/empty installation: start normally.
2. Existing MLMD-backed installation without migration: refuse to start.
3. Migration started but incomplete or failed: refuse to start.
4. Expected migration version completed and validated: start normally.

The gate must use a durable version/completion marker plus any necessary schema/data
inspection. The existence of the new tables, or the presence of some newly created
rows, is not evidence that historical data was migrated.

The startup error should state:

• that MLMD-to-native migration is required;
• the expected migration version;
• how to run, resume, or validate it;
• that the operator should verify the required backup; and
• where to find upgrade and rollback documentation.

Security requirements

• Derive task/artifact ownership from trusted MLMD contexts/associations and existing
  KFP run metadata, never from a caller-supplied namespace alone.
• Persist the correct namespace on every migrated record and relationship.
• Preserve tenant boundaries and the existing SubjectAccessReview/RBAC behavior.
• Fail closed when the required authenticated storage authority cannot be established.
  Metadata API availability is not a universal prerequisite for the separately
  authorized historical content path defined in feat(api): move artifact authorization and content access behind a backend API #14031.
• Do not grant object access from arbitrary Artifact API misses, caller-created
  provenance, or unnamespaced historical URIs. Preserve authorized existing roots
  through trusted storage authority, without reintroducing runtime MLMD.
• Run migration with explicitly documented administrative privileges and least
  privilege where practical.
• Do not log credentials, signed query parameters, or secrets embedded in artifact
  URIs or provider metadata.
• Include multi-user tests proving that one namespace cannot list, inspect, visualize,
  or retrieve another namespace's migrated records.

Acceptance criteria

• A supported migration command/job or explicit one-time migration mode is
  included and documented.
• Migration is idempotent or safely resumable and does not create duplicates.
• Existing MLMD executions are represented by native tasks with correct run,
  namespace, type, hierarchy, state, timing, parameters, and relevant attributes.
• Only eligible completed executions retain cache fingerprints.
• Existing non-metric artifacts are available through the native Artifact API.
• Existing metrics are available through the native metric paths and frontend
  views.
• Artifact-task relationships preserve keys, direction, and supported relationship
  types.
• Historical run graphs and task details render correctly after migration.
• Historical artifacts can be listed, previewed, visualized, and downloaded after
  migration, including authorized non-prefix historical URIs. Migrated ownership
  rows alone do not grant object access; verify the independent storage authority
  and supported historical locators without native IDs.
• Equivalent pipelines reuse eligible migrated cache entries.
• Validation failures do not write the completion marker.
• The API server refuses to start for an existing unmigrated, partially migrated,
  or failed-migration installation.
• The API server starts for a fresh installation and for an installation with the
  expected successfully validated migration version.
• Startup failure contains actionable backup, migration, validation, and recovery
  guidance.
• Multi-user tests prove namespace isolation for all migrated records and artifact
  access paths.
• Upgrade, validation, backup, and rollback documentation is published.

Test plan

Create sanitized pre-upgrade database/MLMD fixtures containing:

• completed, failed, running, and cached executions;
• root/nested DAGs, loops and iterations, conditions and branches, and an exit handler;
• ordinary artifacts, visualization metadata, and metrics;
• producer/consumer relationships with the supported I/O types;
• multiple namespaces; and
• artifacts using both private-artifacts/<namespace>/... and historical
  unnamespaced URI layouts.

Automate the following:

1. Run migration and compare source inventory with destination records and reported
   exclusions.
2. Run migration a second time and verify no duplicates or destructive changes.
3. Interrupt migration and verify safe resume without a premature completion marker.
4. Verify historical and newly created runs side by side through native APIs and UI.
5. Verify historical artifact list, preview, visualization, and download behavior,
   independently checking metadata visibility and object access. Start with actual
   2.18 credentials/roots and no deferred artifact-adoption resources.
6. Verify cache hits from completed executions and cache misses from ineligible
   executions.
7. Verify native metrics retrieval and rendering.
8. Verify cross-namespace list, get, visualization, and download requests are denied.
9. Exercise the startup matrix: fresh, unmigrated, partially migrated, failed, wrong
   migration version, and successfully migrated.
10. Restore the documented backup and verify the prior MLMD-backed release remains
    operable.

Execution transition and 2.18 deferral decision (October 6, 2026)

TaskService authentication/backport PR #14707 is deferred from 2.18; release decision and tradeoffs are recorded in #14409 and #14421. The 2.18 line retains its existing TaskService behavior and exposure. The 3.0 line removes that service and authorizes the native task/cache APIs, but this does not make an in-place upgrade automatically safe.

Track the execution-transition acceptance alongside the historical-data migration in this issue. No particular automatic migration implementation is mandated by this decision, and the 2.18 prototype is not a shipped or verified 3.0 solution. Keep this issue open until both the supported data migration and the supported execution-transition contract have evidence, or link a separately owned release-blocking execution-transition issue before closing this one.

Required execution-transition decisions and verification:

• Define and validate treatment of running, pending and suspended old workflows, including future tasks, lifecycle/exit handlers and outstanding cache registrations when old APIs/MLMD are removed.
• Define and test retries of historical failed runs. Preserve completed nodes, outputs and identities where migration is supported; otherwise provide an explicit preflight rejection or documented alternative, not a silent fresh-run replay.
• Cover newly submitted native V2 IR runs, recurring runs, re-submitted saved packages, V1/V2-compatible/precompiled Argo packages and custom driver/launcher images. State unsupported cases explicitly; a new launch is not proof of a new client.
• Coordinate old/new API and controller replicas and in-flight create/retry operations so that supported old clients cannot unexpectedly appear after cutoff. Do not reopen anonymous APIs to preserve compatibility.
• Verify historical cache eligibility and retained ownership as well as active-client behavior. Disclose independently changed cache fingerprints and resulting re-execution; neither historical data import nor an empty-cache fallback alone proves preserved behavior.
• Publish preflight, operator-visible progress/blockers, interruption/recovery and rollback guidance for the chosen upgrade contract. Demonstrate it with actual old-to-new, multi-namespace runtime/upgrade tests before claiming support.

Immediate hard enforcement can fail old work; cache-miss/no-op compatibility can repeat costly or side-effectful tasks; preserving cache behavior during automatic drain requires additional coordination. Namespace-local drain is one possible approach, not an approved universal compatibility promise. Do not transfer unfinished 2.18 migration machinery wholesale to the different native 3.0 architecture.

Scope boundaries and related work

• feat(backend): replace MLMD tracking with native task and artifact APIs. Part of #11760 #13986 removes MLMD and introduces the native task/artifact runtime path.
• chore(backend/frontend): Remove MLMD as a depedency of KFP #11760 is the parent MLMD-removal effort.
• chore(backend): carry resiliency and scale guarantees into the post-MLMD architecture #14020 tracks post-MLMD runtime API resiliency, retries, deadlines, idempotency, and
  scale; those are not substitutes for migrating historical data.
• chore(frontend): scale and harden native runtime metadata views after MLMD removal #14028 tracks frontend scale and maintainability follow-ups and explicitly excludes
  MLMD-to-native data migration and upgrade gating.

Love this idea? Give it a 👍.