Skip to content

#475 Lineage Maturity (support linking a child partitioning to multiple parents) - #496

Open
lsulak wants to merge 46 commits into
masterfrom
feature/475-multiple-child-parent-linkage-possible
Open

#475 Lineage Maturity (support linking a child partitioning to multiple parents)#496
lsulak wants to merge 46 commits into
masterfrom
feature/475-multiple-child-parent-linkage-possible

Conversation

@lsulak

@lsulak lsulak commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Overview

One idempotent POST /api/v2/partitionings now creates or links a child partitioning to 0..N parents, with a cycle check shared with PATCH /ancestors; measure-definition copying is opt-in and API v1 is deprecated.

Release Notes

  • Breaking: POST /api/v2/partitionings is create-or-link for 0..N parents: 201 created, 200 already existing (was 409), 404 unknown parent, 409 cycle or an existing partitioning with descendants getting a new parent.
  • Breaking: new children no longer inherit the parents' measure definitions; opt in with copyMeasureDefinitions (agent overloads). Additional data is never copied.
  • PATCH /ancestors rejects a self-link, a cycle, or a partitioning with descendants with 409; descendants are never propagated.
  • API v1 is deprecated, behaviour unchanged.
  • Breaking (library): PartitioningSubmitV2DTO changed shape; Dispatcher.createPartitioning takes PartitioningCreateRequest.
  • Breaking (deployment): deploy the server before the agents; a new agent against an older server creates the child unlinked, silently. Development databases that applied earlier unreleased V0.10.0.x migrations must be recreated.

Related

Closes #475

lsulak and others added 21 commits October 19, 2026 22:51
@lsulak
feat(01-01): add private helper runs._copy_parents_data_to_partitioning
08dfd09 
- union + dedup copy of measure definitions across copy-from parents (D-03)
- child-then-skip additional data merge with skipped_ad_keys reporting (D-04, D-04a)
- owned by atum_owner, EXECUTE revoked from PUBLIC (T-01-01)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
test(01-01): add balta suite for runs._copy_parents_data_to_partitioning
9005871 
- 9 tests: measure union + dedup, child-then-skip additional data, NULL value
  semantics, conflict-scope vs copy-from, empty set, array hygiene
- two-column filters use SQL conditions to avoid balta 0.1.0 param-order bug

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
feat(01-02): add multi-parent overload runs.create_partitioning(JSONB…
9176f4d 
…, TEXT, BIGINT[])

- validates all parents up front (status 41, nothing written)
- creates or reuses the child (no status 31), links every parent idempotently
- copies parents' data via runs._copy_parents_data_to_partitioning (RD-2 ancestry-based copy-from set)
- EXECUTE revoked from PUBLIC, granted to atum_user only

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
test(01-02): add balta suite for multi-parent runs.create_partitionin…
ccdeccd 
…g overload

- 15 tests: two-parent link, atomic status 41, NULL/empty/single-element arrays,
  re-link (13 not 31), idempotent resubmit (14), POLICY-01 end-to-end,
  reversed-id topology (RD-2), shipped overload resolution

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
fix(01): harden copy helper - pinned search_path, missing-child statu…
02dae72 
…s, copied_rows, tie-break and privilege tests (WR-04, WR-07, WR-08, IN-03, IN-05, IN-06)

- pin search_path (pg_catalog, public, pg_temp) on the SECURITY DEFINER helper; public is needed
  because column defaults call public.global_id()
- return status 41 for a NULL/non-existent child instead of silently doing nothing or failing on FKs
- add OUT copied_rows (rows actually inserted) so callers can tell a no-op from a write
- document why copied measure definitions keep the parent's created_by/created_at while
  copied additional data is stamped with i_by_user (matches shipped create_partitioning)
- tests: earliest-parent tie-break with explicit created_at, 41 for missing/NULL child,
  has_function_privilege (owner only), pinned search_path; hasNext asserts, drop redundant assert

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
fix(01): rename to runs.create_partitioning_with_parents and fix cycl…
26e752c 
…e, conflict, concurrency and linking semantics (CR-01, CR-02, WR-01, WR-02, WR-03, WR-04, WR-05, WR-06, WR-08, IN-02, IN-04, IN-05)

Amends the unreleased V0.10.0.2 migration in place (renamed to V0.10.0.2__create_partitioning_with_parents.sql).

- WR-02: new function name runs.create_partitioning_with_parents(JSONB, TEXT, BIGINT[]); the shipped
  runs.create_partitioning is the only function of its name again, so untyped positional NULL resolves
- CR-02: reject a parent that is the partitioning itself or its descendant with status 43, before any
  write (write-side guard only; serialized by an advisory xact lock)
- CR-01: additional data conflict scope is every ancestor of the partitioning after linking, so a key
  skipped for a conflict is not filled in later by another parent
- WR-01: data of every supplied parent is always copied (idempotent), so {p1,p2} and {p1} then {p1,p2} converge
- WR-03: INSERT ... ON CONFLICT DO NOTHING + re-select, a concurrent create no longer raises partitioning_unq
- WR-04: SET search_path = pg_catalog, public, pg_temp (public is needed for public.global_id())
- WR-05: descendants of an existing partitioning are added to the flows of the newly linked parents
- WR-06: status 13 whenever links or data were written, 14 only for a pure no-op
- IN-02: skipped keys returned in OUT skipped_ad_keys, status_text only carries their count
- IN-04: existing partitioning with NULL/empty parents returns 14, pinned by a test
- tests: renamed suite, 24 tests incl. CR-01/CR-02/WR-01/WR-02/WR-05/WR-06 scenarios, privileges,
  pinned search_path, AD created_by stamping

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
feat(02-01): multi-parent PartitioningSubmitDTO with compat apply and…
0f99d17 
… custom codec

- store parentPartitionings: Seq[PartitioningDTO]; Option compat apply
- encoder always emits parentPartitionings; decoder accepts legacy parentPartitioning
- extend serialization tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
feat(02-01): adapt agent and server to parentPartitionings
f868277 
- use parentPartitionings.headOption to keep single-parent behaviour

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
feat(03-01): multi-parent PartitioningSubmitV2DTO with compat codec
ea1da28 
- store parentPartitioningIds: Seq[Long]; decode old/new JSON, encode new key only
- CreatePartitioning passes parentPartitioningIds.headOption

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
feat(03-01): route 2+ parents to create_partitioning_with_parents
4650c07 
- new CreatePartitioningWithParents wrapper (typed BIGINT[] arg, 43 -> conflict)
- repository de-duplicates parent ids and routes 0-1 vs 2+ parents
- wire layer in Main; unit and DB integration tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
feat(03-02): dedupe and cap parent ids in v2 postPartitioning, add 40…
626ca6d 
…4 variant

- Controller de-duplicates parent ids, rejects >100 with 400 before service call
- POST /partitionings endpoint exposes 404 variant
- Controller and service tests for cap, dedupe, 404/409 propagation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
test(03-02): tapir stub tests for multi-parent POST /partitionings
2a806de 
- 201 for multi-parent, legacy single-parent JSON and no-parent bodies
- 404/409/400 mapping at HTTP level

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
feat(05-01): send all parent ids in one POST from HttpDispatcher
6c36661 
- Resolve every parent id; 2+ parents always POST once (no child GET, no PATCH)
- 0-1 parent flow unchanged
- Tests decode POST body and assert parentPartitioningIds order

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
test(05-01): avoid deprecated JavaConverters for Scala 2.13 axis
5298e16 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
feat(05-02): add multi-parent getOrCreateAtumSubContext to AtumAgent
bfd162e 
- single createPartitioning dispatch with deduped parents, child identity = subPartitions
- empty parents throw IllegalArgumentException

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
feat(05-02): add AtumContext.subContext for multi-parent sub-contexts
e6e1a91 
- delegates to AtumAgent multi-parent overload
- document in agent README

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
test(04-01): multi-parent ancestors DB tests (two/three parents, diam…
cd5b3d1 
…ond, pagination, unq_flows)

- No duplicates possible (unq_flows + PK); V0.10.0.3 migration not needed

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
test(04-01): server v2 ancestors tests with multi-parent data
6c9fdea 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
test(06-01): add multi-parent E2E agent-server compatibility test
c566df9 
- child created once via getOrCreateAtumSubContext(sub, Seq(p1,p2))
- asserts ancestors == {A,B} via server v2 endpoint and flow links without duplicates

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
docs(06-01): document multi-parent agent/server version skew
ea11321 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
test(06-01): make compatibility tests re-runnable with per-run partit…
e06f707 
…ion values

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 21, 2026 12:15
Copilot AI reviewed Oct 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@github-actions

github-actions Bot commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown

JaCoCo Coverage Report

Metric (instruction) Coverage Threshold Status
Overall 78.57% 64.0% ✅
Changed Files 96.6% 80.0% ✅
Group Coverage (O/Ch) Threshold (O/Ch) Status (O/Ch)
atum-agent 88.56% / 87.61% 78.0% / 80.0% ✅/✅
atum-model 40.44% / 84.62% 34.0% / 80.0% ✅/✅
atum-server 93.84% / 98.73% 78.0% / 80.0% ✅/✅
Report Coverage (O/Ch) Threshold (O/Ch) Status (O/Ch)
Report: agent_spark3 - scala:2.13.18 88.56% / 87.61% 78.0% / 80.0% ✅/✅
Report: model - scala:2.13.18 40.44% / 84.62% 34.0% / 80.0% ✅/✅
Report: server - scala:2.13.18 93.84% / 98.73% 78.0% / 80.0% ✅/✅
File Path Coverage Threshold Status
AtumAgent.scala 85.9% 60.0% ✅
AtumContext.scala 96.77% 60.0% ✅
BaseEndpoints.scala 99.7% 60.0% ✅
CapturingDispatcher.scala 87.21% 60.0% ✅
CreatePartitioningWithParents.scala 97.41% 60.0% ✅
CreationResult.scala 100.0% 60.0% ✅
Endpoints.scala 98.85% 60.0% ✅
GetPartitioningAncestors.scala 97.03% 60.0% ✅
HttpDispatcher.scala 83.8% 60.0% ✅
PartitioningControllerImpl.scala 90.23% 60.0% ✅
PartitioningCreateRequest.scala 100.0% 60.0% ✅
PartitioningRepositoryImpl.scala 98.78% 60.0% ✅
PartitioningServiceImpl.scala 97.32% 60.0% ✅
PartitioningSubmitV2DTO.scala 84.62% 60.0% ✅

Run 42022061062 · Event: pull_request

@lsulak
lsulak marked this pull request as draft October 21, 2026 14:33
@lsulak lsulak added the breaking change Introduces change, that is incompatible with last release label Nov 3, 2026
lsulak and others added 16 commits November 3, 2026 17:57
@lsulak
feat(database): idempotent link-to-parents and measure-copy opt-in (#475
7a29e9c 
)

Add flows._link_to_parents with cycle check and narrow descendant
propagation, rewrite runs.create_partitioning_with_parents, and add
runs.update_partitioning_parent. Measure definitions are copied only on
request; additional data is never copied on create.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
feat(server): single v2 POST for 0..N parents, 201/200, deprecate v1 (#…
c5a9117 
…475)

POST /api/v2/partitionings is an idempotent create-or-link: 201 when
created, 200 when existing, 404 for an unknown parent and 409 for a
cycle. copyMeasureDefinitions is optional and defaults to false. v1
create operations are marked deprecated. GET ancestors of a root now
returns an empty list instead of a 500.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
feat(agent): one POST per sub-context and opt-in copyMeasureDefinitio…
26539c4 
…ns (#475)

HttpDispatcher sends a single create POST for 0..N parents. New overloads
take copyMeasureDefinitions explicitly with no default; existing overloads
keep v0.10.0 signatures and pass false.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
test(api): E2E Postman folder and compat scenarios for linking (#475)
15fff32 
Add the v1.1 Postman folder and five AgentServerCompatibilityTests
scenarios; the original 24 requests and 3 compat tests are unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
docs: linking guarantees, upgrade and compatibility notes (#475)
ebef975 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
Merge branch 'master' into feature/475-multiple-child-parent-linkage-…
69714f4 
…possible
@lsulak
docs: drop ADR 02 from this PR
9e47c75 
The ADR is kept locally and will be proposed separately.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
test(api): trim v1.1 Postman folder to an HTTP smoke test (#475)
5c6ae4c 
Cut 56 requests to 25; edge cases (cap boundary, PATCH copy-flag chain,
duplicate read-backs) are covered by the database and server tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
test: remove overlapping linking tests across database and server (#475)
322aecd 
Drop duplicated cycle, fast-path, privilege and copy-flag cases that are
already proven at a lower layer; ~630 lines fewer. Database ITs 145/0,
server testIT 38/0, server unit 288/0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
ci: fix test filename check and endpoint extractor input for linking …
7fffca2 
…tests (#475)

Exclude the AtumV2StubServer test helper from the filename check and list
the v1.1 Postman folder endpoints in the plain 'METHOD:/path' form the
endpoint extractor expects.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
feat: refuse linking a partitioning that already has descendants (#475)
39996f2 
Descendants are never propagated into the parent's flows. Linking an
existing partitioning that already has descendants under a new parent
is refused with DB status 33 / HTTP 409 and nothing is written;
creation-time linking and idempotent re-links are unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
docs: align READMEs, Scaladoc and SQL comments with leaf-only linking (…
4c76e3c 
…#475)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
perf(db): index partitioning_to_flow(fk_partitioning) for the linking…
5714a4f 
… lookups (#475)

The primary key cannot serve lookups by fk_partitioning alone. flows._link_to_parents does one per
parent while holding the global advisory lock, so each link scanned the whole table under the lock.
Built CONCURRENTLY, so writes are not blocked.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
docs(db): tidy comments in linking migrations (#475)
c57a764 
Comment-only changes (checksums of unreleased V0.10.0.x migrations).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lsulak
last ones
32226d7
@lsulak
compatibility (agent-server) test fix to be able to run also under Ja…
cffe6fb 
…va 8
@lsulak lsulak self-assigned this Nov 4, 2026
@lsulak
lsulak marked this pull request as ready for review November 4, 2026 14:53
@lsulak lsulak changed the title #475 Support linking a child partitioning to multiple parents #475 Lineage Maturity (support linking a child partitioning to multiple parents) Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@benedeki benedeki Awaiting requested review from benedeki benedeki is a code owner
@Zejnilovic Zejnilovic Awaiting requested review from Zejnilovic Zejnilovic is a code owner
@dk1844 dk1844 Awaiting requested review from dk1844 dk1844 is a code owner
@salamonpavel salamonpavel Awaiting requested review from salamonpavel salamonpavel is a code owner
@ABLL526 ABLL526 Awaiting requested review from ABLL526 ABLL526 is a code owner
@MatloaItumeleng MatloaItumeleng Awaiting requested review from MatloaItumeleng MatloaItumeleng is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@lsulak lsulak

Labels

breaking change Introduces change, that is incompatible with last release

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Atum Agent: Support linking Atum Partitioning of a child with multiple parents

2 participants

@lsulak