You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Checkpoints can now be queried by name and properties at once, with several accepted values per property and server-side pagination that is accurate for the combined filter (#478).
JFYI, the audit-oriented part (process start time window, latest-first on flows, getAllCheckpoints, ADR 02) is in follow-up #489, stacked on this one.
Release Notes
Reader: getCheckpointsPage of FlowReader and PartitioningReader accepts a CheckpointFilter combining checkpoint name and properties, with several accepted values per property (e.g. executionID IN (a, b))
Reader: getCheckpointsOfNamePage and getCheckpointsByPropertiesPage deprecated in favour of getCheckpointsPage; getCheckpointsPage gained a parameter, so code compiled against an older Reader needs recompiling
Server: checkpoint-properties of the partitioning and flow checkpoints endpoints accepts multiple values per property; the single-value format is still supported; a property without any value is rejected with 400; pagination applies to the combined name and properties filter
Database: checkpoint retrieval functions take checkpoint properties as JSONB and accept an optional process start time window; new index on runs.checkpoint_properties (fk_checkpoint, property_name)
… work
Keep this change focused on #478 (combined name + multi-value properties filter with accurate
server-side pagination). The audit-oriented additions move to a follow-up PR:
- process start time window (from / to) and latest-first on the flows endpoint, through all layers
- CheckpointFilter.from / to / latestFirst, getAllCheckpoints and Reader.queryAllPages
- flow checkpoints DB-order preservation in CheckpointItemWithPartitioningFromDB
- runs.checkpoints (fk_partitioning, process_start_time) index
- ADR 02 - consumption pattern for audits
(cherry picked from commit 4ce3cae)
…dex in #478
Ship the final shape of flows.get_flow_checkpoints and runs.get_partitioning_checkpoints
(JSONB multi-value properties filter plus optional i_from_time / i_to_time, both DEFAULT NULL)
in this release, so the follow-up audit PR does not need another full copy of both functions.
Positional callers are unaffected: the server keeps passing the leading arguments only.
Also keep the (fk_checkpoint, property_name) index on runs.checkpoint_properties, now
V0.8.0.3: the properties filter looks checkpoints up by fk_checkpoint, which had no index.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A property with an empty set of values (e.g. `{"executionID": []}`) silently matched no checkpoint.
It is now rejected up front: CheckpointFilter throws an IllegalArgumentException on construction,
and the server answers 400 Bad Request for such a `checkpoint-properties` query parameter.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion
The `@throws IllegalArgumentException` tag could not be resolved by Scaladoc,
failing `sbt doc` under -Xfatal-warnings; the exception is now described in the
`properties` parameter doc. Replaced the partial-function `collect` with `filter`
to avoid untestable generated bytecode lowering server coverage below threshold.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds combined checkpoint name/property filtering, multi-value property matching, and accurate server-side pagination across the reader, server, and database layers.
Changes:
Introduces CheckpointFilter and deprecates separate reader filtering methods.
Adds backward-compatible multi-value property decoding and validation.
Migrates database functions to JSONB filtering with deterministic ordering and supporting indexes.
…er integration tests
The server integration tests for flow and partitioning checkpoints used
non-existent IDs, so the database functions returned "not found" before
evaluating the name and property filters or the pagination. They now seed a
partitioning (and its flow) with matching and non-matching checkpoints and
assert both pages and the has-more flag.
A new single-connection, single-transaction test transactor rolls back once the
test ends, so the seeded data is visible to all calls within a test but never
persisted.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The replaced function previously defaulted i_checkpoints_limit to 5, but this changes an omitted limit into an unbounded query and makes has_more always false. Existing callers that invoke the function with only a flow ID will silently receive all checkpoints instead of the first five; preserve the previous default unless this separate API change is explicitly intended.
The replaced function previously defaulted i_checkpoints_limit to 5, but this changes an omitted limit into an unbounded query and makes has_more always false. Existing callers that invoke the function with only a partitioning ID will silently receive all checkpoints instead of the first five; preserve the previous default unless this separate API change is explicitly intended.
…nctions
The new overloads of flows.get_flow_checkpoints and
runs.get_partitioning_checkpoints defaulted i_checkpoints_limit to NULL
(all checkpoints) instead of the established 5, changing the behavior for
direct callers omitting the argument. An explicit NULL still returns all.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…sumption pattern) (#489)
* feat: audit consumption pattern (ADR 02) - process start time window and latest-first on flows
Builds on the #478 reader capability work and exposes what ADR 02 section 4 needs for audits:
- server: optional `from` / `to` (half-open window on process_start_time, from < to validated) on
GET /partitionings/{id}/checkpoints and GET /flows/{id}/checkpoints, and `latest-first` on the
flows endpoint; wired through controllers, services, repositories and DB function wrappers
- server: flow checkpoints keep the DB order instead of re-sorting by start time, so latest-first=false works
- reader: CheckpointFilter.from / to / latestFirst, getAllCheckpoints on FlowReader and PartitioningReader
(Reader.queryAllPages), new query parameter names
- db: V0.8.0.4 composite index runs.checkpoints (fk_partitioning, process_start_time) for the time window
- ADR 02 - consumption pattern for audits
The DB function signatures (i_from_time / i_to_time) already landed with #478, so no function is re-created here.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: stack-safe paging of all checkpoints and ADR 02 response shape
queryAllPages recursed through flatMap, which for strict effects (e.g.
Identity) runs the continuation right away, adding stack frames per page.
The continuation now hands the collected records over to a tail-recursive
loop when it runs synchronously, and recurses within the effect otherwise
(asynchronous or lazy effects), the hand-over being atomic.
ADR 02: a flow checkpoints page item is one checkpoint with its
measurements nested (processStartTime, measurements, partitioning), and
parent -> child edges cannot be reconstructed today (Bet 3).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fixing unit tests
* reader will not need this method most likely, removing
* finalization
* Delete adrs/02-consumption-pattern-for-audits.md
will be in another PR
* fix(db): prevent null flow pagination metadata
Move LIMIT/OFFSET inside the counted subquery so nonzero offsets
cannot discard the aggregate row and return NULL for has_more.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update README with Reader feature requirements
Clarify requirements for Reader features and server compatibility.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lsulak
changed the title
salamonpavel
left a comment
Overview
Checkpoints can now be queried by name and properties at once, with several accepted values per property and server-side pagination that is accurate for the combined filter (#478).
JFYI, the audit-oriented part (process start time window,
latest-firston flows,getAllCheckpoints, ADR 02) is in follow-up #489, stacked on this one.Release Notes
getCheckpointsPageofFlowReaderandPartitioningReaderaccepts aCheckpointFiltercombining checkpoint name and properties, with several accepted values per property (e.g.executionID IN (a, b))getCheckpointsOfNamePageandgetCheckpointsByPropertiesPagedeprecated in favour ofgetCheckpointsPage;getCheckpointsPagegained a parameter, so code compiled against an older Reader needs recompilingcheckpoint-propertiesof the partitioning and flow checkpoints endpoints accepts multiple values per property; the single-value format is still supported; a property without any value is rejected with 400; pagination applies to the combined name and properties filterruns.checkpoint_properties (fk_checkpoint, property_name)Related
Closes #478