Skip to content

feat(runtime): accept typed UnivMon frequency identities - #557

Closed
zzylol wants to merge 1 commit into
stack/528-14-raw-latencyfrom
stack/528-15-frequency-inputs
Closed

zzylol wants to merge 1 commit into
stack/528-14-raw-latencyfrom
stack/528-15-frequency-inputs

Conversation

@zzylol

@zzylol zzylol commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem: the native UnivMon build cannot consume the src_ip identities that #509 Example 2 shares

#509 §Pass 2 (summary-capability rule) and §Example 2 ("One summary for several computations") describe this shared candidate:

One UnivMon over src_ip from flows in the last minute serves three queries refreshed every 10 s: COUNT(DISTINCT src_ip), the entropy of the src_ip distribution, and the L2 norm of per-src_ip counts. Each flow record updates the UnivMon once; a distinct-count, an entropy and an L2 estimation node each compute their statistic from it.

#509 §Physical operator implementation then turns that node into summary build and estimation operators. The build operator must therefore accept a src_ip value as a UnivMon update.

Before this PR, the native summary build accepts only Float64 update columns, for every family:

// crates/asap-physical-operators/src/operators/summary/mod.rs (before)
if plain(&input, value)?.0 != &DataType::Float64 {
    return Err(invalid("summary numeric update requires Float64"));
}

So the Example 2 input fails before any row is read:

flows(src_ip: Utf8)  →  Operator::summary_build(input, UnivMon{..}, value = 0, None, [])
                     →  Err("summary numeric update requires Float64")

Converting the identity to Float64 first is not a fix. Int64 identities above 2^53 collapse: 9_007_199_254_740_992 and 9_007_199_254_740_993 become the same Float64, so the distinct count, L2 and entropy all change.

Scope. This PR covers the physical build side of the Example 2 shared candidate: one native UnivMon state built from typed src_ip-like identities, read by the distinct, L2 and entropy estimators. It does not cover the frontend recognition of the Q2/Q3 SQL forms (#509 §Example 2 TODO), the Pass 2 sharing rule itself, sizing for the strictest ε, or UnivMon accuracy certification.

Proposed method

All changes are in the physical runtime (asap-physical-operators). Planning stages are unchanged.

  1. Build-time type check. Operator::summary_build still requires Float64 for every family, except UnivMon. A UnivMon build also accepts Utf8, Int64 and Bool update columns. Any other type returns "summary update type is unsupported by its family".
  2. Typed update path. The build loop no longer unwraps the row value to f64. It skips Value::Null (as before: SQL aggregates ignore NULL samples while keeping the group) and passes the typed &Value to a new trait method AccumulatorUpdater::update_value.
  3. Default keeps the numeric contract. The default update_value requires Value::Float64, runs validate_single_input, then update_single. Every non-UnivMon family therefore behaves exactly as before.
  4. UnivMon override. UnivMonUpdater::update_value calls the new UnivMonAccumulator::insert_value:
    • Null → no-op.
    • finite Float64 → the existing insert_sample path (unchanged Float64 behavior).
    • Bool / Int64 / Utf8 → encode with the existing type-tagged Value::key() bytes, check the bucket counter for overflow, and insert as DataInput::Bytes with weight 1.
    • anything else (including non-finite Float64) → error.
      Because Value::key() stores Int64 as its 8 little-endian bytes with a type tag, neighboring large integers stay distinct.
  5. Memory accounting. UnivMonAccumulator::approx_memory_bytes now adds the capacity of String/Bytes keys held in each layer's heavy-hitter heap. Long string identities count against the runtime memory reservation. After clear(), the value returns to the empty-state size.

One built state still feeds the existing SketchStatistic::Cardinality, FrequencyL2 and FrequencyEntropy evaluations.

Key code interfaces

crates/asap-physical-operators/src/summary_kernels/factory.rs

pub trait AccumulatorUpdater: Send {
    /// Update from the typed native row. Numeric kernels retain their existing
    /// Float64 contract; frequency kernels may accept nonnumeric identities.
    fn update_value(
        &mut self,
        value: &crate::values::Value,
        timestamp_ms: i64,
    ) -> Result<(), String>;      // default: Float64 only → validate_single_input + update_single
    // existing methods unchanged
}

impl AccumulatorUpdater for UnivMonUpdater {
    fn update_value(&mut self, value: &crate::values::Value, _: i64) -> Result<(), String>;
}

crates/asap-physical-operators/src/summary_kernels/univmon.rs

impl UnivMonAccumulator {
    /// SQL identities are kept in their original type: converting Int64 to
    /// Float64 would collapse neighboring keys above 2^53.
    pub fn insert_value(&mut self, value: &crate::values::Value) -> Result<(), Error>;
}

crates/asap-physical-operators/src/operators/summary/mod.rs (signature unchanged, accepted types widened)

pub fn summary_build(
    input: SchemaRef,
    family: SummaryFamilyType,
    value: usize,
    time: Option<usize>,
    groups: Vec<usize>,
) -> Result<Self, Error>;

Usage, as in tests/univmon_execution.rs:

let input = Arc::new(Schema::new(vec![Field::plain("src_ip", DataType::Utf8, true)]));
let build = Operator::summary_build(input, family(), 0, None, vec![])?; // family() = UnivMon
// node 1 = build; nodes 2,3,4 = Operator::evaluation(output, 0, SummaryEvaluation::Sketch(stat))
// for stat in [Cardinality, FrequencyL2, FrequencyEntropy]

Fields

AccumulatorUpdater::update_value

Parameter Type Meaning
self &mut self The per-group summary updater being built.
value &Value The typed update value from the input row. The build loop never passes Null; it skips NULL rows before the call.
timestamp_ms i64 Row timestamp from the time column, or 0 when time is None. Passed to update_single by the default; ignored by UnivMon.
return Result<(), String> Err for a type the family does not accept or for an input validate_single_input rejects. The build maps it to Error::Operator.

UnivMonAccumulator::insert_value

Input value Effect
Value::Null Ignored, Ok(()).
Value::Float64(x), x finite insert_sample(x), same as before this PR.
Value::Bool / Value::Int64 / Value::Utf8 Inserted once as type-tagged key bytes (Value::key()), weight 1. Errors with "UnivMon count overflow" if bucket_size + 1 overflows.
other, or non-finite Float64 Err("unsupported UnivMon identity type or nonfinite sample").

Operator::summary_build

Parameter Type Meaning / allowed values
input SchemaRef Input row schema.
family SummaryFamilyType Summary family and parameters. Validated by validate_family. UnivMon is detected with kind.algorithm() == SketchAlgorithm::UnivMon.
value usize Index of the update column. Must be plain. Float64 for all families; for UnivMon also Utf8, Int64 or Bool.
time Option<usize> Optional timestamp column; must be plain, non-null Timestamp (unchanged).
groups Vec<usize> Grouping columns; validated by validate_groups (unchanged).

Examples

End to end (tests/univmon_execution.rs::typed_frequency_keys_preserve_identity). For each key type, the input column src_ip holds two keys, each twice, and one NULL:

src_ip: k0, k0, k1, k1, NULL
PhysicalDAG: 0 source → 1 summary_build(UnivMon) → {2 Cardinality, 3 FrequencyL2, 4 FrequencyEntropy}
Key type k0, k1 Distinct L2 Entropy (bits)
Utf8 "192.0.2.1", "192.0.2.2" 2 √8 1
Int64 9_007_199_254_740_992, 9_007_199_254_740_993 2 √8 1
Bool false, true 2 √8 1

All values are checked within 0.01. The NULL row contributes nothing. The two Int64 keys are 2^53 and 2^53+1, which would be one key after a Float64 cast.

Kernel tests (src/summary_kernels/univmon.rs).

  • typed_keys_merge_and_roundtrip: two Utf8 states (192.0.2.1 ×2 and 192.0.2.2 ×2) are merged, serialized and restored. The restored state still gives distinct 2, L2 √8, entropy 1.
  • memory_accounts_for_string_identities: inserting one 4,096-byte Utf8 key raises approx_memory_bytes by at least 4,096. After clear() it equals the empty size.

Accepted vs rejected update columns

Family Column type Result
UnivMon Float64 accepted (unchanged path)
UnivMon Utf8, Int64, Bool accepted (new, typed keys)
UnivMon other types rejected at construction
UnivMon non-finite Float64 value rejected at update
any other family Float64 accepted (unchanged)
any other family non-Float64 rejected at construction, and update_value default rejects non-Float64 values

The developer note in docs/develop_docs/operator-design-acceptance.md §"Planner-layering follow-up: typed frequency inputs" records the same contract.

Out of scope

Stack and validation

Stacked on #556 · Next: #559 · Reference/tracker: #528

Validation: the typed-input regression fails before the change and passes afterward; all 225 native runtime tests/doctests passed; formatting; all-target Clippy for native runtime with warnings denied.

The full restacked tip passes all 1,608 workspace tests/doctests (two existing ignores), formatting and workspace/all-target/all-feature Clippy with warnings denied. The inherited UnivMon precompute fixture was fixed in #552, and the latency fixture lint in #554; later branches were restacked atomically with explicit leases. CI has been retriggered.

@zzylol
zzylol force-pushed the stack/528-15-frequency-inputs branch from fb5b29d to e0e1e2d Compare October 3, 2026 04:08
@zzylol
zzylol force-pushed the stack/528-14-raw-latency branch from 902d3b5 to a331146 Compare October 3, 2026 04:08
@zzylol
zzylol marked this pull request as draft October 3, 2026 19:28
@zzylol

zzylol commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Parked as draft: PR priorities changed (see #528). Order is now (A) finish #511 operator sharing, (B) the #572 crate/module reorganization, (C) #509 end-to-end stages. This PR sits on the old stack/528-legacy-physical-base chain, and Phase B moves the files it touches. Its content will be re-scoped onto the new layout in Phase C.

@zzylol

zzylol commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

Ported onto the current stack in #597

zzylol added a commit that referenced this pull request Oct 4, 2026
…ntities

Port of the parked runtime PRs for #509 Example 2 onto asap-executor:

- native UnivMon build and Cardinality / FrequencyL2 / FrequencyEntropy
  readouts (the old stack's 14c8ac8, a prerequisite missing here);
- #557: a UnivMon build accepts Utf8, Int64 and Bool identities as
  type-tagged keys; heap key bytes count toward state memory;
- #559: exact FrequencyL2 / FrequencyEntropy reducers (bits, NULL skipped,
  0 for an empty population, memory-accounted, cooperative);
- #563: exact Cardinality reducer over typed tuples.

The legacy raw-cost adapter lists the three intents as hash aggregates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol

zzylol commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Closing: this parked chain (on stack/528-09-accuracy) has been superseded. Its content was ported or reimplemented further up the current stack, in #597, on the unified IR. Closed with the author's approval during the 2026-10-13 stack cleanup.

@zzylol zzylol closed this Oct 13, 2026
zzylol added a commit that referenced this pull request Oct 15, 2026
UnivMon had a kernel but no native DAG state: a build was rejected
("summary family has no native DAG state implementation"), a summary
build accepted only Float64 samples, and Pass 1's UnivMon update (each
value of the item column one unit occurrence) did not bind, so no
UnivMon candidate of #509 Example 2 could execute.

- UnivMon is a native family; its states validate against their
  parameters, and one state answers its total, distinct count, L2 norm
  and entropy.
- A UnivMon build takes Bool, Int64 and Utf8 identities as type-tagged
  key bytes (Int64 above 2^53 stays distinct) besides Float64 samples;
  NULL is skipped. Retained heap key bytes count toward state memory.
- A unit-frequency update over one item column binds, for UnivMon and
  HLL, as a value-frequency build over that column: grouped (SQL
  src_ip) and per series (PromQL sample values).

Ports the UnivMon part of #597 (from #557).
zzylol added a commit that referenced this pull request Oct 15, 2026
UnivMon had a kernel but no native DAG state: a build was rejected
("summary family has no native DAG state implementation"), a summary
build accepted only Float64 samples, and Pass 1's UnivMon update (each
value of the item column one unit occurrence) did not bind, so no
UnivMon candidate of #509 Example 2 could execute.

- UnivMon is a native family; its states validate against their
  parameters, and one state answers its total, distinct count, L2 norm
  and entropy.
- A UnivMon build takes Bool, Int64 and Utf8 identities as type-tagged
  key bytes (Int64 above 2^53 stays distinct) besides Float64 samples;
  NULL is skipped. Retained heap key bytes count toward state memory.
- A unit-frequency update over one item column binds, for UnivMon and
  HLL, as a value-frequency build over that column: grouped (SQL
  src_ip) and per series (PromQL sample values).

Ports the UnivMon part of #597 (from #557).
zzylol added a commit that referenced this pull request Oct 15, 2026
…and UnivMon frequency statistics (#707)

feat(types): summary-capability facts in the family-capability table

Each family lists the readouts one of its states answers (Readout:
total, item frequency, any quantile rank, distinct count, L2, entropy,
top-k), so a planner rule can check that one state serves every
consumer's statistic: UnivMon answers its total and every frequency
moment, KLL every rank, CountSketch item frequencies and L2, the heap
sketches top-k and item frequencies. Exact accumulators are read as
finalized values and answer no readout.
feat(logical-asap-optimization): Pass 2 summary-capability rule

Composition sizes each consumer's summary build alone, for its own
target. The summary-capability rule groups the builds of one candidate
that chose the same family over the same input sub-DAG (window, WHERE
and time range included), item and weight, grouping and filter, whose
every reader is an estimate the family answers (the family-capability
table's readouts), and that differ only in size. For every choice, per
group, of no sharing or one subset of two or more members whose sizes
differ (partial sharing included), the subset becomes one build sized
with each parameter's largest value among the members (a KLL's failure
probability and DDSketch's alpha: the smallest), and every guarantee
above it is composed again, so plan selection checks each consumer
against the shared state. Builds of equal size stay the
identical-expression rule's; unshared builds are untouched and no
consumer changes family.

Stage 1 lists each composed candidate, its identical-expression
candidates, then each summary-capability candidate followed by its own
identical-expression candidates, every distinct DAG once. The Pass 2
record (Pass2Candidate) names the shared summaries ("UnivMon for Q1,
Q3") and the shared identical groups.

Example 2 (PromQL stand-in): with #702's sizing the three UnivMons
differ (Q1 a distinct-count heap on 19 layers, Q2 the baseline shape for
entropy, Q3 a one-layer Count Sketch for L2), so the rule offers one
UnivMon for each pair and one for all three, each sized with the
members' largest heap, rows, columns and layers and the smallest
recorded distinct-count failure probability; the third query keeps its
own options. A shared UnivMon whose rows and columns together exceed
asap_sketchlib's hash budget (rows * (ceil(log2 cols) + 1) <= 128) is
not offered, since the library would not build it. The shared UnivMon
stays one build in Stage 2.
feat(frontend-sql): recognize the SQL frequency L2 and entropy idioms

#509 Example 2 writes L2(src_ip) and Entropy(src_ip) as SQL idioms over
grouped unit counts. The SQL lowering (single query and batch) now names
them at the query root as FrequencyL2 / FrequencyEntropy aggregates
carrying the query's accuracy target, so Pass 1 offers exact and summary
candidates for each:

- L2: SQRT(SUM(c * c)) over COUNT(*) per key, only when the product is
  already Float64 (CAST(c AS DOUBLE) factors or CAST(COUNT(*) AS DOUBLE)).
  The Int64 product stays relational: SQL integer overflow is observable.
- Entropy: -SUM(p * LN(p)) with p = COUNT(*)[*1.0] / SUM(COUNT(*)) OVER ()
  over the complete unordered frame; the IR's bits are converted to nats
  and SQL's -0.0 is kept.

Both require a single non-nullable Bool/Int64/Utf8 key, no HAVING and no
measure filter, follow projections only, keep the original output schema,
and decide SQL's empty-input NULL by an exact population count joined
beside the statistic, never by an estimate. Scaled probabilities, LOG2,
partitioned or ordered windows and nullable keys stay relational.

Ports the frontend part of #598 (originally #562 and #564).
feat(physical-operators): exact distinct, L2 and entropy reducers

Exact Cardinality, FrequencyL2 and FrequencyEntropy aggregates had no
native implementation, so the exact option of #509 Example 2's three
statistics could not execute.

- Reduction::Cardinality(cols) counts distinct tuples; a tuple with any
  NULL is skipped, and grouped key encoding keeps tuple boundaries and
  folds signed zero and NaN payloads.
- Reduction::FrequencyL2(col) and FrequencyEntropy(col) count the unit
  frequencies of one Bool/Int64/Float64/Utf8 identity column: NULL is
  skipped, a non-finite float is an error, an empty population gives 0.0,
  and entropy is in bits, as AggIntent::FrequencyEntropy and UnivMon's
  readout define it.
- The dictionaries are memory-accounted and cooperative.
- Logical aggregates bind them: grouped ones (SQL COUNT(DISTINCT)) and
  per-series range reductions (distinct_over_time, l2_over_time,
  entropy_over_time), which reduce each series' window samples, restore
  the run's timestamp and drop the metric name as range functions do.
- The analytical cost lowering prices them as hash aggregates.

Ports the exact-reducer part of #597 (from #559 and #563).
feat(physical-operators): native UnivMon state over typed identities

UnivMon had a kernel but no native DAG state: a build was rejected
("summary family has no native DAG state implementation"), a summary
build accepted only Float64 samples, and Pass 1's UnivMon update (each
value of the item column one unit occurrence) did not bind, so no
UnivMon candidate of #509 Example 2 could execute.

- UnivMon is a native family; its states validate against their
  parameters, and one state answers its total, distinct count, L2 norm
  and entropy.
- A UnivMon build takes Bool, Int64 and Utf8 identities as type-tagged
  key bytes (Int64 above 2^53 stays distinct) besides Float64 samples;
  NULL is skipped. Retained heap key bytes count toward state memory.
- A unit-frequency update over one item column binds, for UnivMon and
  HLL, as a value-frequency build over that column: grouped (SQL
  src_ip) and per series (PromQL sample values).

Ports the UnivMon part of #597 (from #557).
test(integration-tests): Example 2 in SQL; the recognized idioms execute

Example 2's SQL Q3 casts its counts to DOUBLE (the design's Int64
SUM(c * c) can overflow and stays relational), and the SQL Q2 and Q3
now lower to the entropy and L2 intents, so the SQL workload has the
PromQL stand-in's Pass 1 options. The recognized entropy executes in
nats with SQL's -0.0, L2 executes exactly, and both give NULL on empty
input, decided by the exact population count.
feat(devtools): stage_pipeline example planner-layering-2

stage_pipeline plans #509 Example 2: three SQL dashboard panels over
the last minute of flows (distinct sources at eps 0.02, entropy at
0.05, L2 at 0.01, delta 0.01), lowered over its flows(ts, src_ip)
table. SQL workloads lower every entry, batch and repeating, each with
its own accuracy target (lower_sql_batch read only batch entries).

The stage-viewer tab 2 row is regenerated from it: 82 plans, 16 from
Pass 1, the rest from the identical-expression and summary-capability
rules (one UnivMon for each pair of queries and for all three); 6
valid, P1 (all exact) selected at 6.08 per second. The row previously
said 8 plans were valid.
feat(logical-asap-optimization): guarantees through a join beside exact inputs and an exactly decided CASE

SQL Example 2's L2 decides the empty-input NULL from an exact population
count cross-joined beside the estimate and a CASE on that count. No
guarantee rule covered the join or the CASE, so every SQL plan with
UnivMon for Q3 was rejected ("the output has no guarantee"), although
the L2 estimate is certified.

- A join (inner, outer or cross) whose inputs are exact but one keeps
  that input's guarantee when its predicate reads only exact columns:
  which rows match, and how often, then does not depend on an estimate.
  A key on the approximate input's columns, a semi- or anti-join, or two
  approximate inputs have none.
- A CASE whose operand and conditions are exact keeps its weakest
  branch's guarantee; a literal or NULL branch, or a missing ELSE, is
  exact. A condition over an approximate value has none.
- A projection over such a join composes each output column this way.

Guarantees are per node, so every column of the approximate input counts
as approximate; column-level tracking (#689) stays the general solution.

A root's default metric (#690) looks through the same shape: a
projection whose CASE value branches read columns (or literals, cast or
not) takes those columns' metric, following them through joins and
row-preserving operators to an aggregate's measure, so SQL's L2 root
resolves to a relative target as the PromQL one does. Any other
expression stays absolute.
docs(stage-viewer): regenerate the Example 2 row with SQL L2 through UnivMon

With the join and CASE guarantee rules, the SQL Q3 UnivMon plans are
valid: 12 of the 82 plans are valid (Q3 exact or UnivMon), 70 invalid
(Q1 HLL or UnivMon without a distinct bound, Q2 UnivMon without an
entropy bound). P1 (all exact) is still selected at 6.08 per second;
UnivMon on Q3 costs 8.08.
fix(physical-operators): charge UnivMon key memory without walking its heaps

A build charges a state's memory after every input row, and UnivMon's
charge summed the bytes of every key on every layer's heap, so building
was quadratic in the heap: a 20,000-key heap on 19 layers over 40,000
keys took 3.3 s in release. The charge now bounds the retained keys by
every heap slot holding the longest key seen (tracked on insert, merge
and adoption of a decoded sketch), in constant time. A test checks that
the charge does not grow with heap occupancy; a timing test (ignored,
release) checks the large build.
fix(logical-asap-optimization): summary-capability review fixes

- A capability group of more than 63 builds is an error: one 64-bit mask
  enumerates the subsets of at most 63, and the shift was clamped, so a
  larger group was enumerated wrongly (and took 2^63 steps) instead of
  refused.
- A summary-capability variant whose guarantee composition fails carries
  its own error, so Stage 1 lists it as a candidate that cannot be
  composed and keeps the remaining variants; one failure used to drop
  them all.
- shared_size_params no longer claims every sizing bound is monotone (a
  Count Sketch's collision bound is tighter at a power-of-two width than
  one column wider); the shared state is checked by composing every
  guarantee above it again.
fix(logical-asap-optimization): read each column's guarantee in a projection over a join

A projection's guarantee assumed the error lies only in its child's
sample-value column (the column named "value", or the only numeric
one). Over a join that column can be an exact input's, so the
approximate input's columns were read as exact: projecting
sum(m) * 1000 beside the exact side's value composed to the
approximate side's absolute error 1, while the truth is 1000.

A projection over a join (directly, or through projections, filters,
sorts, limits or deduplication) now reads each column's own guarantee
(column_guarantees), and a projection over a computed projection over a
join, whose columns' guarantees are unknown, has none. Plan selection's
operand-domain lookup reads columns the same way. column_guarantees and
join_guarantee document their assumption that every approximate node of
Stage 1 has an exact row set. Per-column tracking in general is #689.
fix(logical-asap-optimization): refuse summary-capability groups of more than 20 builds

Every subset of two or more builds of a group is a candidate, 2^n of
them; past 20 builds (about a million) the rule returns an error naming
the limit instead of enumerating.
fix(physical-operators): bound UnivMon key memory by the key bytes inserted

Charging every heap slot at the longest key seen overcounts badly: one
1 KB key in a sketch of 19 layers of 20,000 slots was charged 380 MB.
Each layer's heap keeps at most one copy of a key, so the retained keys
are also bounded by the layers times the bytes of every key inserted
(or merged in); the charge takes the smaller bound, still in constant
time. asap_sketchlib exposes no eviction hook for an exact running
total.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant