Repository navigation
Conversation
zzylol
force-pushed
the
stack/528-15-frequency-inputs
branch
from
October 3, 2026 04:08
fb5b29d to
e0e1e2d
Compare
zzylol
force-pushed
the
stack/528-14-raw-latency
branch
from
October 3, 2026 04:08
902d3b5 to
a331146
Compare
This was referenced Oct 3, 2026
zzylol
marked this pull request as draft
October 3, 2026 19:28
Contributor
Author
|
Parked as draft: PR priorities changed (see #528). Order is now (A) finish #511 operator sharing, (B) the #572 crate/module reorganization, (C) #509 end-to-end stages. This PR sits on the old |
Contributor
Author
|
Ported onto the current stack in #597 |
zzylol
added a commit
that referenced
this pull request
Oct 4, 2026
…ntities Port of the parked runtime PRs for #509 Example 2 onto asap-executor: - native UnivMon build and Cardinality / FrequencyL2 / FrequencyEntropy readouts (the old stack's 14c8ac8, a prerequisite missing here); - #557: a UnivMon build accepts Utf8, Int64 and Bool identities as type-tagged keys; heap key bytes count toward state memory; - #559: exact FrequencyL2 / FrequencyEntropy reducers (bits, NULL skipped, 0 for an empty population, memory-accounted, cooperative); - #563: exact Cardinality reducer over typed tuples. The legacy raw-cost adapter lists the three intents as hash aggregates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Closing: this parked chain (on |
zzylol
added a commit
that referenced
this pull request
Oct 15, 2026
UnivMon had a kernel but no native DAG state: a build was rejected
("summary family has no native DAG state implementation"), a summary
build accepted only Float64 samples, and Pass 1's UnivMon update (each
value of the item column one unit occurrence) did not bind, so no
UnivMon candidate of #509 Example 2 could execute.
- UnivMon is a native family; its states validate against their
parameters, and one state answers its total, distinct count, L2 norm
and entropy.
- A UnivMon build takes Bool, Int64 and Utf8 identities as type-tagged
key bytes (Int64 above 2^53 stays distinct) besides Float64 samples;
NULL is skipped. Retained heap key bytes count toward state memory.
- A unit-frequency update over one item column binds, for UnivMon and
HLL, as a value-frequency build over that column: grouped (SQL
src_ip) and per series (PromQL sample values).
Ports the UnivMon part of #597 (from #557).
zzylol
added a commit
that referenced
this pull request
Oct 15, 2026
UnivMon had a kernel but no native DAG state: a build was rejected
("summary family has no native DAG state implementation"), a summary
build accepted only Float64 samples, and Pass 1's UnivMon update (each
value of the item column one unit occurrence) did not bind, so no
UnivMon candidate of #509 Example 2 could execute.
- UnivMon is a native family; its states validate against their
parameters, and one state answers its total, distinct count, L2 norm
and entropy.
- A UnivMon build takes Bool, Int64 and Utf8 identities as type-tagged
key bytes (Int64 above 2^53 stays distinct) besides Float64 samples;
NULL is skipped. Retained heap key bytes count toward state memory.
- A unit-frequency update over one item column binds, for UnivMon and
HLL, as a value-frequency build over that column: grouped (SQL
src_ip) and per series (PromQL sample values).
Ports the UnivMon part of #597 (from #557).
zzylol
added a commit
that referenced
this pull request
Oct 15, 2026
…and UnivMon frequency statistics (#707) feat(types): summary-capability facts in the family-capability table Each family lists the readouts one of its states answers (Readout: total, item frequency, any quantile rank, distinct count, L2, entropy, top-k), so a planner rule can check that one state serves every consumer's statistic: UnivMon answers its total and every frequency moment, KLL every rank, CountSketch item frequencies and L2, the heap sketches top-k and item frequencies. Exact accumulators are read as finalized values and answer no readout. feat(logical-asap-optimization): Pass 2 summary-capability rule Composition sizes each consumer's summary build alone, for its own target. The summary-capability rule groups the builds of one candidate that chose the same family over the same input sub-DAG (window, WHERE and time range included), item and weight, grouping and filter, whose every reader is an estimate the family answers (the family-capability table's readouts), and that differ only in size. For every choice, per group, of no sharing or one subset of two or more members whose sizes differ (partial sharing included), the subset becomes one build sized with each parameter's largest value among the members (a KLL's failure probability and DDSketch's alpha: the smallest), and every guarantee above it is composed again, so plan selection checks each consumer against the shared state. Builds of equal size stay the identical-expression rule's; unshared builds are untouched and no consumer changes family. Stage 1 lists each composed candidate, its identical-expression candidates, then each summary-capability candidate followed by its own identical-expression candidates, every distinct DAG once. The Pass 2 record (Pass2Candidate) names the shared summaries ("UnivMon for Q1, Q3") and the shared identical groups. Example 2 (PromQL stand-in): with #702's sizing the three UnivMons differ (Q1 a distinct-count heap on 19 layers, Q2 the baseline shape for entropy, Q3 a one-layer Count Sketch for L2), so the rule offers one UnivMon for each pair and one for all three, each sized with the members' largest heap, rows, columns and layers and the smallest recorded distinct-count failure probability; the third query keeps its own options. A shared UnivMon whose rows and columns together exceed asap_sketchlib's hash budget (rows * (ceil(log2 cols) + 1) <= 128) is not offered, since the library would not build it. The shared UnivMon stays one build in Stage 2. feat(frontend-sql): recognize the SQL frequency L2 and entropy idioms #509 Example 2 writes L2(src_ip) and Entropy(src_ip) as SQL idioms over grouped unit counts. The SQL lowering (single query and batch) now names them at the query root as FrequencyL2 / FrequencyEntropy aggregates carrying the query's accuracy target, so Pass 1 offers exact and summary candidates for each: - L2: SQRT(SUM(c * c)) over COUNT(*) per key, only when the product is already Float64 (CAST(c AS DOUBLE) factors or CAST(COUNT(*) AS DOUBLE)). The Int64 product stays relational: SQL integer overflow is observable. - Entropy: -SUM(p * LN(p)) with p = COUNT(*)[*1.0] / SUM(COUNT(*)) OVER () over the complete unordered frame; the IR's bits are converted to nats and SQL's -0.0 is kept. Both require a single non-nullable Bool/Int64/Utf8 key, no HAVING and no measure filter, follow projections only, keep the original output schema, and decide SQL's empty-input NULL by an exact population count joined beside the statistic, never by an estimate. Scaled probabilities, LOG2, partitioned or ordered windows and nullable keys stay relational. Ports the frontend part of #598 (originally #562 and #564). feat(physical-operators): exact distinct, L2 and entropy reducers Exact Cardinality, FrequencyL2 and FrequencyEntropy aggregates had no native implementation, so the exact option of #509 Example 2's three statistics could not execute. - Reduction::Cardinality(cols) counts distinct tuples; a tuple with any NULL is skipped, and grouped key encoding keeps tuple boundaries and folds signed zero and NaN payloads. - Reduction::FrequencyL2(col) and FrequencyEntropy(col) count the unit frequencies of one Bool/Int64/Float64/Utf8 identity column: NULL is skipped, a non-finite float is an error, an empty population gives 0.0, and entropy is in bits, as AggIntent::FrequencyEntropy and UnivMon's readout define it. - The dictionaries are memory-accounted and cooperative. - Logical aggregates bind them: grouped ones (SQL COUNT(DISTINCT)) and per-series range reductions (distinct_over_time, l2_over_time, entropy_over_time), which reduce each series' window samples, restore the run's timestamp and drop the metric name as range functions do. - The analytical cost lowering prices them as hash aggregates. Ports the exact-reducer part of #597 (from #559 and #563). feat(physical-operators): native UnivMon state over typed identities UnivMon had a kernel but no native DAG state: a build was rejected ("summary family has no native DAG state implementation"), a summary build accepted only Float64 samples, and Pass 1's UnivMon update (each value of the item column one unit occurrence) did not bind, so no UnivMon candidate of #509 Example 2 could execute. - UnivMon is a native family; its states validate against their parameters, and one state answers its total, distinct count, L2 norm and entropy. - A UnivMon build takes Bool, Int64 and Utf8 identities as type-tagged key bytes (Int64 above 2^53 stays distinct) besides Float64 samples; NULL is skipped. Retained heap key bytes count toward state memory. - A unit-frequency update over one item column binds, for UnivMon and HLL, as a value-frequency build over that column: grouped (SQL src_ip) and per series (PromQL sample values). Ports the UnivMon part of #597 (from #557). test(integration-tests): Example 2 in SQL; the recognized idioms execute Example 2's SQL Q3 casts its counts to DOUBLE (the design's Int64 SUM(c * c) can overflow and stays relational), and the SQL Q2 and Q3 now lower to the entropy and L2 intents, so the SQL workload has the PromQL stand-in's Pass 1 options. The recognized entropy executes in nats with SQL's -0.0, L2 executes exactly, and both give NULL on empty input, decided by the exact population count. feat(devtools): stage_pipeline example planner-layering-2 stage_pipeline plans #509 Example 2: three SQL dashboard panels over the last minute of flows (distinct sources at eps 0.02, entropy at 0.05, L2 at 0.01, delta 0.01), lowered over its flows(ts, src_ip) table. SQL workloads lower every entry, batch and repeating, each with its own accuracy target (lower_sql_batch read only batch entries). The stage-viewer tab 2 row is regenerated from it: 82 plans, 16 from Pass 1, the rest from the identical-expression and summary-capability rules (one UnivMon for each pair of queries and for all three); 6 valid, P1 (all exact) selected at 6.08 per second. The row previously said 8 plans were valid. feat(logical-asap-optimization): guarantees through a join beside exact inputs and an exactly decided CASE SQL Example 2's L2 decides the empty-input NULL from an exact population count cross-joined beside the estimate and a CASE on that count. No guarantee rule covered the join or the CASE, so every SQL plan with UnivMon for Q3 was rejected ("the output has no guarantee"), although the L2 estimate is certified. - A join (inner, outer or cross) whose inputs are exact but one keeps that input's guarantee when its predicate reads only exact columns: which rows match, and how often, then does not depend on an estimate. A key on the approximate input's columns, a semi- or anti-join, or two approximate inputs have none. - A CASE whose operand and conditions are exact keeps its weakest branch's guarantee; a literal or NULL branch, or a missing ELSE, is exact. A condition over an approximate value has none. - A projection over such a join composes each output column this way. Guarantees are per node, so every column of the approximate input counts as approximate; column-level tracking (#689) stays the general solution. A root's default metric (#690) looks through the same shape: a projection whose CASE value branches read columns (or literals, cast or not) takes those columns' metric, following them through joins and row-preserving operators to an aggregate's measure, so SQL's L2 root resolves to a relative target as the PromQL one does. Any other expression stays absolute. docs(stage-viewer): regenerate the Example 2 row with SQL L2 through UnivMon With the join and CASE guarantee rules, the SQL Q3 UnivMon plans are valid: 12 of the 82 plans are valid (Q3 exact or UnivMon), 70 invalid (Q1 HLL or UnivMon without a distinct bound, Q2 UnivMon without an entropy bound). P1 (all exact) is still selected at 6.08 per second; UnivMon on Q3 costs 8.08. fix(physical-operators): charge UnivMon key memory without walking its heaps A build charges a state's memory after every input row, and UnivMon's charge summed the bytes of every key on every layer's heap, so building was quadratic in the heap: a 20,000-key heap on 19 layers over 40,000 keys took 3.3 s in release. The charge now bounds the retained keys by every heap slot holding the longest key seen (tracked on insert, merge and adoption of a decoded sketch), in constant time. A test checks that the charge does not grow with heap occupancy; a timing test (ignored, release) checks the large build. fix(logical-asap-optimization): summary-capability review fixes - A capability group of more than 63 builds is an error: one 64-bit mask enumerates the subsets of at most 63, and the shift was clamped, so a larger group was enumerated wrongly (and took 2^63 steps) instead of refused. - A summary-capability variant whose guarantee composition fails carries its own error, so Stage 1 lists it as a candidate that cannot be composed and keeps the remaining variants; one failure used to drop them all. - shared_size_params no longer claims every sizing bound is monotone (a Count Sketch's collision bound is tighter at a power-of-two width than one column wider); the shared state is checked by composing every guarantee above it again. fix(logical-asap-optimization): read each column's guarantee in a projection over a join A projection's guarantee assumed the error lies only in its child's sample-value column (the column named "value", or the only numeric one). Over a join that column can be an exact input's, so the approximate input's columns were read as exact: projecting sum(m) * 1000 beside the exact side's value composed to the approximate side's absolute error 1, while the truth is 1000. A projection over a join (directly, or through projections, filters, sorts, limits or deduplication) now reads each column's own guarantee (column_guarantees), and a projection over a computed projection over a join, whose columns' guarantees are unknown, has none. Plan selection's operand-domain lookup reads columns the same way. column_guarantees and join_guarantee document their assumption that every approximate node of Stage 1 has an exact row set. Per-column tracking in general is #689. fix(logical-asap-optimization): refuse summary-capability groups of more than 20 builds Every subset of two or more builds of a group is a candidate, 2^n of them; past 20 builds (about a million) the rule returns an error naming the limit instead of enumerating. fix(physical-operators): bound UnivMon key memory by the key bytes inserted Charging every heap slot at the longest key seen overcounts badly: one 1 KB key in a sketch of 19 layers of 20,000 slots was charged 380 MB. Each layer's heap keeps at most one copy of a key, so the retained keys are also bounded by the layers times the bytes of every key inserted (or merged in); the charge takes the smaller bound, still in constant time. asap_sketchlib exposes no eviction hook for an exact running total.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem: the native UnivMon build cannot consume the
src_ipidentities that #509 Example 2 shares#509 §Pass 2 (summary-capability rule) and §Example 2 ("One summary for several computations") describe this shared candidate:
#509 §Physical operator implementation then turns that node into summary build and estimation operators. The build operator must therefore accept a
src_ipvalue as a UnivMon update.Before this PR, the native summary build accepts only Float64 update columns, for every family:
So the Example 2 input fails before any row is read:
Converting the identity to Float64 first is not a fix. Int64 identities above 2^53 collapse:
9_007_199_254_740_992and9_007_199_254_740_993become the same Float64, so the distinct count, L2 and entropy all change.Scope. This PR covers the physical build side of the Example 2 shared candidate: one native UnivMon state built from typed
src_ip-like identities, read by the distinct, L2 and entropy estimators. It does not cover the frontend recognition of the Q2/Q3 SQL forms (#509 §Example 2 TODO), the Pass 2 sharing rule itself, sizing for the strictest ε, or UnivMon accuracy certification.Proposed method
All changes are in the physical runtime (
asap-physical-operators). Planning stages are unchanged.Operator::summary_buildstill requires Float64 for every family, except UnivMon. A UnivMon build also acceptsUtf8,Int64andBoolupdate columns. Any other type returns"summary update type is unsupported by its family".f64. It skipsValue::Null(as before: SQL aggregates ignore NULL samples while keeping the group) and passes the typed&Valueto a new trait methodAccumulatorUpdater::update_value.update_valuerequiresValue::Float64, runsvalidate_single_input, thenupdate_single. Every non-UnivMon family therefore behaves exactly as before.UnivMonUpdater::update_valuecalls the newUnivMonAccumulator::insert_value:Null→ no-op.Float64→ the existinginsert_samplepath (unchanged Float64 behavior).Bool/Int64/Utf8→ encode with the existing type-taggedValue::key()bytes, check the bucket counter for overflow, and insert asDataInput::Byteswith weight 1.Because
Value::key()stores Int64 as its 8 little-endian bytes with a type tag, neighboring large integers stay distinct.UnivMonAccumulator::approx_memory_bytesnow adds the capacity ofString/Byteskeys held in each layer's heavy-hitter heap. Long string identities count against the runtime memory reservation. Afterclear(), the value returns to the empty-state size.One built state still feeds the existing
SketchStatistic::Cardinality,FrequencyL2andFrequencyEntropyevaluations.Key code interfaces
crates/asap-physical-operators/src/summary_kernels/factory.rscrates/asap-physical-operators/src/summary_kernels/univmon.rscrates/asap-physical-operators/src/operators/summary/mod.rs(signature unchanged, accepted types widened)Usage, as in
tests/univmon_execution.rs:Fields
AccumulatorUpdater::update_valueself&mut selfvalue&ValueNull; it skips NULL rows before the call.timestamp_msi64timecolumn, or0whentimeisNone. Passed toupdate_singleby the default; ignored by UnivMon.Result<(), String>Errfor a type the family does not accept or for an inputvalidate_single_inputrejects. The build maps it toError::Operator.UnivMonAccumulator::insert_valueValue::NullOk(()).Value::Float64(x),xfiniteinsert_sample(x), same as before this PR.Value::Bool/Value::Int64/Value::Utf8Value::key()), weight 1. Errors with"UnivMon count overflow"ifbucket_size + 1overflows.Err("unsupported UnivMon identity type or nonfinite sample").Operator::summary_buildinputSchemaReffamilySummaryFamilyTypevalidate_family. UnivMon is detected withkind.algorithm() == SketchAlgorithm::UnivMon.valueusizeUtf8,Int64orBool.timeOption<usize>Timestamp(unchanged).groupsVec<usize>validate_groups(unchanged).Examples
End to end (
tests/univmon_execution.rs::typed_frequency_keys_preserve_identity). For each key type, the input columnsrc_ipholds two keys, each twice, and one NULL:Utf8"192.0.2.1","192.0.2.2"Int649_007_199_254_740_992,9_007_199_254_740_993Boolfalse,trueAll values are checked within 0.01. The NULL row contributes nothing. The two Int64 keys are 2^53 and 2^53+1, which would be one key after a Float64 cast.
Kernel tests (
src/summary_kernels/univmon.rs).typed_keys_merge_and_roundtrip: twoUtf8states (192.0.2.1×2 and192.0.2.2×2) are merged, serialized and restored. The restored state still gives distinct 2, L2 √8, entropy 1.memory_accounts_for_string_identities: inserting one 4,096-byteUtf8key raisesapprox_memory_bytesby at least 4,096. Afterclear()it equals the empty size.Accepted vs rejected update columns
Float64Utf8,Int64,BoolFloat64update_valuedefault rejects non-Float64 valuesThe developer note in
docs/develop_docs/operator-design-acceptance.md§"Planner-layering follow-up: typed frequency inputs" records the same contract.Out of scope
Stack and validation
Stacked on #556 · Next: #559 · Reference/tracker: #528
Validation: the typed-input regression fails before the change and passes afterward; all 225 native runtime tests/doctests passed; formatting; all-target Clippy for native runtime with warnings denied.
The full restacked tip passes all 1,608 workspace tests/doctests (two existing ignores), formatting and workspace/all-target/all-feature Clippy with warnings denied. The inherited UnivMon precompute fixture was fixed in #552, and the latency fixture lint in #554; later branches were restacked atomically with explicit leases. CI has been retriggered.