Conversation
Alena0704
added
type: Bug
|
|
||
| if (root_ndistinct > 0) | ||
| { | ||
| GpPolicy *policy = GpPolicyFetch(stats->attr->attrelid); |
There was a problem hiding this comment.
Added the pfree(policy), thanks.
The cap is only an upper bound. When a value sits on a single segment it is still too high by the number of segments, and ORCA multiplies the per-column values, so with k grouping columns the error is numsegments^k and the multi-stage plan is still lost. Instead of capping, the merged value is now the root's ndistinct times the average number of segments a value sits on, taken from the partitions themselves. The sum of the partitions is still kept as an upper bound, and the average is at most the number of segments, so the value never exceeds the root's ndistinct times the number of segments either.
Reproduction (3 segments, ORCA). k has 13 values and is independent of the partitioning column, so every partition holds all of them, and a to f are bijections of k, so each of their values sits on one segment:
CREATE TABLE ndvbs_dk (id bigint, pk int, k int,
a text, b text, c text, d text, e text, f text)
DISTRIBUTED BY (k)
PARTITION BY RANGE (pk) (START (1) END (14) EVERY (1));
INSERT INTO ndvbs_dk
SELECT g, (g % 13) + 1, (g / 13) % 13,
md5((((g / 13) % 13) )::text), md5((((g / 13) % 13) + 100)::text),
md5((((g / 13) % 13) + 200)::text), md5((((g / 13) % 13) + 300)::text),
md5((((g / 13) % 13) + 400)::text), md5((((g / 13) % 13) + 500)::text)
FROM generate_series(1, 6000000) g;
ANALYZE ndvbs_dk;
SET optimizer = on;
EXPLAIN (ANALYZE) SELECT a, b, c, d, e, f, count(*)
FROM ndvbs_dk GROUP BY a, b, c, d, e, f;Before: a partition has 13, the root gets 39, and all 2.3M rows per segment go through the motion to produce 13 groups.
Gather Motion 3:1 (cost=0.00..4334.14 rows=15307 width=200) (actual time=11650.023..11650.023 rows=13 loops=1)
-> HashAggregate (cost=0.00..4322.73 rows=5103 width=200) (actual time=11650.023..11650.023 rows=5 loops=1)
-> Redistribute Motion 3:3 (cost=0.00..2613.36 rows=2000000 width=192) (actual time=2.000..8058.016 rows=2307694 loops=1)
-> Dynamic Seq Scan on ndvbs_dk (cost=0.00..697.20 rows=2000000 width=192) (actual rows=2307695 loops=1)
Execution Time: 11651.606 ms
After: the root gets 13, and 5 rows go through the motion.
Gather Motion 3:1 (cost=0.00..3141.49 rows=15307 width=200) (actual time=3744.007..3744.007 rows=13 loops=1)
-> Finalize HashAggregate (cost=0.00..3130.08 rows=5103 width=200) (actual time=3743.007..3743.007 rows=5 loops=1)
-> Redistribute Motion 3:3 (cost=0.00..3125.15 rows=5103 width=200) (actual time=2461.005..3743.007 rows=5 loops=1)
-> Streaming Partial HashAggregate (cost=0.00..3121.96 rows=5103 width=200) (actual time=3742.007..3742.007 rows=5 loops=1)
-> Dynamic Seq Scan on ndvbs_dk (cost=0.00..697.20 rows=2000000 width=192) (actual rows=2307695 loops=1)
Execution Time: 3744.836 ms
When ANALYZE builds statistics for a partitioned table, it adds up the per-segment ndistinct of all partitions. If the same values appear in every partition, they are counted once per partition, so the result is too big: 8 partitions give 8 times the real value. ORCA uses this number to estimate how many rows a partial aggregate returns. With the inflated number it thinks the partial aggregate removes almost no rows, and picks a one-stage aggregate that sends all rows through the motion. A segment cannot have more distinct values than the whole table, so cap the value at the table's ndistinct times the number of segments.
204d280 to
45bb7bd
Compare
3 participants
ORCA does not use two-stage aggregation on partitioned tables
ANALYZE adds up the per-segment ndistinct of all partitions to get the value
for the parent table (
STATISTIC_KIND_NDV_BY_SEGMENTS). When the same valuesare in every partition, they are counted once per partition, so the value is
too big.
ORCA uses this value to guess how many rows are left after the partial
aggregate. With a value that is too big, ORCA thinks the partial aggregate
does not help and sends all rows through the motion. The more partitions and
grouping columns, the bigger the error.
A segment cannot have more distinct values than the whole table, so
cap the value at the table's ndistinct times the number of segments.
Reproduction (3 segments, ORCA):
Before (
REL_2_STABLE): one-stage plan, all 7M rows go through the motion.After: two-stage plan, only 660 rows go through the motion.
Type of Change
Breaking Changes
Test Plan
make installcheckmake -C src/test installcheck-cbdb-parallelImpact
Performance:
User-facing changes:
Dependencies:
Checklist
Additional Context
CI Skip Instructions