Skip to content

Commit d2fb03f

Browse files
committed
Keep no-match rows when pulling up a correlated aggregate subquery
With the Postgres planner (optimizer=off or an ORCA fallback), a correlated scalar subquery with an aggregate is pulled up into an INNER join with a grouped subquery (convert_EXPR_to_join), which drops outer rows that have no match. The original subquery keeps them: it computes the aggregate over empty input, so e.g. COUNT yields 0 there: select ... from t1 where t1.a > (select count(*) from t2 where t2.a = t1.d); A row with no match in t2 must be compared as "t1.a > 0" and can pass, but the INNER join dropped it. To fix this, pull the subquery up into a LEFT join, so no-match rows survive as null-extended rows, and rewrite the comparison to return the same value the subquery would: outer OP CASE WHEN match_flag THEN expr ELSE empty_input_default END match_flag is a constant TRUE column added to the subquery. For a matched row the CASE returns the real expression; for a null-extended row the flag is NULL and the CASE returns the empty-input default (0 for COUNT, NULL for other aggregates). The comparison runs above the LEFT join as a filter, not as the join condition: as a join qual it would null-extend matched rows that fail it, and the default would let them back in. The LEFT join is not always needed. If a no-match row cannot pass the comparison anyway -- e.g. "1 = (select count(*) ...)" turns into "1 = 0" for it -- dropping it is fine and the INNER join is kept as before. This is detected by substituting the empty-input default into the comparison and constant-folding it. Ordinary sum/avg/min/max comparisons fall into this group: their empty-input value is NULL, and a comparison with NULL does not pass, so those plans do not change. If the comparison cannot be placed above the join (the sublink is in an outer join's ON clause) or the subquery's targetlist is correlated, the pull-up bails out and the sublink runs as a SubPlan, as before. Adapted from open-gpdb open-gpdb/gpdb#397 and Greengage GreengageDB/greengage#546. Co-Authored-By: excaliiibur excaliiibur@foxmail.com
1 parent 9404105 commit d2fb03f

6 files changed

Lines changed: 447 additions & 24 deletions

File tree

‎contrib/pax_storage/src/test/regress/expected/eagerfree.out‎

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1398,21 +1398,20 @@ where i < (select count(*) from smallt where smallt.i = smallt2.i) order by 1,2,
13981398

13991399
explain select smallt2.* from smallt2
14001400
where i < (select count(*) from smallt where smallt.i = smallt2.i);
1401-
QUERY PLAN
1402-
---------------------------------------------------------------------------------------
1401+
QUERY PLAN
1402+
-----------------------------------------------------------------------------------------------------------------
14031403
Gather Motion 3:1 (slice1; segments: 3) (cost=1.57..3.56 rows=11 width=15)
1404-
-> Hash Join (cost=1.57..2.82 rows=4 width=15)
1404+
-> Hash Left Join (cost=1.57..2.82 rows=4 width=15)
14051405
Hash Cond: (smallt2.i = "Expr_SUBQUERY".csq_c0)
1406-
Join Filter: (smallt2.i < "Expr_SUBQUERY".csq_c1)
1406+
Filter: (smallt2.i < CASE WHEN "Expr_SUBQUERY".csq_c1 THEN "Expr_SUBQUERY".csq_c2 ELSE '0'::bigint END)
14071407
-> Seq Scan on smallt2 (cost=0.00..1.17 rows=17 width=15)
1408-
-> Hash (cost=1.55..1.55 rows=1 width=12)
1409-
-> Subquery Scan on "Expr_SUBQUERY" (cost=1.50..1.55 rows=1 width=12)
1410-
-> HashAggregate (cost=1.50..1.54 rows=1 width=12)
1408+
-> Hash (cost=1.55..1.55 rows=1 width=13)
1409+
-> Subquery Scan on "Expr_SUBQUERY" (cost=1.50..1.55 rows=1 width=13)
1410+
-> HashAggregate (cost=1.50..1.54 rows=1 width=13)
14111411
Group Key: smallt.i
1412-
Filter: (smallt.i < count(*))
14131412
-> Seq Scan on smallt (cost=0.00..1.33 rows=33 width=4)
14141413
Optimizer: Postgres query optimizer
1415-
(12 rows)
1414+
(11 rows)
14161415

14171416
-- Sort in MergeJoin
14181417
-- start_ignore

‎src/backend/cdb/cdbsubselect.c‎

Lines changed: 255 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
#include "parser/parse_relation.h" /* addRangeTableEntryForSubquery() */
3030
#include "parser/parsetree.h" /* rt_fetch() */
3131
#include "rewrite/rewriteManip.h"
32+
#include "utils/fmgroids.h" /* F_COUNT_ANY, F_COUNT_ */
3233
#include "utils/lsyscache.h" /* get_op_btree_interpretation() */
3334
#include "utils/syscache.h"
3435
#include "cdb/cdbsubselect.h" /* me */
@@ -42,6 +43,11 @@ static JoinExpr *make_join_expr(Node *larg, int r_rtindex, int join_type);
4243
static Node *make_lasj_quals(PlannerInfo *root, SubLink *sublink, int subquery_indx);
4344

4445
static Node *add_null_match_clause(Node *clause);
46+
static Expr *build_match_flag_case_expr(Var *flagVar, Var *aggVar, Expr *defaultExpr);
47+
static Expr *build_empty_input_default_expr(Node *expr, bool *sawNonNullDefault);
48+
static Node *replace_agg_with_empty_default_mutator(Node *node, void *context);
49+
static bool no_match_row_survives(PlannerInfo *root, OpExpr *opexp,
50+
Expr *defaultExpr);
4551

4652
typedef struct NonNullableVarsContext
4753
{
@@ -545,6 +551,18 @@ safe_to_convert_EXPR(SubLink *sublink, ConvertSubqueryToJoinContext *ctx1)
545551
if (!subselect->hasAggs)
546552
return false;
547553

554+
/*
555+
* A window function cannot survive the pull-up. Without it the subquery
556+
* has a plain aggregate and so produces exactly one row per outer row, and
557+
* the window runs over that single row. The pulled-up subquery is grouped
558+
* by the correlation columns, so the same window would run over every
559+
* group at once and compute a different value. (The rewrite of the
560+
* comparison below would also copy the WindowFunc into a qual above the
561+
* join, where there is no WindowAgg node to evaluate it.)
562+
*/
563+
if (subselect->hasWindowFuncs)
564+
return false;
565+
548566
/**
549567
* A LIMIT or OFFSET could interfere with the transformation of the
550568
* correlated qual to GROUP BY. (LIMIT >0 in a subquery that contains a
@@ -567,6 +585,14 @@ safe_to_convert_EXPR(SubLink *sublink, ConvertSubqueryToJoinContext *ctx1)
567585
if (list_length(subselect->targetList) != 1)
568586
return false;
569587

588+
/**
589+
* Correlation in the targetlist cannot be handled: the pulled-up
590+
* expression (and the empty-input default derived from it) would carry
591+
* upper-level Vars out of the subquery.
592+
*/
593+
if (contain_vars_of_level_or_above((Node *) subselect->targetList, 1))
594+
return false;
595+
570596

571597
/**
572598
* Walk the quals of the subquery to do a more fine grained check as to whether this subquery
@@ -630,6 +656,72 @@ convert_EXPR_to_join(PlannerInfo *root, OpExpr *opexp)
630656

631657
subselect->jointree->quals = ctx1.innerQual;
632658

659+
/*
660+
* An INNER join drops outer rows that have no matching inner
661+
* rows. Without the pull-up they are kept: the subquery
662+
* computes its expression over empty input (COUNT = 0, other
663+
* aggregates NULL) and the comparison may still pass.
664+
*
665+
* So plug the empty-input value into the comparison and run
666+
* eval_const_expressions() on it. FALSE or NULL means no-match
667+
* rows cannot pass and the INNER join is correct; otherwise use
668+
* a LEFT join to keep them.
669+
*/
670+
Expr *defaultExpr;
671+
TargetEntry *flagTLE = NULL;
672+
bool use_left_join;
673+
bool sawNonNullDefault;
674+
675+
defaultExpr = build_empty_input_default_expr((Node *) origSubqueryTLE->expr,
676+
&sawNonNullDefault);
677+
678+
/*
679+
* defaultExpr ends up in a qual of the outer query, where the planner
680+
* folds constant expressions at plan time. Substituting 0 for a COUNT
681+
* can turn a subexpression the original query only ever evaluated per
682+
* row into a constant one: "1/count(*)" becomes "1/0" and raises
683+
* "division by zero" while planning, even for a query that returns no
684+
* rows at all. A NULL default cannot do that -- it only propagates
685+
* through strict functions, which the planner folds without calling
686+
* them, and anything it does evaluate was already constant in the
687+
* original expression.
688+
*
689+
* So take the LEFT-join path only when the invented value is a plain
690+
* constant; otherwise leave the sublink to be planned as a SubPlan,
691+
* which keeps the original semantics.
692+
*/
693+
if (sawNonNullDefault && !IsA(defaultExpr, Const))
694+
return NULL;
695+
696+
use_left_join = no_match_row_survives(root, opexp, defaultExpr);
697+
698+
if (use_left_join)
699+
{
700+
/*
701+
* After the LEFT join the expression column is NULL both for a
702+
* no-match row and for a matched group whose expression is
703+
* genuinely NULL. To tell them apart, add a constant-TRUE
704+
* match-flag column to the subquery: it can be NULL only when
705+
* the LEFT join found no match and filled the subquery's
706+
* columns with NULLs.
707+
*
708+
* The flag goes BEFORE the expression column: with this
709+
* order the planner can drop the SubqueryScan node from the
710+
* plan.
711+
*/
712+
TargetEntry *aggTLE = (TargetEntry *) llast(subselect->targetList);
713+
714+
flagTLE = makeTargetEntry((Expr *) makeBoolConst(true, false),
715+
aggTLE->resno,
716+
pstrdup("csq_count_flag"),
717+
false);
718+
aggTLE->resno++;
719+
subselect->targetList = list_truncate(subselect->targetList,
720+
list_length(subselect->targetList) - 1);
721+
subselect->targetList = lappend(subselect->targetList, flagTLE);
722+
subselect->targetList = lappend(subselect->targetList, aggTLE);
723+
}
724+
633725
/**
634726
* Construct a new range table entry for the new pulled up subquery.
635727
*/
@@ -651,7 +743,8 @@ convert_EXPR_to_join(PlannerInfo *root, OpExpr *opexp)
651743

652744
join_expr->quals = joinQual;
653745

654-
TargetEntry *subselectAggTLE = (TargetEntry *) list_nth(subselect->targetList, list_length(subselect->targetList) - 1);
746+
/* The pulled-up expression column is last in either layout. */
747+
TargetEntry *subselectAggTLE = (TargetEntry *) llast(subselect->targetList);
655748

656749
/**
657750
* modify the op expr to involve the column that has the computed aggregate that needs to compared.
@@ -663,14 +756,174 @@ convert_EXPR_to_join(PlannerInfo *root, OpExpr *opexp)
663756
exprCollation((Node *) subselectAggTLE->expr),
664757
0);
665758

666-
list_nth_replace(opexp->args, 1, aggVar);
759+
if (use_left_join)
760+
{
761+
Var *flagVar;
762+
RangeTblEntry *joinRTE;
763+
int joinRTIndex;
764+
765+
join_expr->jointype = JOIN_LEFT;
766+
767+
/*
768+
* Give the outer join a range table entry and mark the Vars the
769+
* comparison uses as nulled by it. Since the removal of
770+
* outerjoin_delayed the planner keeps a clause above an outer
771+
* join only when the clause's Vars carry the join's relid in
772+
* varnullingrels; without this the comparison would be pushed
773+
* down to the subquery rel (or the join removed as useless) and
774+
* the no-match default would never apply.
775+
*/
776+
joinRTE = makeNode(RangeTblEntry);
777+
joinRTE->rtekind = RTE_JOIN;
778+
joinRTE->jointype = JOIN_LEFT;
779+
joinRTE->joinmergedcols = 0;
780+
joinRTE->eref = makeAlias("unnamed_join", NIL);
781+
joinRTE->inFromCl = false;
782+
root->parse->rtable = lappend(root->parse->rtable, joinRTE);
783+
joinRTIndex = list_length(root->parse->rtable);
784+
join_expr->rtindex = joinRTIndex;
785+
786+
flagVar = (Var *) makeVar(rteIndex, flagTLE->resno, BOOLOID, -1,
787+
InvalidOid, 0);
788+
flagVar->varnullingrels = bms_make_singleton(joinRTIndex);
789+
aggVar->varnullingrels = bms_make_singleton(joinRTIndex);
790+
list_nth_replace(opexp->args, 1,
791+
build_match_flag_case_expr(flagVar, aggVar, defaultExpr));
792+
}
793+
else
794+
{
795+
list_nth_replace(opexp->args, 1, aggVar);
796+
}
667797

668798
return join_expr;
669799
}
670800

671801
return NULL;
672802
}
673803

804+
/*
805+
* Build "CASE WHEN flagVar THEN aggVar ELSE defaultExpr END".
806+
*
807+
* flagVar is the subquery's match-flag column: TRUE for a matched group,
808+
* NULL for a null-extended no-match row.
809+
*/
810+
static Expr *
811+
build_match_flag_case_expr(Var *flagVar, Var *aggVar, Expr *defaultExpr)
812+
{
813+
CaseWhen *casewhen;
814+
CaseExpr *caseexpr;
815+
816+
Assert(flagVar != NULL);
817+
Assert(aggVar != NULL);
818+
Assert(defaultExpr != NULL);
819+
820+
casewhen = makeNode(CaseWhen);
821+
casewhen->expr = (Expr *) flagVar;
822+
casewhen->result = (Expr *) aggVar;
823+
casewhen->location = -1;
824+
825+
caseexpr = makeNode(CaseExpr);
826+
caseexpr->casetype = exprType((Node *) aggVar);
827+
caseexpr->casecollid = exprCollation((Node *) aggVar);
828+
caseexpr->arg = NULL;
829+
caseexpr->args = list_make1(casewhen);
830+
caseexpr->defresult = defaultExpr;
831+
caseexpr->location = -1;
832+
833+
return (Expr *) caseexpr;
834+
}
835+
836+
/*
837+
* Build the value the subquery's expression takes over empty input.
838+
*
839+
* *sawNonNullDefault is set if any aggregate contributed a non-NULL value,
840+
* i.e. if the result depends on a value this code invented rather than on a
841+
* NULL that the original expression would have produced anyway.
842+
*/
843+
static Expr *
844+
build_empty_input_default_expr(Node *expr, bool *sawNonNullDefault)
845+
{
846+
Node *rewritten;
847+
848+
*sawNonNullDefault = false;
849+
rewritten = replace_agg_with_empty_default_mutator(copyObject(expr),
850+
sawNonNullDefault);
851+
return (Expr *) rewritten;
852+
}
853+
854+
static Node *
855+
replace_agg_with_empty_default_mutator(Node *node, void *context)
856+
{
857+
Aggref *aggref;
858+
Oid default_type;
859+
Oid default_collation;
860+
int16 typlen;
861+
bool typbyval;
862+
863+
if (node == NULL)
864+
return NULL;
865+
866+
if (IsA(node, Aggref))
867+
{
868+
bool is_count;
869+
870+
aggref = (Aggref *) node;
871+
is_count = (aggref->aggfnoid == F_COUNT_ANY ||
872+
aggref->aggfnoid == F_COUNT_);
873+
if (is_count)
874+
{
875+
default_type = INT8OID;
876+
default_collation = InvalidOid;
877+
}
878+
else
879+
{
880+
default_type = aggref->aggtype;
881+
default_collation = exprCollation((Node *) aggref);
882+
}
883+
884+
/*
885+
* COUNT is 0 over empty input; every other aggregate is NULL. The
886+
* choice must follow the aggregate, not its result type: sum(int4)
887+
* also returns int8 but its empty-input value is NULL.
888+
*/
889+
get_typlenbyval(default_type, &typlen, &typbyval);
890+
if (is_count)
891+
*((bool *) context) = true;
892+
return (Node *) makeConst(default_type, -1, default_collation, typlen,
893+
is_count ? Int64GetDatum(0) : (Datum) 0,
894+
!is_count, typbyval);
895+
}
896+
897+
return expression_tree_mutator(node, replace_agg_with_empty_default_mutator,
898+
context);
899+
}
900+
901+
/*
902+
* no_match_row_survives
903+
*
904+
* Could a no-match row satisfy "outerExpr OP (subquery)"? Plug defaultExpr in
905+
* for the subquery and constant-fold: false if it folds to FALSE/NULL, else true.
906+
*/
907+
static bool
908+
no_match_row_survives(PlannerInfo *root, OpExpr *opexp, Expr *defaultExpr)
909+
{
910+
OpExpr *testexpr = (OpExpr *) copyObject(opexp);
911+
Node *folded;
912+
913+
list_nth_replace(testexpr->args, 1, copyObject(defaultExpr));
914+
folded = eval_const_expressions(root, (Node *) testexpr);
915+
916+
if (IsA(folded, Const))
917+
{
918+
Const *c = (Const *) folded;
919+
920+
if (c->constisnull || !DatumGetBool(c->constvalue))
921+
return false;
922+
}
923+
924+
return true;
925+
}
926+
674927
/* NOTIN subquery transformation -start */
675928

676929
/* check if NOT IN conversion to antijoin is possible */

‎src/backend/optimizer/prep/prepjointree.c‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -858,11 +858,39 @@ pull_up_sublinks_qual_recurse(PlannerInfo *root, Node *node,
858858

859859
if (IsA(rarg, SubLink))
860860
{
861+
/*
862+
* The pulled-up join is spliced in at *jtlink1, and in the
863+
* LEFT-join case the comparison itself moves there too, so
864+
* every Var of this query level used by the clause must be
865+
* available at that attach point. Otherwise (e.g. an outer
866+
* join's ON clause referencing the non-nullable side) leave
867+
* the sublink to be planned as a SubPlan.
868+
*/
869+
if (!bms_is_subset(pull_varnos(root, node), available_rels1))
870+
return node;
871+
861872
j = convert_EXPR_to_join(root, opexp);
862873
if (j)
863874
{
864875
/* Yes, insert the new join node into the join tree */
865876
j->larg = *jtlink1;
877+
878+
if (j->jointype == JOIN_LEFT)
879+
{
880+
/*
881+
* COUNT-preserving pull-up (see convert_EXPR_to_join).
882+
* opexp must run ABOVE the LEFT JOIN, not as its join
883+
* condition: as a join qual a matched row that fails it
884+
* would be treated as unmatched, null-extended, and let
885+
* back in by the no-match default of the CASE built by
886+
* convert_EXPR_to_join. Wrap the join in a FromExpr so
887+
* opexp stays a post-join filter.
888+
*/
889+
*jtlink1 = (Node *) makeFromExpr(list_make1(j), node);
890+
return NULL;
891+
}
892+
893+
/* Inner-join case: opexp stays as an ordinary qual. */
866894
*jtlink1 = (Node *) j;
867895
}
868896
return node;

0 commit comments

Comments
 (0)