Include ALL modifier for set operators in multi-stage query engine physical explain plan

[yashmayya]

• Implement INTERSECT ALL, EXCEPT ALL set operations for the multi-stage query engine #13151 added support for INTERSECT ALL and EXCEPT ALL set operators in the multi-stage query engine.
• The logical plan for queries using these operators does contain whether or not the ALL modifier was used.
• For instance, the output for EXPLAIN PLAN FOR SELECT * FROM (SELECT a FROM A) EXCEPT ALL (SELECT b FROM B); looks like this (Calcite syntax):

LogicalMinus(all=[true])
  PinotLogicalExchange(distribution=[hash[0]])
    LogicalProject(a=[$3])
      LogicalTableScan(table=[[default, A]])
  PinotLogicalExchange(distribution=[hash[0]])
    LogicalProject(b=[$3])
      LogicalTableScan(table=[[default, B]])

• The physical plan for queries, however, currently doesn't include whether or not the ALL modifier was used. So the output for both EXPLAIN IMPLEMENTATION PLAN FOR SELECT * FROM (SELECT a FROM A) EXCEPT ALL (SELECT b FROM B); and EXPLAIN IMPLEMENTATION PLAN FOR SELECT * FROM (SELECT a FROM A) EXCEPT (SELECT b FROM B); would look the same:

[0]@192.168.29.25:64480|[0] MAIL_RECEIVE(BROADCAST_DISTRIBUTED)
└── [1]@192.168.29.25:64487|[0] MAIL_SEND(BROADCAST_DISTRIBUTED)->{[0]@192.168.29.25:64480|[0]}
    └── [1]@192.168.29.25:64487|[0] MINUS
        └── [1]@192.168.29.25:64487|[0] MAIL_RECEIVE(HASH_DISTRIBUTED)
        └── [1]@192.168.29.25:64487|[0] MAIL_RECEIVE(HASH_DISTRIBUTED)

• This PR fixes the issue by updating SetOpNode::explain which is called in the PhysicalExplainPlanVisitor here. The output for EXPLAIN IMPLEMENTATION PLAN FOR SELECT * FROM (SELECT a FROM A) EXCEPT ALL (SELECT b FROM B); now looks like:

[0]@192.168.29.25:64480|[0] MAIL_RECEIVE(BROADCAST_DISTRIBUTED)
└── [1]@192.168.29.25:64487|[0] MAIL_SEND(BROADCAST_DISTRIBUTED)->{[0]@192.168.29.25:64480|[0]}
    └── [1]@192.168.29.25:64487|[0] MINUS_ALL
        └── [1]@192.168.29.25:64487|[0] MAIL_RECEIVE(HASH_DISTRIBUTED)
        └── [1]@192.168.29.25:64487|[0] MAIL_RECEIVE(HASH_DISTRIBUTED)

[gortiz]

Does it works in Union? AFAIR union always behaves as all = true. Is this attribute always true in union?

[yashmayya]

Yeah it does look like this attribute is always true for the union set operation (i.e., regardless of whether we use UNION or UNION ALL). I guess that isn't inaccurate though since the actual semantics being implemented are indeed the UNION ALL semantics and the logical plan also contains LogicalUnion(all=[true]) for both UNION and UNION ALL?

[yashmayya]

Ah, I just noticed that the query plan for UNION contains an aggregation on top to only get unique values from the result of the UNION ALL, interesting.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 62.15%. Comparing base (59551e4) to head (dd67957).
Report is 451 commits behind head on master.

Files	Patch %	Lines
...apache/pinot/query/planner/plannode/SetOpNode.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #13166      +/-   ##
============================================
+ Coverage     61.75%   62.15%   +0.40%     
+ Complexity      207      198       -9     
============================================
  Files          2436     2517      +81     
  Lines        133233   137895    +4662     
  Branches      20636    21340     +704     
============================================
+ Hits          82274    85713    +3439     
- Misses        44911    45798     +887     
- Partials       6048     6384     +336     

Flag	Coverage Δ
custom-integration1	<0.01% <0.00%> (-0.01%)	⬇️
integration	<0.01% <0.00%> (-0.01%)	⬇️
integration1	<0.01% <0.00%> (-0.01%)	⬇️
integration2	0.00% <0.00%> (ø)
java-11	62.12% <0.00%> (+0.41%)	⬆️
java-21	62.04% <0.00%> (+0.41%)	⬆️
skip-bytebuffers-false	62.14% <0.00%> (+0.39%)	⬆️
skip-bytebuffers-true	62.01% <0.00%> (+34.28%)	⬆️
temurin	62.15% <0.00%> (+0.40%)	⬆️
unittests	62.15% <0.00%> (+0.40%)	⬆️
unittests1	46.74% <0.00%> (-0.15%)	⬇️
unittests2	27.77% <0.00%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.