-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[null-aggr] Add null handling support in mode
aggregation
#12227
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #12227 +/- ##
============================================
- Coverage 61.75% 61.73% -0.03%
Complexity 207 207
============================================
Files 2436 2451 +15
Lines 133233 133592 +359
Branches 20636 20684 +48
============================================
+ Hits 82274 82467 +193
- Misses 44911 45044 +133
- Partials 6048 6081 +33
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
...org/apache/pinot/core/query/aggregation/function/NullableSingleInputAggregationFunction.java
Show resolved
Hide resolved
57881de
to
18a2c7f
Compare
18a2c7f
to
1b0c69a
Compare
…pport null handling
1b0c69a
to
6c00881
Compare
8361870
to
ecf7235
Compare
ecf7235
to
1b06f49
Compare
...org/apache/pinot/core/query/aggregation/function/NullableSingleInputAggregationFunction.java
Show resolved
Hide resolved
...org/apache/pinot/core/query/aggregation/function/NullableSingleInputAggregationFunction.java
Show resolved
Hide resolved
@Jackie-Jiang this should be ready to merge. Can you take a look? As far as I can see the only discussion pending is whether the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly good. I saw you added mode into the AllNullQueriesTest
, but somehow I didn't find where the null
intermediate result is handled
.../src/main/java/org/apache/pinot/core/query/aggregation/function/ModeAggregationFunction.java
Outdated
Show resolved
Hide resolved
.../src/main/java/org/apache/pinot/core/query/aggregation/function/ModeAggregationFunction.java
Outdated
Show resolved
Hide resolved
...org/apache/pinot/core/query/aggregation/function/NullableSingleInputAggregationFunction.java
Outdated
Show resolved
Hide resolved
...org/apache/pinot/core/query/aggregation/function/NullableSingleInputAggregationFunction.java
Outdated
Show resolved
Hide resolved
@@ -467,7 +501,11 @@ public ColumnDataType getFinalResultColumnType() { | |||
@Override | |||
public Double extractFinalResult(Map<? extends Number, Long> intermediateResult) { | |||
if (intermediateResult.isEmpty()) { | |||
return DEFAULT_FINAL_RESULT; | |||
if (_nullHandlingEnabled) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When _nullHandlingEnabled
is true
, and all input values are null
, will this map be null
?
Same for merge()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this map = intermediateResult
?
In that case, no. intermediateResult
is created in extractAggregationResult(AggregationResultHolder)
, which calls extractIntermediateResult
which returns not null.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm adding a new test to show the result when all stored data is null
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/FieldConfig.java
Outdated
Show resolved
Hide resolved
The change I made there was to create one test per query instead of executing all in the same test method. The only reason to apply the change is to improve the UX in the case there is at least one failure. Before only the first error was reported and once you fix the issue there you had to run the tests again to check if there were other failures. Now all failures are reported at the same time. I've modified the code long ago, so I don't remember the exact changes, but reading the diff it doesn't look I've changed anything related to null intermediate result. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
This PR adds null handling support in the
mode
aggregation. Specifically, it modifies the code to ignore null values whenmode
is evaluated withnullHandlingEnabled
, trying to follow the following postgres logic:It is recommended to review this PR assuming PR #12226 is merged, which is equivalent to compare this PR from commit commit f966b1b295134a08a3b201dadcedd91ca7d01c0a to head. As you can see there, this PR also changes the test and shows that when null handling is enabled, nulls are ignored.