Support NOT in StarTree Index

[Jackie-Jiang]

Support NOT in StarTree Index when it is followed by predicate or NOT (not with nested AND/OR)
E.g.

• WHERE NOT d1 > 10
• WHERE d1 > 10 AND NOT d2 < 10
• WHERE d1__COLUMN_NAME > 50 OR NOT d1__COLUMN_NAME > 10
• WHERE NOT NOT d1 > 10
• WHERE d2 < 95 AND (NOT d1 > 10 OR d1 > 50)
• WHERE d2 < 95 AND NOT d2 < 25 AND (d1 > 10 OR d1 < 50)
• WHERE NOT d1 = 95 AND (d1 > 90 OR d1 < 100)

In order to support NOT, dictionary based BasePredicateEvaluator are modified to support getNonMatchingDictIds(). Some refactor is done to avoid duplicate code.

Bugfix:

• Fix the unsorted dict ids returned from IN/NOT_IN predicate evaluator

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 64.97462% with 69 lines in your changes missing coverage. Please review.

Project coverage is 62.17%. Comparing base (59551e4) to head (5a96ce6).
Report is 1199 commits behind head on master.

Files with missing lines	Patch %	Lines
...core/startree/operator/StarTreeFilterOperator.java	51.42%	16 Missing and 1 partial ⚠️
.../org/apache/pinot/core/startree/StarTreeUtils.java	72.22%	8 Missing and 7 partials ⚠️
...core/operator/filter/predicate/PredicateUtils.java	35.00%	12 Missing and 1 partial ⚠️
...edicate/BaseDictionaryBasedPredicateEvaluator.java	52.94%	8 Missing ⚠️
...lter/predicate/RangePredicateEvaluatorFactory.java	78.12%	6 Missing and 1 partial ⚠️
...lter/predicate/NotInPredicateEvaluatorFactory.java	63.63%	3 Missing and 1 partial ⚠️
...icate/FSTBasedRegexpPredicateEvaluatorFactory.java	66.66%	2 Missing and 1 partial ⚠️
.../filter/predicate/InPredicateEvaluatorFactory.java	90.00%	1 Missing ⚠️
.../predicate/NotEqualsPredicateEvaluatorFactory.java	66.66%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #12988      +/-   ##
============================================
+ Coverage     61.75%   62.17%   +0.42%     
+ Complexity      207      198       -9     
============================================
  Files          2436     2502      +66     
  Lines        133233   136600    +3367     
  Branches      20636    21151     +515     
============================================
+ Hits          82274    84937    +2663     
- Misses        44911    45379     +468     
- Partials       6048     6284     +236     

Flag	Coverage Δ
custom-integration1	<0.01% <0.00%> (-0.01%)	⬇️
integration	<0.01% <0.00%> (-0.01%)	⬇️
integration1	<0.01% <0.00%> (-0.01%)	⬇️
integration2	0.00% <0.00%> (ø)
java-11	62.13% <64.97%> (+0.42%)	⬆️
java-21	62.06% <64.97%> (+0.44%)	⬆️
skip-bytebuffers-false	62.15% <64.97%> (+0.41%)	⬆️
skip-bytebuffers-true	62.04% <64.97%> (+34.31%)	⬆️
temurin	62.17% <64.97%> (+0.42%)	⬆️
unittests	62.17% <64.97%> (+0.42%)	⬆️
unittests1	46.73% <64.97%> (-0.16%)	⬇️
unittests2	27.94% <0.00%> (+0.21%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[npawar]

Do we need docs updated for this?

[Jackie-Jiang]

Do we need docs updated for this?

Yes. Let me do that!