Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for raw JSON filter predicates #13283

Merged
merged 1 commit into from
Jun 1, 2024

Conversation

yashmayya
Copy link
Collaborator

@yashmayya yashmayya commented May 31, 2024

  • Currently, raw JSON values (non dictionary encoded) can't be queried directly via predicates like =, !=, IN, NOT IN and JSON columns are typically queried using the JSON_MATCH filter predicate that uses the JSON index.
  • Attempting to do so results in errors like:
QueryExecutionError:
org.apache.pinot.spi.exception.BadQueryRequestException: java.lang.IllegalStateException: Unsupported data type: JSON
	at org.apache.pinot.core.operator.filter.predicate.PredicateEvaluatorProvider.getPredicateEvaluator(PredicateEvaluatorProvider.java:94)
	at org.apache.pinot.core.operator.filter.predicate.PredicateEvaluatorProvider.getPredicateEvaluator(PredicateEvaluatorProvider.java:100)
	at org.apache.pinot.core.plan.FilterPlanNode.constructPhysicalOperator(FilterPlanNode.java:310)
	at org.apache.pinot.core.plan.FilterPlanNode.run(FilterPlanNode.java:93)
...
Caused by: java.lang.IllegalStateException: Unsupported data type: JSON
	at org.apache.pinot.core.operator.filter.predicate.EqualsPredicateEvaluatorFactory.newRawValueBasedEvaluator(EqualsPredicateEvaluatorFactory.java:87)
	at org.apache.pinot.core.operator.filter.predicate.PredicateEvaluatorProvider.getPredicateEvaluator(PredicateEvaluatorProvider.java:76)
	... 24 more
  • However, for certain simple JSON values, it could be useful to allow direct comparison using predicates like =, !=, IN, NOT IN via direct string matching like is done for STRING columns since the stored type for JSON columns is also STRING.
  • We'd need to document that things like formatting / spacing and object key order aren't considered (since that would slow things down significantly) and such use cases are better served with the JSON index and JSON_MATCH filter predicate.

@codecov-commenter
Copy link

codecov-commenter commented May 31, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 62.14%. Comparing base (59551e4) to head (8d7e1e8).
Report is 1150 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #13283      +/-   ##
============================================
+ Coverage     61.75%   62.14%   +0.38%     
+ Complexity      207      198       -9     
============================================
  Files          2436     2536     +100     
  Lines        133233   139403    +6170     
  Branches      20636    21548     +912     
============================================
+ Hits          82274    86626    +4352     
- Misses        44911    46282    +1371     
- Partials       6048     6495     +447     
Flag Coverage Δ
custom-integration1 <0.01% <ø> (-0.01%) ⬇️
integration <0.01% <ø> (-0.01%) ⬇️
integration1 <0.01% <ø> (-0.01%) ⬇️
integration2 0.00% <ø> (ø)
java-11 62.11% <ø> (+0.40%) ⬆️
java-21 62.03% <ø> (+0.40%) ⬆️
skip-bytebuffers-false 62.13% <ø> (+0.38%) ⬆️
skip-bytebuffers-true 62.01% <ø> (+34.28%) ⬆️
temurin 62.14% <ø> (+0.38%) ⬆️
unittests 62.13% <ø> (+0.38%) ⬆️
unittests1 46.71% <ø> (-0.18%) ⬇️
unittests2 27.73% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Jackie-Jiang Jackie-Jiang marked this pull request as ready for review June 1, 2024 00:29
@Jackie-Jiang Jackie-Jiang merged commit 34bab22 into apache:master Jun 1, 2024
19 checks passed
gortiz pushed a commit to gortiz/pinot that referenced this pull request Jun 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants