49 Pull requests merged by 18 people

• Fix timestamp literal handling in the multi-stage query engine

  #14502 merged Nov 20, 2024

• add logging + metric on segment selection to upsert compaction task

  #14469 merged Nov 19, 2024

• Realtime segment size threshold metrics

  #14485 merged Nov 19, 2024

• Make TableDataManagerProvider pluggable

  #14470 merged Nov 19, 2024

• Bump com.google.errorprone:error_prone_annotations from 2.35.1 to 2.36.0

  #14492 merged Nov 19, 2024

• Bump software.amazon.awssdk:bom from 2.29.15 to 2.29.16

  #14491 merged Nov 19, 2024

• Bump net.openhft:chronicle-core from 2.27ea0 to 2.27ea1

  #14490 merged Nov 19, 2024

• Apply filter reduce expressions Calcite rule at the end to prevent literal-only filter pushdown to leaf stage

  #14448 merged Nov 19, 2024

• Fix BitmapCollection::orCardinality API when combining a non-inverted bitmap with an inverted one

  #14487 merged Nov 19, 2024

• Show multi-stage engine explain plan for integration test query failures with the multi-stage query engine

  #14475 merged Nov 19, 2024

• Add TLS support to mailboxes used in the multi-stage engine

  #14476 merged Nov 19, 2024

• Use TransformPipeline in SegmentMapper

  #14425 merged Nov 19, 2024

• Revert "Using local copy of segment instead of downloading from remote (#14429)"

  #14484 merged Nov 19, 2024

• Spark upsert table backfill support

  #14443 merged Nov 18, 2024

• Add Support to add prefix/suffix to Tasks

  #14478 merged Nov 18, 2024

• [spool] Define equivalence between stages

  #14296 merged Nov 18, 2024

• Bump net.openhft:chronicle-core from 2.26ea2 to 2.27ea0

  #14481 merged Nov 18, 2024

• Bump software.amazon.awssdk:bom from 2.29.14 to 2.29.15

  #14480 merged Nov 18, 2024

• Bump net.openhft:posix from 2.26ea3 to 2.27ea0

  #14482 merged Nov 18, 2024

• Upgrade orc from 1.9.4 to 1.9.5

  #14471 merged Nov 17, 2024

• Refactor String.format to Concatenation

  #14418 merged Nov 16, 2024

• Fix Flaky test # 14456

  #14462 merged Nov 15, 2024

• Bump software.amazon.awssdk:bom from 2.29.11 to 2.29.14

  #14466 merged Nov 15, 2024

• log the instance configs properly

  #14468 merged Nov 15, 2024

• Bump codecov/codecov-action from 4 to 5

  #14463 merged Nov 15, 2024

• Bump com.nimbusds:nimbus-jose-jwt from 9.46 to 9.47

  #14465 merged Nov 15, 2024

• Enhance funnel functions to accept a new option for maxStepDuration

  #14434 merged Nov 15, 2024

• Handle kafka backward incompatible props for AWS

  #14446 merged Nov 15, 2024

• Add error log sampling to CLPLogMessageDecoder

  #14366 merged Nov 15, 2024

• add metric of startup time for all components

  #14445 merged Nov 14, 2024

• log segment name at best effort upon query exception to help locate the error segment

  #14440 merged Nov 14, 2024

• use offheapfststore as before

  #14457 merged Nov 14, 2024

• Bump org.codehaus.mojo:versions-maven-plugin from 2.17.1 to 2.18.0

  #14454 merged Nov 14, 2024

• Fix flaky upsert integration tests

  #14447 merged Nov 14, 2024

• [Flaky-test] Fix flaky kafka consumer tests

  #14458 merged Nov 14, 2024

• Add instance level consumer dir usage metric

  #14430 merged Nov 14, 2024

• Update window function test plans to enable tests for supported functionality

  #14449 merged Nov 14, 2024

• Using local copy of segment instead of downloading from remote

  #14429 merged Nov 14, 2024

• Upgrade kafka3 from 3.8.0 to 3.9.0

  #14442 merged Nov 14, 2024

• Add hex decimal to long scalar functions

  #14435 merged Nov 13, 2024

• Bump net.openhft:posix from 2.26ea2 to 2.26ea3

  #14410 merged Nov 13, 2024

• Bump nl.jqno.equalsverifier:equalsverifier from 3.17.2 to 3.17.3

  #14412 merged Nov 13, 2024

• Bump com.puppycrawl.tools:checkstyle from 10.20.0 to 10.20.1

  #14416 merged Nov 13, 2024

• Bump com.nimbusds:nimbus-jose-jwt from 9.45 to 9.46

  #14421 merged Nov 13, 2024

• Bump org.webjars:swagger-ui from 5.18.0 to 5.18.2

  #14422 merged Nov 13, 2024

• Bump parquet.version from 1.14.3 to 1.14.4

  #14427 merged Nov 13, 2024

• Bump org.quartz-scheduler:quartz from 2.3.2 to 2.5.0

  #14436 merged Nov 13, 2024

• Bump io.netty:netty-bom from 4.1.114.Final to 4.1.115.Final

  #14437 merged Nov 13, 2024

• Bump software.amazon.awssdk:bom from 2.29.5 to 2.29.11

  #14438 merged Nov 13, 2024

21 Pull requests opened by 13 people

• Add `optimizeDictionaryType` config to automatically choose dictionary type

  #14444 opened Nov 13, 2024

• Add a Server API to list segments that need to be refreshed for a table

  #14451 opened Nov 14, 2024

• Add broker setting to override default implicit query response limit

  #14452 opened Nov 14, 2024

• Allow override of segment name when uploading a segment tar file

  #14455 opened Nov 14, 2024

• Pauseless Consumption

  #14460 opened Nov 15, 2024

• Support for S3A Connector #14312

  #14474 opened Nov 16, 2024

• Upsert small segment merger task in minions

  #14477 opened Nov 18, 2024

• [WIP] Stops consumption if Num Of Values per segment column crosses a threshold

  #14479 opened Nov 18, 2024

• Fix default value handling in REGEXP_EXTRACT transform function

  #14489 opened Nov 19, 2024

• Add test to check data and index types compatibility

  #14493 opened Nov 19, 2024

• Allow reload for UploadedRealtimeSegmentName segments

  #14494 opened Nov 19, 2024

• [Spool] Introduce stage replacer and change send nodes to be able to send to more than one stage

  #14495 opened Nov 19, 2024

• Add metrics for bytes ingested and bytes dropped.

  #14496 opened Nov 19, 2024

• Retain boolean type from CLPLogRecordExtractor decoder.

  #14497 opened Nov 19, 2024

• [timeseries] Response Size Limit, Metrics and Series Limit

  #14501 opened Nov 20, 2024

• [WIP] Adds check if controller tmp dir exists

  #14503 opened Nov 20, 2024

• Bump software.amazon.awssdk:bom from 2.29.16 to 2.29.17

  #14504 opened Nov 20, 2024

• Bump commons-io:commons-io from 2.11.0 to 2.18.0

  #14505 opened Nov 20, 2024

• Fix crc mismatch during deepstore upload retry task

  #14506 opened Nov 20, 2024

• [Spool] Actual implementation

  #14507 opened Nov 20, 2024

• restrict UploadedRealtimeSegmentName upload to realtime table

  #14508 opened Nov 20, 2024

11 Issues closed by 5 people

• Query execution error when filtering constant columns from subquery

  #14010 closed Nov 19, 2024

• FastFilteredCountOperator returns incorrect results for an OR filter on range index filter operator + sorted index exclusive filter operator

  #14486 closed Nov 19, 2024

• [Flaky-Test] Fix Flaky tests in Pinot-Core org.apache.pinot.queries.ExplainPlanQueriesTest

  #11939 closed Nov 15, 2024

• [flakytest] timeout in fork most likely on RealtimeKinesisIntegrationTest

  #11135 closed Nov 15, 2024

• Flaky Test `QueryLoggerTest.shouldHandleRaceConditionsWithDroppedQueries `

  #10512 closed Nov 15, 2024

• RealtimeNonReplicaGroupSegmentAssignmentTest.testRelocateCompletedSegments is flaky

  #8791 closed Nov 15, 2024

• [Flaky test] PinotHelixResourceManagerTest.testSegmentReplacement is flaky

  #9651 closed Nov 15, 2024

• [Flaky-test] HybridClusterIntegrationTest.testReload()

  #14456 closed Nov 15, 2024

• Nullity Error When Querying `MAX` On Non-Numeric Column

  #14459 closed Nov 15, 2024

• pinot 1.2.0 unrecognizedProperties in fromat following documentation exactly

  #14464 closed Nov 15, 2024

• [Flaky-test] UpsertTableIntegrationTest.testUpsertCompactionWithSoftDelete

  #13594 closed Nov 14, 2024

10 Issues opened by 7 people

• [pinot-client] JsonAsyncHttpPinotClientTransportFactory builder pattern is implemented incorrectly for scheme

  #14500 opened Nov 20, 2024

• Add support for lazy reload during schema evolution

  #14499 opened Nov 19, 2024

• make the query rate limiting bucket configurable (minute/second) level

  #14498 opened Nov 19, 2024

• REGEXP_EXTRACT incorrect default value

  #14488 opened Nov 19, 2024

• RefreshSegmentTaskGenerator: Evaluate and Use Server APIs to determine if refresh is needed.

  #14483 opened Nov 18, 2024

• [Flaky-test] EqualsTransformFunctionTest.testBinaryOperatorTransformFunction

  #14473 opened Nov 15, 2024

• [Flaky-test] MapFieldTypeRealtimeTest.testQueries

  #14472 opened Nov 15, 2024

• Support for multi path data directories per server to utilize multiple disks present in a node

  #14461 opened Nov 15, 2024

• Add a Server API to list segments of a table that are out of date

  #14450 opened Nov 14, 2024

• [Multi-stage] Improve planner rule to not push down expression/filter into the dimension table side

  #14441 opened Nov 13, 2024

32 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Minion Task to support automatic Segment Refresh

  #14300 commented on Nov 20, 2024 • 30 new comments

• Add multi stream ingestion support

  #13790 commented on Nov 20, 2024 • 12 new comments

• Allows multiple requests per server per request ID

  #13742 commented on Nov 16, 2024 • 11 new comments

• Add support for storing non-CLP-encodable values in a separate column; Replace CLP row value that are too large to store in FixedByteMVMutableForwardIndex with an error message.

  #14365 commented on Nov 19, 2024 • 7 new comments

• Support TIMESTAMP_NTZ, DATE and TIME datatype

  #14398 commented on Nov 19, 2024 • 2 new comments

• Add immtuable CLPForwardIndex creator and related classes

  #14288 commented on Nov 15, 2024 • 2 new comments

• Configurable sketch accuracy in merge rollup task

  #14373 commented on Nov 19, 2024 • 2 new comments

• Controller AccessControl interface accepts Request parameter

  #14414 commented on Nov 18, 2024 • 1 new comment

• Add `DATE_TRUNC` Optimizer

  #14385 commented on Nov 19, 2024 • 1 new comment

• [WIP] Pluggable fsm

  #14088 commented on Nov 19, 2024 • 0 new comments

• Extend the merge rollup task capabilities

  #14355 commented on Nov 17, 2024 • 0 new comments

• Strip Matrix parameter from BasePath check

  #14383 commented on Nov 19, 2024 • 0 new comments

• Add table and column information to query errors

  #14388 commented on Nov 14, 2024 • 0 new comments

• add PulsarAvroMessageDecoder

  #14397 commented on Nov 20, 2024 • 0 new comments

• Spark batch ingestion common code abstraction.

  #14415 commented on Nov 18, 2024 • 0 new comments

• Add Option to Optimize `NOT` Predicates with Push Down

  #14432 commented on Nov 14, 2024 • 0 new comments

• Pauseless consumption

  #14439 commented on Nov 13, 2024 • 0 new comments

• [proposal] Reuse common expressions in a query

  #14196 commented on Nov 20, 2024 • 0 new comments

• SQL Date functions not available

  #4835 commented on Nov 20, 2024 • 0 new comments

• pinot-clients have too many dependencies

  #11507 commented on Nov 20, 2024 • 0 new comments

• Support new data type: `TimestampWithTimeZone`

  #13004 commented on Nov 20, 2024 • 0 new comments

• Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

  #14372 commented on Nov 19, 2024 • 0 new comments

• Pushing the segment files generated through spark with cluster mode/standalone job are failing with "Could not find paramName tableName in path or query params of the API v2/segmments" with s3 as a deep store

  #14407 commented on Nov 17, 2024 • 0 new comments

• Controller and server total documents gauge for realtime tables are inconsistent

  #14390 commented on Nov 17, 2024 • 0 new comments

• MultiNodesOfflineClusterIntegrationTest is flaky

  #11122 commented on Nov 15, 2024 • 0 new comments

• Flaky Tests: TPCHQueryIntegrationTest and pinot-kafka-3.0

  #14099 commented on Nov 15, 2024 • 0 new comments

• Handle `IS NULL/IS NOT NULL` filtering when ExpressionTransformer outputs `null`

  #14433 commented on Nov 14, 2024 • 0 new comments

• Unsupported Range index types should fail with table validation

  #14431 commented on Nov 14, 2024 • 0 new comments

• [Feature Request] Provide user access to all tables except tableX

  #14076 commented on Nov 14, 2024 • 0 new comments

• Pauseless Consumption

  #10147 commented on Nov 14, 2024 • 0 new comments

• [Pinot 1.2.0] Batch upload for realtime table using Spark fails with error "Creation time must be set for uploaded realtime segment name generator"

  #14083 commented on Nov 13, 2024 • 0 new comments

• Reduce the amount of trace info

  #9523 commented on Nov 13, 2024 • 0 new comments