What Changed

This release comes with several Improvements and Bug Fixes for the Multistage Engine, Upserts and Compaction. There are a ton of other small features and general bug fixes.

Multistage Engine Improvements

Features

New Window Functions: LEAD, LAG, FIRST_VALUE, LAST_VALUE #12878 #13340

• LEAD allows you to access values after the current row in a frame.
• LAG allows you to access values before the current row in a frame.
• FIRST_VALUE and LAST_VALUE return the respective extremal values in the frame.

Support for Logical Database in V2 Engine #12591 #12695

• V2 Engine now supports a "database" construct, enabling table namespace isolation within the same Pinot cluster.
• Improves user experience when multiple users are using the same Pinot Cluster.
• Access control policies can be set at the database level.
• Database can be selected in a query using a SET statement, such as SET database=my_db;.

Improved Multi-Value (MV) and Array Function Support

• Added array sum aggregation functions for point-wise array operations #13324.
• Added support for valueIn MV transform function #13443.
• Fixed bug in numeric casts for MV columns in filters #13425.
• Fixed NPE in ArrayAgg when a column contains no data #13358.
• Fixed array literal handling #13345.

Support for WITHIN GROUP Clause and ListAgg #13146

• WITHIN GROUP Clause can be used to process rows in a given order within a group.
• One of the most common use-cases for this is the ListAgg function, which when combined with WITHIN GROUP can be used to concatenate strings in a given order.

Scalar/Transform Function and Set Operation Improvements

• Added Geospatial Scalar Function support for use in intermediate stage in the v2 query engine #13457.
• Fix 'WEEK' transform function #13483.
• Support EXTRACT as a scalar function #13463.
• Added support for ALL modifier for INTERSECT and EXCEPT Set Operations #13151 #13166.

Improved Literal Handling Support

• Fixed bug in handling literal arguments in aggregation functions like Percentile #13282.
• Allow INT and FLOAT literals #13078.
• Fixed literal handling for all types #13344 #13345.
• Fixed null literal handling for null intolerant functions #13255.

Metrics Improvements

• Added new metrics for tracking queries executed globally and at the table level #12982.
• New metrics to track join counts and window function counts #13032.
• Multiple meters and timers to track Multistage Engine Internals #13035.

Notable Improvements and Bug Fixes

• Improved Window operators resiliency, with new checks to make sure the window doesn't grow too large #13180 #13428 #13441.
• Optimized Group Key generation #12394.
• Fixed SortedMailboxReceiveOperator to honor convention of pulling at most 1 EOS block #12406.
• Improvement in how execution stats are handled #12517 #12704 #13136.
• Use Protobuf instead of Reflection for Plan Serialization #13221.

Upsert Compaction and Minion Improvements

Features and Improvements

Minion Resource Isolation #12459 #12786

• Minions now support resource isolation based on an instance tag.
• Instance tag is configured at table level, and can be set for each task on a table.
• This enables you to implement arbitrary resource isolation strategies, i.e. you can use a set of Minion Nodes for running any set of tasks across any set of tables.

Greedy Upsert Compaction Scheduling #12461

• Upsert compaction now schedules segments for compaction based on the number of invalid docs.
• This helps the compaction task to handle arbitrary temporal distribution of invalid docs.

Notable Improvements

• Minions can now download segments from servers when deepstore copy is missing. This feature is enabled via a cluster level config allowDownloadFromServer #12960 #13247.
• Added support for TLS Port in Minions #12943.
• New metrics added for Minions to track segment/record processing information #12710.

Bug Fixes

• Minions can now handle invalid instance tags in Task Configs gracefully. Prior to this change, Minions would be stuck in IN_PROGRESS state until task timeout #13092.
• Fix bug to return validDocIDsMetadata from all servers #12431.
• Upsert compaction doesn't retain maxLength information and trims string fields #13157.

Upsert Improvements

Features and Improvements

Consistent Table View for Upsert Tables #12976

• Adds different modes of consistency guarantees for Upsert tables.
• Adds a new UpsertConfig called consistencyMode which can be set to NONE, SYNC, SNAPSHOT.
• SYNC is optimized for data freshness but can lead to elevated query latencies and is best for low-qps use-cases. In this mode, the ingestion threads will take a WLock when updating validDocID bitmaps.
• SNAPSHOT mode can handle high-qps/high-ingestion use-cases by getting the list of valid docs from a snapshot of validDocID. The snapshot can be refreshed every few seconds and the tolerance can be set via a query option upsertViewFreshnessMs.

Pluggable Partial Upsert Merger #11983

• Partial Upsert merges the old record and the new incoming record to generate the final ingested record.
• Pinot now allows users to customize how this merge of an old row and the new row is computed.
• This allows a column value in the new row to be an arbitrary function of the old and the new row.

Support for Uploading Externally Partitioned Segments for Upsert Backfill 13107

• Segments uploaded for Upsert Backfill can now explicitly specify the Kafka partition they belong to.
• This enables backfilling an Upsert table where the externally generated segments are partitioned using an arbitrary hash function on an arbitrary primary key.

Misc Improvements and Bug Fixes

• Fixed a Bug in Handling Equal Comparison Column Values in Upsert, which could lead to data inconsistency (#12395)
• Upsert snapshot will now snapshot only those segments which have updates. #13285.

Notable Features

JSON Support Improvements

• JSON Index can now be used for evaluating Regex and Range Predicates. #12568
• jsonExtractIndex now supports contextual array filters. #12683 #12531.
• JSON column type now supports filter predicates like =, !=, IN and NOT IN. This is convenient for scenarios where the JSON values are very small. #13283.
• JSON_MATCH now supports exclusive predicates correctly. For instance, you can use predicates such as JSON_MATCH(person, '"$.addresses[*].country" != ''us''' to find all people who have at least one address that is not in the US. #13139.
• jsonExtractIndex supports extracting Multi-Value JSON Fields, and also supports providing any default value when the key doesn't exist. #12748.
• Added isJson UDF which increases your options to handle invalid JSONs. This can be used in queries and for filtering invalid json column values in ingestion. #12603.
• Fix ArrayIndexOutOfBoundsException in jsonExtractIndex. #13479.

Lucene and Text Search Improvements

• Improved Segment Build Time for Lucene Text Index by 40-60%. This improvement is realized when a consuming segment commits and changes to an ImmutableSegment. This significantly helps in lowering ingestion lag at commit time due to a large text index #12744 #13094 #13050.
• Phrase Search can run 3x faster when the Lucene Index Config enablePrefixSuffixMatchingInPhraseQueries is set to true. This is achieved by rewriting phrase search query to a wildcard and prefix matching query #12680.
• Fixed bug in TextMatchFilterOptimizer that was not applying precedence to the filter expressions properly, which could lead to incorrect results. #13009.
• Fixed bug in handling NOT text_match which could have returned incorrect results. #12372.
• Added SchemaConformingTranformerV2 to enhance text search abilities. #12788.
• Added metrics to track Lucene NRT Refresh Delay #13307.
• Switched to NRTCachingDirectory for Realtime segments and prevented duplicates in the Realtime Lucene Index to avoid IndexOutOfBounds query time exceptions. #13308.
• Lucene Version is upgraded to 9.11.1. #13505.

New Funnel Functions #13176 #13231 #13228

• Added funnelMaxStep function which can be used to calculate max funnel steps for a given sliding window .
• Added funnelCompleteCount to calculate the number of completed funnels, and funnelMatchStep to get the funnel match array.

Support for Interning for OnHeapByteDictionary #12342

• This can reduce the heap usage of a dictionary encoded byte column, for a certain distribution of duplicate values. See #12223 for details.

Column Major Builder On By Default for New Tables #12770

• Prior to this feature, on a segment commit, Pinot would convert all the columnar data from the Mutable Segment to row-major, and then re-build column major Immutable Segments.
• This feature skips the row-major conversion and is expected to be both space and time efficient.
• It can help lower ingestion lag from segment commits, especially helpful when your segments are large.

Support for SQL Formatting in Query Editor #11725

• You can now prettify SQL right in the Controller UI!

Hash Function for UUID Primary Keys #12538

• Added a new lossless hash-function for Upsert Primary Keys optimized for UUIDs.
• The hash function can reduce Old Gen by up to 30%.
• It maps a UUID to a 16 byte array, vs encoding it in a UTF string which would take 36 bytes.

Column Level Index Skip Query Option #12414

• Convenient for debugging impact of indexes on query performance or results.
• You can add the skipIndexes option to your query to skip any number of indexes. e.g. SET skipIndexes=inverted,range;

New UDFs and Scalar Functions

• New GeoHash functions: encodeGeoHash, decodeGeoHash, decodeGeoHashLatitude and decodeGeoHashLongitude.
• dateBin can be used to align a timestamp to the nearest time bucket.
• prefixes, suffixes and uniqueNgrams UDFs for generating all respective string subsequences from a string input. #12392.
• Added isJson UDF which increases your options to handle invalid JSONs. This can be used in queries and for filtering invalid json column values in ingestion. #12603.
• splitPart UDF has minor improvements. #12437.

CLP Compression Codec in Forward Indexes #12504

• CLP is a compressed log processor which has really high compression ratio for certain log types.
• To enable this, you can set the compressionCodec in the fieldConfigList of the column you want to target.

Misc. Improvements

• Enable segment preloading at partition level #12451.

• Use Temurin instead of AdoptOpenJdk #12533

• Adding record reader config/context param to record transformer #12520

• Removing legacy commons-lang dependency #13480

• 12508: Feature add segment rows flush config #12681

• ADSS Race Condition and update to client error codes #13104

• Add ExceptionMapper to convert Exception to Response Object for Broker REST API's #13292

• Add FunnelMaxStepAggregationFunction and FunnelCompleteCountAggregationFunction #13231

• Add GZIP Compression Codec (#11434) #12668

• Add PodDisruptionBudgets to the Pinot Helm chart #13153

• Add Postgres compliant name aliasing for String Functions. #12795

• Add SchemaConformingTransformerV2 to enhance text search abilities #12788

• Add a benchmark to measure multi-stage block serde cost #13336

• Add a plan version field to QueryRequest Protobuf Message #13267

• Add a post-validator visitor that verifies there are no cast to bytes #12475

• Add a safe version of CLStaticHttpHandler that disallows path traversal. #13124

• Add ability to track filtered messages offset #12602

• Add back 'numRowsResultSet' to BrokerResponse, and retain it when result table id hidden #13198

• Add back profile for shade #12979

• Add back some exclude deps from hadoop-mapreduce-client-core #12638

• Add backward compatibility regression test suite for multi-stage query engine #13193

• Add base class for custom object accumulator #12685

• Add clickstream example table for funnel analysis #13379

• Add config option for timezone #12386

• Add config to skip record ingestion on string column length exceeding configured max schema length #13103

• Add controller API to get allLiveInstances #12498

• Add isJson UDF #12603

• Add list of collaborators to asf.yaml #13346

• Add locking logic to get consistent table view for upsert tables #12976

• Add metric to track number of segments missed in upsert-snapshot #12581

• Add metrics for SEGMENTS_WITH_LESS_REPLICAS monitoring #12336

• Add mode to allow adding dummy events for non-matching steps #13382

• Add offset based lag metrics #13298

• Add protobuf codegen decoder #12980

• Add retry policy to wait for job id to persist during rebalancing #13372

• Add round-robin logic during downloadSegmentFromPeer #12353

• Add schema as input to the decoder. #12981

• Add splitPartWithLimit and splitPartFromEnd UDFs #12437

• Add support for creating raw derived columns during segment reload #13037

• Add support for raw JSON filter predicates #13283

• Add the possibility of configuring ForwardIndexes with compressionCodec #12218

• Add upsert-snapshot timer metric #12383

• Add validation check for forward index disabled if it's a REALTIME table #12838

• Added PR compatability test against release 1.1.0 #12921

• Added kafka partition number to metadata. #13447

• Added pinot-error-code header in query response #12338

• Added tests for additional data types in SegmentPreProcessorTest.java #12755

• Adding a cluster config to enable instance pool and replica group configuration in table config #13131

• Adding batch api support for WindowFunction #12993

• Adding bytes string data type integration tests #12387

• Adding registerExtraComponents to allow registering additional components in various services #13465

• Adding support of insecure TLS #12416

• Adding support to insecure TLS when creating SSLFactory #12425

• Adds AGGREGATE_CASE_TO_FILTER rule #12643

• Adds per-column, query-time index skip option #12414

• Allow Aggregations in Case Expressions #12613

• Allow PintoHelixResourceManager subclasses to be used in the controller starter by providing an overridable PinotHelixResouceManager object creator function #13495

• Allow RequestContext to consider http-headers case-insensitivity #13169

• Allow Server throttling just before executing queries on server to allow max CPU and disk utilization #12930

• Allow all raw index config in star-tree index #13225

• Allow apply both environment variables and system properties to user and table configs, Environment variables take precedence over system properties #13011

• Allow configurable queryWorkerThreads in Pinot server side GrpcQueryServer #13404

• Allow dynamically setting the log level even for loggers that aren't already explicitly configured #13156

• Allow passing custom record reader to be inited/closed in SegmentProcessorFramework #12529

• Allow passing database context through database http header #12417

• Allow stop to interrupt the consumer thread and safely release the resource #13418

• Allow user configurable regex library for queries #13005

• Allow using 'serverReturnFinalResult' to optimize server partitioned table #13208

• Assign default value to newly added derived column upon reload #12648

• Avoid port conflict in integration tests #13390

• Better handling of null tableNames #12654

• CLP as a compressionCodec #12504

• Change helm app version to 1.0.0 for Apache Pinot latest release version #12436

• Clean Google Dependencies #13297

• Clean up BrokerRequestHandler and BrokerResponse #13179

• Clean up arbitrary sleep in /GrpcBrokerClusterIntegrationTest #12379

• Cleaning up vector index comments and exceptions #13150

• Cleanup HTTP components dependencies and upgrade Thrift #12905

• Cleanup Javax and Jakarta dependencies #12760

• Cleanup deprecated query options #13040

• Cleanup the consumer interfaces and legacy code #12697

• Cleanup unnecessary dependencies under pinot-s3 #12904

• Cleanup unused aggregate internal hint #13295

• Consistency in API response for live broker #12201

• Consolidate bouncycastle libraries #12706

• Consolidate nimbus-jose-jwt version to 9.37.3 #12609

• ControllerRequestClient accepts headers. Useful for authN tests #13481

• Custom configuration property reader for segment metadata files #12440

• Delete database API #12765

• Deprecate PinotHelixResourceManager#getAllTables() in favour of getAllTables(String databaseName) #12782

• Detect expired messages in Kafka. Log and set a gauge. #12608

• Do not hard code resource class in BaseClusterIntegrationTest #13400

• Do not pause ingestion when upsert snapshot flow errors out #13257

• Don't drop original field during flatten #13490

• Don't enforce -realTimeInstanceCount and -offlineInstanceCount options when creating broker tenants #13236

• Egalpin/skip indexes minor changes #12514

• Emit Metrics for Broker Adaptive Server Selector type #12482

• Emit table size related metrics only in lead controller #12747

• Enable complexType handling in SegmentProcessFramework #12942

• Enable more integration tests to run on the v2 multi-stage query engine #13467

• Enabling avroParquet to read Int96 as bytes #12484

• Enhance Kinesis consumer #12806

• Enhance Parquet Test #13082

• Enhance ProtoSerializationUtils to handle class move #12946

• Enhance Pulsar consumer #12812

• Enhance PulsarConsumerTest #12948

• Enhance commit threshold to accept size threshold without setting rows to 0 #12684

• Enhance json index to support regexp and range predicate evaluation #12568

• Enhancement: Sketch value aggregator performance #13020

• Ensure FieldConfig.getEncodingType() is never null #12430

• Ensure all the lists used in PinotQuery are ArrayList #13017

• Ensure brokerId and requestId are always set in BrokerResponse #13200

• Enter segment preloading at partition level #12451

• Exclude dimensions from star-tree index stored type check #13355

• Expose more helper API in TableDataManager #13147

• Extend compatibility verifier operation timeout from 1m to 2m to reduce flakiness #13338

• Extract json individual array elements from json index for the transform function jsonExtractIndex #12466

• Fetch query quota capacity utilization rate metric in a callback function #12767

• First with time #12235

• GitHub Actions checkout v4 #12550

• Gzip compression, ensure uncompressed size can be calculated from compressed buffer #12802

• Handle errors gracefully during multi-stage stats collection in the broker #13496

• Handle shaded classes in all methods of kafka factory #13087

• Hash Function for UUID Primary Keys #12538

• Ignore case when checking for Direct Memory OOM #12657

• Improve Retention Manager Segment Lineage Clean Up #13232

• Improve error message for max rows in join limit breach #13394

• Improve exception logging when we fail to index / transform message #12594

• Improve logging in range index handler for index updates #13381

• Improve upsert compaction threshold validations #13424

• Improve warn logs for requesting validDocID snapshots #13280

• Improved metrics for server grpc query #13177

• Improved null check for varargs #12673

• Improved segment build time for Lucene text index realtime to offline conversion #12744

• In ClusterTest, make start port higher to avoid potential conflict with Kafka #13402

• Introduce PinotLogicalAggregate and remove internal hint #13291

• Introduce retries while creating stream message decoder for more robustness #13036

• Isolate bad server configs during broker startup phase #12931

• Issue #12367 #12922

• Json extract index filter support #12683

• Json extract index mv #12532

• Keep get tables API with and without database #12804

• Lint failure #12294

• Logging a warn message instead of throwing exception #12546

• Made the error message around dimension table size clearer #13163

• Make Helix state transition handling idempotent #12886

• Make KafkaConsumerFactory method less restrictive to avoid incompatibility #12815

• Make task manager APIs database aware #12766

• Metric for count of tables configured with various tier backends #12940

• Metric for upsert tables count #12505

• Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions #12522

• Minmaxrange null #12252

• Modify consumingSegmentsInfo endpoint to indicate how many servers failed #12523

• Move offset validation logic to consumer classes #13015

• Move package org.apache.calcite to org.apache.pinot.calcite #12837

• Move resolveComparisonTies from addOrReplaceSegment to base class #13396

• Move some mispositioned tests under pinot-core #12884

• Move wildfly-openssl dependency management to root pom #12597

• Moving deleteSegment call from POST to DELETE call #12663

• Optimize unnecessary extra array allocation and conversion for raw derived column during segment reload #13115

• Pass explicit TypeRef when evaluating MV jsonPath #12524

• Percentile operations supporting null #12271

• Prepare for next development iteration #12530

• Propagate Disable User Agent Config to Http Client #12479

• Properly handle complex type transformer in segment processor framework #13258

• Properly return response if SegmentCompletion is aborted #13206

• Publish helm 0.2.8 #12465

• Publish helm 0.2.9 #13230

• Pull janino dependency to root pom #12724

• Pull pulsar version definitaion into root POM #13002

• Query response opt #13420

• Re-enable the Spotless plugin for Java 21 #12992

• Readme - How to setup Pinot UI for development #12408

• Record enricher #12243

• Refactor PinotTaskManager class #12964

• Refactored CommonsConfigurationUtils for loading properties configuration. #13201

• Refactored compatibility-verifier module #13359

• Refactoring removeSegment flow in upsert #13449

• Refine PeerServerSegmentFinder #12933

• Refine SegmentFetcherFactory #12936

• Replace custom fmpp plugin with fmpp-maven-plugin #12737

• Reposition query submission spot for adaptive server selection #13327

• Reset controller port when stopping the controller in ControllerTest #13399

• Rest Endpoint to Create ZNode #12497

• Return clear error message when no common broker found for multi-stage query with tables from different tenants #13235

• Returning tables names failing authorization in Exception of Multi State Engine Queries #13195

• Revert " Adding record reader config/context param to record transformer (#12520)" #12526

• Revert "Using local copy of segment instead of downloading from remote (#12863)" #13114

• Short circuit SubPlanFragmenter because we don't support multiple sub-plans yet #13306

• Simplify Google dependencies by importing BOM #12456

• Specify version for commons-validator #12935

• Support NOT in StarTree Index #12988

• Support empty strings as json nodes^ #12555

• Supporting human-readable format when configuring broker response size #12510

• Use ArrayList instead of LinkedList in SortOperator #12783

• Use a two server setup for multi-stage query engine backward compatibility regression test suite #13371

• Use more efficient variants of URLEncoder::encode and URLDecoder::decode #13030

• Use parameterized log messages instead of string concatenation #13145

• Use separate action for /tasks/scheduler/jobDetails API #13054

• Use try-with-resources to close file walk stream in LocalPinotFS #13029

• Using local copy of segment instead of downloading from remote #12863

• [Adaptive Server Selector] Add metrics for Stats Manager Queue Size #12340

• [Cleanup] Move classes in pinot-common to the correct package #13478

• [Feature] Add Support for SQL Formatting in Query Editor #11725

• [HELM]: Added additional probes options and startup probe. #13165

• [HELM]: Added checksum config annotation in stateful set for broker, controller and server #13059

• [HELM]: Added namespace support in K8s deployment. #13380

• [HELM]: zookeeper chart upgrade to version 13.2.0 #13083

• [Minor] Add Nullable annotation to HttpHeaders in BrokerRequestHandler #12816

• [Minor] Small refactor of raw index creator constructor to be more clear #13093

• [Multi-stage] Clean up RelNode to Operator handling #13325

• [null-aggr] Add null handling support in mode aggregation #12227

• [partial-upsert] configure early release of _partitionGroupConsumerSemaphore in RealtimeSegmentDataManager #13256

• [spark-connector] Add option to fail read when there are invalid segments #13080

• add Netty arm64 dependencies #12493

• add Netty unit test #12486

• add SegmentContext to collect validDocIds bitmaps for many segments together #12694

• add skipUnavailableServers query option #13387

• add insecure mode when Pinot uses TLS connections #12525

• add instrumentation to json index getMatchingFlattenedDocsMap() #13164

• add jmx to promethues metric exporting rule for realtimeRowsFiltered #12759

• add metrics for IdeaState update #13266

• add some metrics for upsert table preloading #12722

• add some tests on jsonPathString #12954

• add test cases in RequestUtilsTest #12557

• add unit test for JsonAsyncHttpPinotClientTransport #12633

• add unit test for QueryServer #12599

• add unit test for ServerChannels #12616

• add unit test for StringFunctions encodeUrl #13391

• add unit tests for pinot-jdbc-client #13137

• add url assertion to SegmentCompletionProtocolTest #13373

• adjust the llc partition consuming metric reporting logic #12627

• allow passing null http headers object to translateTableName #12764

• allow to set segment when use SegmentProcessorFramework #13341

• auto renew jvm default sslconext when it's loaded from files #12462

• avoid useless intermediate byte array allocation for VarChunkV4Reader's getStringMV #12978

• aws sdk 2.25.3 #12562

• build-helper-maven-plugin 3.5.0 #12548

• cache ssl contexts and reuse them #12404

• clean up jetbrain nullable annotation #13427

• cleanup: maven no transfer progress #12444

• close JDBC connections #12494

• do not fail on duplicate relaxed vars (#13214)z

• dropwizard metrics 4.2.25 #12600

• dynamic chunk sizing for v4 raw forward index #12945

• enable Netty leak detection #12483

• enable parallel Maven in pinot linter script #12751

• ensure inverse And/OrFilterOperator implementations match the query #13199

• exclude .mvn directory from source assembly #12558

• extend CompactedPinotSegmentRecordReader so that it can skip deleteRecord #13352

• get startTime outside the executor task to avoid flaky time checks #13250

• handle absent segments so that catchup checker doesn't get stuck on them #12883

• handle overflow for MutableOffHeapByteArrayStore buffer starting size #13215

• handle segments not tracked by partition mgr and add skipUpsertView query option #13415

• handle table name translation on missed api resources #12792

• hash4j version upgrade to 0.17.0 #12968

• including the underlying exception in the logging output #13248

• int96 parity with native parquet reader #12496

• jsonExtractIndex support array of default values #12748

• log the log rate limiter rate for dropped broker logs #13041

• make http listener ssl config swappable #12455

• make reflection calls compatible with 0.9.11 [#12958](https://github.com/apache/

• maven: no transfer progress #12528

• missed to delete the temp dir #12637

• move shouldReplaceOnComparisonTie to base class to be more reusable #13353

• reduce Java enum .values() usage in TimerContext #12579

• reduce logging for SpecialValueTransformer #12970

• reduce regex pattern compilation in Pinot jdbc #13138

• refactor TlsUtils class #12515

• refine when to registerSegment while doing addSegment and replaceSegment for upsert tables for better data consistency #12709

• reformat AdminConsoleIntegrationTest.java #12552

• reformat ClusterTest.java #12531

• release segment mgrs more reliably #13216

• replaced getServer with getServers #12545

• report rebalance job status for the early returns like noops #13281

• require noDictionaryColumns with aggregationConfigs #12464

• share the same table config object #12463

• track segments for snapshotting even if they lost all comparisons #13388

• untrack the segment out of TTL #12449

• update ControllerJobType from enum to string #12518

• update RewriterConstants so that expr min max would not collide with columns start with "parent" #13357

• update access control check error handling to catch throwable and log errors #13209

Bug Fixes

• Use gte(lte) to replace between() which has a bug #12595
• Fix the ConcurrentModificationException for And/Or DocIdSet #12611
• Upgrade RoaringBitmap to 1.0.5 to pick up the fix for RangeBitmap.between() #12604
• bugfix: do not move src ByteBuffer position for LZ4 length prefixed decompress #12539
• Bug Fix createDictionaryForColumn does not take into account inverted index #13048
• fix Cluster Manager error #12632
• fix for quick start Cluster Manager issue #12610
• Adding config for having suffix for client ID for realtime consumer #13168
• Addressed comments and fixed tests from pull request 12389. /uptime and /start-time endpoints working all components #12512
• Bigfix. Added missing paramName #13060
• Bug fix: Do not ignore scheme property #12332
• Bug fix: Handle missing shade config overwrites for Kafka #13437
• BugFix: Fix merge result from more than one server #12778
• Bugfix. Allow tenant rebalance with downtime as true #13246
• Bugfix. Avoid passing null table name input to translation util #12726
• Bugfix. Correct wrong method call from scheduleTask() to scheduleTaskForDatabase() #12791
• Bugfix. Maintain literal data type during function evaluation #12607
• Cleanup: Fix grammar in error message, also improve readability. #13451
• Fix Bug in Handling Equal Comparison Column Values in Upsert #12395
• Fix ColumnMinMaxValueGenerator #12502
• Fix JavaEE related dependencies #13058
• Fix Logging Location for CPU-Based Query Killing #13318
• Fix PulsarUtils to not share buffer #12671
• Fix URI construction so that AddSchema command line tool works when override flag is set to true #13320
• Fix [Type]ArrayList elements() method usage #13354
• Fix a typo when calculating query freshness #12947
• Fix an overflow in PinotDataBuffer.readFrom #13152
• Fix bug in logging in UpsertCompaction task #12419
• Fix bug to return validDocIDsMetadata from all servers #12431
• Fix connection issues if using JDBC and Hikari (#12267) #12411
• Fix controller host / port / protocol CLI option description for admin commands #13237
• Fix environment variables not applied when creating table #12560
• Fix error message for insufficient number of untagged brokers during tenant creation #13234
• Fix few metric rules which were affected by the database prefix handling #13290
• Fix file handle leaks in Pinot Driver (#12263) #12356
• Fix flakiness of ControllerPeriodicTasksIntegrationTest #13337
• Fix issue with startree index metadata loading for columns with '__' in name #12554
• Fix metric rule pattern regex #12856
• Fix pinot-parquet NoClassFound issue #12615
• Fix segment size check in OfflineClusterIntegrationTest #13389
• Fix some resource leak in tests #12794
• Fix the NPE from IS update metrics #13313
• Fix the NPE when metadataTTL is enabled without delete column #13262
• Fix the ServletConfig loading issue with swagger. #13122
• Fix the issue that map flatten shouldn't remove the map field from the record #13243
• Fix the race condition for H3InclusionIndexFilterOperator #12487
• Fix the time segment pruner on TIMESTAMP data type #12789
• Fix time stats in SegmentIndexCreationDriverImpl #13429
• Fixed infer logical type name from avro union schema #13224
• Fixing instance type to resolve #12677 and #12678
• Helm: bug fix for chart rendering issue. #13264
• Try to amend kafka common package with pinot shaded package prefix #13056
• Update getValidDocIdsMetadataFromServer to make call in batches to servers and other bug fixes #13314
• Upgrade com.microsoft.azure:msal4j from 1.3.5 to 1.3.10 for CVE fixing #12580
• [bugfix] Handling null value for kafka client id suffix #13279
• bugfix: fixing jdbc client sql feature not supported exception #12480
• bugfix: re-add support for not text_match #12372
• bugfix: reduce enum array allocation in QueryLogger #12478
• bugfix: use consumerDir during lucene realtime segment conversion #13094
• cleanup: fix apache rat violation #12476
• fix GuavaRateLimiter acquire method #12500
• fix fieldsToRead class not in decoder #13186
• fix flakey test, avoid early finalization #13095
• fix merging null multi value in partial upsert #13031
• fix race condition in ScalingThreadPoolExecutor #13360
• fix shared buffer, tests #12587
• fix(build): update node version to 16 #12924
• fixing CVE critical issues by resolving kerby/jline and wildfly libraries #12566
• fixing pinot-adls high severity CVEs #12571
• fixing swagger setup using localhost as host name #13254
• swagger-ui upgrade to 5.15.0 Fixes #12908
• upgrade jettison version to fix CVE #12567