0.2.0

The 0.2.0 release is the first release after the initial one and includes several improvements, reported following.

New Features and Bug Fixes

• Added support for Kafka 2.0

• Table rebalancer now supports a minimum number of serving replicas during rebalance

• Added support for UDF in filter predicates and selection

• Added support to use hex string as the representation of byte array for queries (see PR #4041)

• Added support for parquet reader (see PR #3852)

• Introduced interface stability and audience annotations (see PR #4063)

• Refactor HelixBrokerStarter to separate constructor and start() - backwards incompatible (see PR #4100)

• Admin tool for listing segments with invalid intervals for offline tables

• Migrated to log4j2 (see PR #4139)

• Added simple avro msg decoder

• Added support for passing headers in Pinot client

• Table rebalancer now supports a minimum number of serving replicas during rebalance

• Support transform functions with AVG aggregation function (see PR #4557)

• Configurations additions/changes

  • Allow customized metrics prefix (see PR #4392)

  • Controller.enable.batch.message.mode to false by default (see PR #3928)

  • RetentionManager and OfflineSegmentIntervalChecker initial delays configurable (see PR #3946)

  • Config to control kafka fetcher size and increase default (see PR #3869)

  • Added a percent threshold to consider startup of services (see PR #4011)

  • Make SingleConnectionBrokerRequestHandler as default (see PR #4048)

  • Always enable default column feature, remove the configuration (see PR #4074)

  • Remove redundant default broker configurations (see PR #4106)

  • Removed some config keys in server(see PR #4222)

  • Add config to disable HLC realtime segment (see PR #4235)

  • Make RetentionManager and OfflineSegmentIntervalChecker initial delays configurable (see PR #3946)

  • The following config variables are deprecated and will be removed in the next release:

    • pinot.broker.requestHandlerType will be removed, in favor of using the "singleConnection" broker request handler. If you have set this configuration, remove it and use the default type ("singleConnection") for broker request handler.

Work in Progress

• We are in the process of separating Helix and Pinot controllers, so that administrators can have the option of running independent Helix controllers and Pinot controllers.

• We are in the process of moving towards supporting SQL query format and results.

• We are in the process of separating instance and segment assignment using instance pools to optimize the number of Helix state transitions in Pinot clusters with thousands of tables.

Other Notes

• Task management does not work correctly in this release, due to bugs in Helix. We will upgrade to Helix 0.9.2 (or later) version to get this fixed.

• You must upgrade to this release before moving onto newer versions of Pinot release. The protocol between Pinot-broker and Pinot-server has been changed and this release has the code to retain compatibility moving forward. Skipping this release may (depending on your environment) cause query errors if brokers are upgraded and servers are in the process of being upgraded.

• As always, we recommend that you upgrade controllers first, and then brokers and lastly the servers in order to have zero downtime in production clusters.

• Pull Request #4100 introduces a backwards incompatible change to Pinot broker. If you use the Java constructor on HelixBrokerStarter class, then you will face a compilation error with this version. You will need to construct the object and call start() method in order to start the broker.

• Pull Request #4139 introduces a backwards incompatible change for log4j configuration. If you used a custom log4j configuration (log4j.xml), you need to write a new log4j2 configuration (log4j2.xml). In addition, you may need to change the arguments on the command line to start Pinot components.

  If you used Pinot-admin command to start Pinot components, you don't need any change. If you used your own commands to start pinot components, you will need to pass the new log4j2 config as a jvm parameter (i.e. substitute -Dlog4j.configuration or -Dlog4j.configurationFile argument with -Dlog4j2.configurationFile=log4j2.xml).