-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Custom configuration property reader for segment metadata files #12440
Custom configuration property reader for segment metadata files #12440
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #12440 +/- ##
============================================
+ Coverage 61.75% 62.16% +0.41%
+ Complexity 207 198 -9
============================================
Files 2436 2540 +104
Lines 133233 139455 +6222
Branches 20636 21554 +918
============================================
+ Hits 82274 86697 +4423
- Misses 44911 46265 +1354
- Partials 6048 6493 +445
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
c429ffb
to
3d68b96
Compare
pinot-spi/src/main/java/org/apache/pinot/spi/env/SegmentMetadataPropertyReader.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In order to keep backward compatibility, we will need to introduce versioning into the segment metadata file. Imagine we use the new property reader to read a old metadata file, it will fail to unescape the values, thus reading the wrong values.
We should introduce a new segment metadata version, and ensure there is no escaping in the new version, then we can use this new property reader for the new version. For the old version, we'll need to fall back to the old property reader.
a67b71a
to
15601b7
Compare
pinot-spi/src/main/java/org/apache/pinot/spi/env/SegmentMetadataPropertyWriter.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/SegmentMetadataPropertyReader.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we add a test to ensure it can read the old format properly? We can put a old format file into the resource folder, and use the new reader to read it
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/V1Constants.java
Outdated
Show resolved
Hide resolved
157b780
to
86a2fff
Compare
Added the required tests. |
@Jackie-Jiang / @klsince, any further comments on the implementation side. I am going to start work on it's usage based on the cluster configuration or table configuration. |
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/SegmentMetadataPropertyWriter.java
Outdated
Show resolved
Hide resolved
c6ae01b
to
1bc252a
Compare
pinot-spi/src/main/java/org/apache/pinot/spi/env/PropertyIOFactoryKind.java
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/SegmentMetadataPropertyReader.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/test/resources/segment-metadata-with-version-header.properties
Outdated
Show resolved
Hide resolved
d8104f6
to
2b0357a
Compare
Based on the offline discussion, we decided to have pair of separate custom segment metadata reader/writer. We will decide to use custom reader/writer based on the version of the segment metadata. |
2b0357a
to
0f1dec9
Compare
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/SegmentMetadataPropertyWriter.java
Outdated
Show resolved
Hide resolved
0f1dec9
to
7afcc03
Compare
pinot-spi/src/test/resources/segment-metadata-with-version-header.properties
Outdated
Show resolved
Hide resolved
pinot-spi/src/test/resources/segment-metadata-without-version-header.properties
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
4a6b4c3
to
18e562b
Compare
@Jackie-Jiang / @klsince, please review it once more. Thanks |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM overall, w/ a few more questions and nits.
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/test/java/org/apache/pinot/spi/env/SegmentMetadataPropertyConfigTest.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/test/java/org/apache/pinot/spi/env/SegmentMetadataPropertyConfigTest.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/test/java/org/apache/pinot/spi/env/SegmentMetadataPropertyConfigTest.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/test/java/org/apache/pinot/spi/env/SegmentMetadataPropertyConfigTest.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/test/java/org/apache/pinot/spi/env/SegmentMetadataPropertyConfigTest.java
Outdated
Show resolved
Hide resolved
18e562b
to
5d89c55
Compare
…arator and added unit test case for it.
0552e9e
to
3bdbae3
Compare
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/VersionedPropertyReader.java
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/VersionedPropertyReader.java
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java
Outdated
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/env/VersionedPropertyReader.java
Outdated
Show resolved
Hide resolved
…he#12440) * Added versioned configuration reader/writer * Added methods to read/write segment metadata with versioned reader/writer
This is the first set of changes for the issue.
In this PR, we introduce a new custom property reader for the segment metadata files. This custom property reader is based on the commons-configuration2 custom property reader/writer(here is the link). We are introducing this custom property reader to avoid unescaping operation happen for the key in default implementation.
cc: @klsince @Jackie-Jiang @itschrispeck