Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up BrokerRequestHandler and BrokerResponse #13179

Merged
merged 1 commit into from
May 19, 2024

Conversation

Jackie-Jiang
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang commented May 18, 2024

Clean up BrokerRequestHandler:

  • Extracted single-stage and multi-stage common methods into BaseBrokerRequestHandler
  • Created BaseSingleStageRequestHandler for the single-stage methods
  • Simplified the response stats handling and ensure they are consistent

Clean up BrokerResponse:

  • Remove getters from BrokerResponse
  • Cleanup unnecessary methods from BrokerResponseNative and BrokerResponseNativeV2
  • Ensure the stats ordering is consistent

Refactor BrokerQueryEventListenerFactory to be easier to use and more efficient

Incompatible

  • Involves interface signature change on BrokerResponse and BrokerRequestHandler, which might cause compile failure, but no backward incompatibility introduced
  • It also changed the stat enum order for leaf operator introduce in Multi stage stats #12704 to be more natural, but that one is merged quite recently, so not maintain backward compatibility between these 2 PRs

@Jackie-Jiang Jackie-Jiang added incompatible Indicate PR that introduces backward incompatibility bugfix cleanup labels May 18, 2024
@codecov-commenter
Copy link

codecov-commenter commented May 18, 2024

Codecov Report

Attention: Patch coverage is 46.64634% with 700 lines in your changes are missing coverage. Please review.

Project coverage is 62.21%. Comparing base (59551e4) to head (86c9ed3).
Report is 463 commits behind head on master.

Files Patch % Lines
...sthandler/BaseSingleStageBrokerRequestHandler.java 41.67% 457 Missing and 107 partials ⚠️
...common/response/broker/BrokerResponseNativeV2.java 21.95% 32 Missing ⚠️
...istener/query/BrokerQueryEventListenerFactory.java 0.00% 29 Missing ⚠️
...requesthandler/MultiStageBrokerRequestHandler.java 33.33% 18 Missing ⚠️
...roker/requesthandler/BaseBrokerRequestHandler.java 69.56% 10 Missing and 4 partials ⚠️
.../pinot/query/service/dispatch/QueryDispatcher.java 80.48% 4 Missing and 4 partials ⚠️
...thandler/SingleConnectionBrokerRequestHandler.java 33.33% 6 Missing ⚠️
...pinot/broker/api/resources/PinotClientRequest.java 42.85% 4 Missing ⚠️
...roker/requesthandler/GrpcBrokerRequestHandler.java 0.00% 4 Missing ⚠️
...t/common/response/broker/BrokerResponseNative.java 94.02% 3 Missing and 1 partial ⚠️
... and 7 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #13179      +/-   ##
============================================
+ Coverage     61.75%   62.21%   +0.46%     
+ Complexity      207      198       -9     
============================================
  Files          2436     2527      +91     
  Lines        133233   138313    +5080     
  Branches      20636    21394     +758     
============================================
+ Hits          82274    86050    +3776     
- Misses        44911    45833     +922     
- Partials       6048     6430     +382     
Flag Coverage Δ
custom-integration1 <0.01% <0.00%> (-0.01%) ⬇️
integration <0.01% <0.00%> (-0.01%) ⬇️
integration1 <0.01% <0.00%> (-0.01%) ⬇️
integration2 0.00% <0.00%> (ø)
java-11 62.19% <46.64%> (+0.48%) ⬆️
java-21 35.08% <10.28%> (-26.55%) ⬇️
skip-bytebuffers-false 62.19% <46.64%> (+0.44%) ⬆️
skip-bytebuffers-true 35.08% <10.28%> (+7.35%) ⬆️
temurin 62.21% <46.64%> (+0.46%) ⬆️
unittests 62.20% <46.64%> (+0.46%) ⬆️
unittests1 46.73% <60.81%> (-0.16%) ⬇️
unittests2 27.92% <36.35%> (+0.19%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Jackie-Jiang Jackie-Jiang force-pushed the broker_response branch 2 times, most recently from aa1e0a7 to ea983ce Compare May 18, 2024 22:06
Copy link
Contributor

@xiangfu0 xiangfu0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Jackie-Jiang Jackie-Jiang merged commit e71d1c6 into apache:master May 19, 2024
19 checks passed
@Jackie-Jiang Jackie-Jiang deleted the broker_response branch May 19, 2024 00:48
Comment on lines 50 to +54
/**
* Set the number of servers responded to the broker.
*
* @param numServersResponded number of servers responded.
* Sets the result table. We expose this method to allow modifying the results on the client side, e.g. hiding the
* results and only showing the stats.
*/
void setNumServersResponded(int numServersResponded);
void setResultTable(@Nullable ResultTable resultTable);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be the only mutable method in this interface. Wouldn't be better to implement the case shown in the example using a delegate/decorate pattern? Basically having an implementation that delegates all methods but getResultTable and getNumRowsResultSet to a another BrokerResponse stored as attribute

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel that is a little bit overkilling if the only benefit is getting rid of this setter. I'm not a big fan of this only setter as well, but I feel it is much simpler and easier to understand if we want to allow client to override it

gortiz pushed a commit to gortiz/pinot that referenced this pull request Jun 14, 2024
mcvsubbu added a commit to mcvsubbu/incubator-pinot that referenced this pull request Jul 18, 2024
As of PR apache#13179, the query response time metrics started to get extremely high for
all tables. The root cause tured out to be a missed initialization in which
the query arrival time was set to 0 by default.

The refactor broke the contract that the broker request handler always returns the
time taken to handle the request.

Added code to initialize the arrival time.

Since the refactor attempted to include client pre-processing time, added this code
under the condition that the value is not already set.

Ideally, callers of the method should add on their own pre and post-processing time
to the time returned by the handleRequest() method.
mcvsubbu added a commit that referenced this pull request Jul 24, 2024
* Fix query response time metric

As of PR #13179, the query response time metrics started to get extremely high for
all tables. The root cause tured out to be a missed initialization in which
the query arrival time was set to 0 by default.

The refactor broke the contract that the broker request handler always returns the
time taken to handle the request.

Added code to initialize the arrival time.

Since the refactor attempted to include client pre-processing time, added this code
under the condition that the value is not already set.

Ideally, callers of the method should add on their own pre and post-processing time
to the time returned by the handleRequest() method.

* Changed as per review comments and offline discussion

* Added comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bugfix cleanup incompatible Indicate PR that introduces backward incompatibility
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants