Add backward compatibility regression test suite for multi-stage query engine

[yashmayya]

• This patch adds a backward compatibility regression test suite for the multi-stage query engine. It uses the framework added in Framework for adding compatibility tests #6129, complete compatibility regression testing #6650, Create test suite across pinot releases and services #4001 and extends it to use the multi-stage query engine with queries like joins, windows, and set operations.
• We're using a separate new test suite for now because we only want to run the multi-stage query engine backward compatibility tests against master whereas the existing tests currently run against release-1.0.0, release-1.1.0 and master.
• We can't currently run these multi-stage query engine backward compatibility tests against older releases because there are already known breakages in backward compatibility between these releases. Going forward, however, we want to maintain backward compatibility in the multi-stage query engine as well, which is why these tests are being added.
• Future releases release-1.2.0 , release-1.3.0 etc. will be added to the list of branches that the multi-stage query engine backward compatibility tests are run against.
• These new tests use the existing framework for backward compatibility regression tests and hence run tests with these scenarios using a single controller, broker, and server setup:
  • Controller on older version, Broker on older version, Server on older version
  • Controller on newer version, Broker on older version, Server on older version
  • Controller on newer version, Broker on newer version, Server on older version
  • Controller on newer version, Broker on newer version, Server on newer version
• Ideally, for the multi-stage query engine, we'd want to test for multi-server scenarios as well (where different servers are running different versions). Support for this will be added in a future patch. The current tests help catch compatibility issues between Broker <-> Server interactions.
• Note that a new GitHub Actions workflow file has been created for the multi-stage query engine compatibility regression tests since currently we don't want to run it against older versions whereas we'd still want to run the regular compatibility regression test suite against older versions (for instance, during releases).
• Once we reach a state where we have multiple released versions where backward compatibility has been maintained for the multi-stage query engine, we can merge the two compatibility regression test suites and run them against the same versions.
• Suggested labels: testing

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 44 lines in your changes are missing coverage. Please review.

Project coverage is 62.12%. Comparing base (59551e4) to head (f03f4c7).
Report is 523 commits behind head on master.

Files	Patch %	Lines
...apache/pinot/common/utils/SqlResultComparator.java	0.00%	21 Missing ⚠️
...src/main/java/org/apache/pinot/compat/QueryOp.java	0.00%	18 Missing ⚠️
...r/src/main/java/org/apache/pinot/compat/Utils.java	0.00%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #13193      +/-   ##
============================================
+ Coverage     61.75%   62.12%   +0.37%     
+ Complexity      207      198       -9     
============================================
  Files          2436     2536     +100     
  Lines        133233   139442    +6209     
  Branches      20636    21555     +919     
============================================
+ Hits          82274    86629    +4355     
- Misses        44911    46320    +1409     
- Partials       6048     6493     +445     

Flag	Coverage Δ
custom-integration1	<0.01% <0.00%> (-0.01%)	⬇️
integration	<0.01% <0.00%> (-0.01%)	⬇️
integration1	<0.01% <0.00%> (-0.01%)	⬇️
integration2	0.00% <0.00%> (ø)
java-11	62.09% <0.00%> (+0.38%)	⬆️
java-21	62.01% <0.00%> (+0.38%)	⬆️
skip-bytebuffers-false	62.11% <0.00%> (+0.36%)	⬆️
skip-bytebuffers-true	61.99% <0.00%> (+34.26%)	⬆️
temurin	62.12% <0.00%> (+0.37%)	⬆️
unittests	62.12% <0.00%> (+0.37%)	⬆️
unittests1	46.70% <0.00%> (-0.19%)	⬇️
unittests2	27.73% <0.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[vrajat]

IIUC, this PR currently has a couple of tables and a few queries. What would it take to support other datasets and queries? For example testHardCodedQueries has a larger of set of queries. The test framework also has a way to generate random queries. Not suggesting that this work has to be done for this PR. I am curious what the effort will be to increase coverage of different query plans.

Another question for the future. The concept of plan versions has to be introduced. Compatibility of a specific version has to be tested ?

[yashmayya]

For example testHardCodedQueries has a larger of set of queries

Is this what you're referring to -

pinot/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java

Line 287 in 518fd18

private void testHardcodedQueriesV2()

?

If yes, that's from the integration test suite and is completely separate from the backward compatibility regression test framework (added in #6129). The latter is a set of bash scripts and yaml configurations to set up controllers, brokers, servers and upgrading / downgrading each sequentially while also ingesting real-time / offline data and running a bunch of static queries.

I am curious what the effort will be to increase coverage of different query plans.

I think it should be pretty straightforward to add more datasets and queries once we have this baseline in place. It's currently quite important that we at least have some backward compatibility regression tests for the multi-stage query engine since there's none currently and the only way to detect that a commit is breaking backward compatibility with it is via either eyeballing the code or manually setting up a cluster for an upgrade test. The compatibility verifier has a script to generate test data but nothing to generate random queries though.

Another question for the future. The concept of plan versions has to be introduced. Compatibility of a specific version has to be tested ?

I'm not sure I follow? Currently, this PR adds a new GitHub Actions workflow that can be manually triggered during Pinot releases to verify that backward compatibility for the multi-stage query engine is maintained with previously released versions. This also adds a new check that runs on every PR to ensure that the PR isn't introducing a backward incompatible change for the multi-stage query engine (currently only run versus master since we already know that there were some backward incompatible changes made previously). In the future, this will be updated to also run against newer releases since we want to maintain compatibility going forward.