Fix Bug in Handling Equal Comparison Column Values in Upsert

[ankitsultana]

See issue for explanation of the issue: #12398

This PR fixes the issue by de-duplicating the records before the map is updated. This adds a small cost in terms of memory, but it should be quite low (e.g. for 1M records, with 32 byte primary-keys, we will have 32MB + memory address and other Record info parameter costs).

There's a small latency cost also added due to the extra dedup step. But we think it's quite low compared to the rest of operations so shouldn't be too big an issue (e.g. after dedup the ConcurrentHashMap is updated for all records, which can contend with Realtime ingestion and other addOrReplaceSegment calls).

We only do this for Partial Upsert tables right now since Full Upsert tables don't suffer from the issue mentioned in #12398.

Test Plan: Added a unit-test.

[codecov-commenter]

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (5c81c0a) 61.56% compared to head (563fba0) 61.63%.
Report is 24 commits behind head on master.

Files	Patch %	Lines
...cal/upsert/BasePartitionUpsertMetadataManager.java	84.61%	1 Missing and 1 partial ⚠️
...t/ConcurrentMapPartitionUpsertMetadataManager.java	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #12395      +/-   ##
============================================
+ Coverage     61.56%   61.63%   +0.06%     
- Complexity      201      207       +6     
============================================
  Files          2426     2436      +10     
  Lines        132709   133117     +408     
  Branches      20519    20620     +101     
============================================
+ Hits          81706    82050     +344     
- Misses        44994    45027      +33     
- Partials       6009     6040      +31     

Flag	Coverage Δ
custom-integration1	<0.01% <0.00%> (?)
integration	<0.01% <0.00%> (+<0.01%)	⬆️
integration1	<0.01% <0.00%> (?)
integration2	0.00% <0.00%> (ø)
java-11	27.73% <73.33%> (+27.73%)	⬆️
java-21	34.89% <0.00%> (-26.68%)	⬇️
skip-bytebuffers-false	61.62% <73.33%> (+0.07%)	⬆️
skip-bytebuffers-true	34.85% <0.00%> (+0.02%)	⬆️
temurin	61.63% <73.33%> (+0.06%)	⬆️
unittests	61.63% <73.33%> (+0.06%)	⬆️
unittests1	46.73% <0.00%> (+<0.01%)	⬆️
unittests2	27.73% <73.33%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.