Do not pause ingestion when upsert snapshot flow errors out

[tibrewalpratik17]

label:
upsert
bugfix

Saw this in our Uber cluster today where ingestion got stopped due to some exception in snapshot flow. The following was the error we encountered (pinot-version-build: 1.0.0):

java.lang.IllegalArgumentException: 
Path: /data/upinot/stream-pinot-server/dataDir/<tableName>/<tableName>__3__31__20240427T0642Z is not a directory at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:210) at 
org.apache.pinot.segment.spi.store.SegmentDirectoryPaths.findSegmentDirectory(SegmentDirectoryPaths.java:50) at 
org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentImpl.getValidDocIdsSnapshotFile(ImmutableSegmentImpl.java:172) at 
org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentImpl.hasValidDocIdsSnapshotFile(ImmutableSegmentImpl.java:152) at 
org.apache.pinot.segment.local.upsert.BasePartitionUpsertMetadataManager.doTakeSnapshot(BasePartitionUpsertMetadataManager.java:667) at 
org.apache.pinot.segment.local.upsert.BasePartitionUpsertMetadataManager.takeSnapshot(BasePartitionUpsertMetadataManager.java:637) at 
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:679) 
at java.base/java.lang.Thread.run(Thread.java:829)

Ideally the segment was actually deleted but somehow was still a part of trackedSegments.

This patch prevents the consumption thread from erroring out if the snapshot flow has encountered any issue.

cc @Jackie-Jiang @klsince

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 35.23%. Comparing base (59551e4) to head (541cf6a).
Report is 515 commits behind head on master.

Files	Patch %	Lines
...cal/upsert/BasePartitionUpsertMetadataManager.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #13257       +/-   ##
=============================================
- Coverage     61.75%   35.23%   -26.52%     
- Complexity      207     1110      +903     
=============================================
  Files          2436     2459       +23     
  Lines        133233   135358     +2125     
  Branches      20636    20987      +351     
=============================================
- Hits          82274    47694    -34580     
- Misses        44911    84141    +39230     
+ Partials       6048     3523     -2525     

Flag	Coverage Δ
custom-integration1	<0.01% <0.00%> (-0.01%)	⬇️
integration	<0.01% <0.00%> (-0.01%)	⬇️
integration1	<0.01% <0.00%> (-0.01%)	⬇️
integration2	0.00% <0.00%> (ø)
java-11	35.23% <0.00%> (-26.48%)	⬇️
java-21	<0.01% <0.00%> (-61.63%)	⬇️
skip-bytebuffers-false	35.23% <0.00%> (-26.52%)	⬇️
skip-bytebuffers-true	?
temurin	35.23% <0.00%> (-26.52%)	⬇️
unittests	46.63% <0.00%> (-15.12%)	⬇️
unittests1	46.63% <0.00%> (-0.26%)	⬇️
unittests2	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[klsince]

this PR is related to this patch: #13216

but it makes sense to me to not block ingestion if taking snapshot fails, as snapshots are mainly to speed up bootstrapping upsert metadata