The use of modern computer vision for defect detection is not a new concept. However, in many inspection processes, it is difficult to quantify the severity of the defects. Specifically, in cases where the current process relies on subjective human quantification, it is difficult to produce ground truth data, since each expert has a unique and sometimes non-repeatable opinion and, hence, the ground truth annotation might differ across different experts. Instead of averaging over expert opinions, as done implicitly when training traditional deterministic models, in this paper, we employ a Probabilistic U - Net to capture the hidden distribution of the subjective expert assessments. We investigate sampling the predicted expert distribution with a tiling approach down to a pixel level instead of on a per-image basis as originally proposed. We show that this sampling strategy does not affect the model's ability to represent the expert distribution. We evaluate the model's ability to converge by measuring the sum of absolute difference in means as more samples are added for each pixel. Our results show that sampling on a pixel level has the advantageous property of making the average over multiple samples converge towards a stable mean in a less fluctuating fashion thus making the model less sensitive to outliers, which makes it easier to decide on a final sample size.