Cardiovascular disease (CVD) has become the most concerning disease worldwide. A Phonocardiogram (PCG), the graphical representation of heart sound, is a non-invasive method that helps to detect CVD by analyzing its characteristics. Several machine learning (ML) approaches have been proposed in the last decade to assist practitioners in interpreting this disease accurately. However, the ML-based method requires a considerable amount of PCG data with a balance between data categories for unbiased performance. Moreover, PCG data in the literature is scarce, and the available database has a strong imbalance between the normal and abnormal categories. This data imbalance causes outcomes to be severely biased towards classes with greater samples. This work proposes a variable-hop fragment selection method with a pre-trained CNN model to counter the issues of data scarcity and imbalance. The proposed framework improves 7.12% of unweighted average recall (UAR) value for assessing an imbalanced PCG dataset compared to the state-of-the-art method and reports an overall UAR of 92.46% on the PhysioNet/CinC Challenge 2016 dataset. The improved performance signifies the clinical relevance of the work providing reliable assistance for heart auscultation and has the potential to screen for heart pathologies in data constraint applications.