Data processing

The users’ demographic data and raw heart sound recordings were automatically downloaded using Python Firebase API called Pyrebase for offline analysis.¹¹ During the whole analysis, data were only accessible by a researcher (H.L.).

Visual assessment of heart sound quality

All raw heart sound recordings were plotted using Python software to observe signal morphologies. Then the heart sound quality was visually assessed per recording by one observer (H.L.). Each recording was assigned to one of the following three categories: good quality, unsure quality and bad quality. Good-quality signal was defined by at least 1 heartbeat where both the first (S1) and the second heart sound (S2) were clearly visible in the raw recordings. Bad-quality signal was defined by absence of any heartbeat with S1 and S2. Unsure-quality signal was defined if the observer deemed the signal to be in need of further processing for reliable identification of any heartbeats. To test the inter-observer variability of this visual quality assessment approach, the second assessor (J.L.) independently evaluated a subsample of 1000 heart sound recordings randomly selected from the heart sound database according to the criteria above.

For validation purpose, examples of good-, unsure- and bad-quality heart sounds were processed using a time-frequency analysis method called S-transform.¹² The S-transform projects the one-dimensional heart sound signals to a two-dimensional time-frequency plane for better observations of signal components. S-transform was only used for representative heart sound examples of good, unsure and bad quality in order to confirm whether our proposed visual assessment of heart sound quality was consistent with the advanced signal processing technique.

Factors affecting heart sound quality

Heart sounds classified as good quality were deemed interpretable for analysis. Success rate was defined by the proportion of users who made at least 1 good-quality recording within a given number of attempts. We analysed the factors affecting the users’ success rate, including hardware (i.e. iPhone version) and the users’ demographics including sex, age, and BMI [calculated by weight (kilogram) divided by height (metre) squared]. We could not analyse the effect of recording location on heart sound quality because all recordings from those users who indicated recording location were at the same area (Spot 1 of the app) on the chest.

Visual heart sound quality assessment

Good-, unsure-, and bad-quality recordings amounted to 5647 (74.6%), 466 (6.2%), and 1457 (19.2%), respectively. A total of 80.1% (919/1148) users were able to make at least one good-quality recording. Median number of attempts needed to make the first good-quality recording was 1 (IQR 1–1) per user. Interobserver agreement of heart sound quality was 84.7% in the 1000 randomly selected subsamples (see Supplementary material online, Table S1). For good-, unsure-, and bad-quality recordings, the agreements of the second assessor with the first assessor were 89.1% (672/754), 41.0% (25/61), and 81.1% (150/185), respectively. Most (61.4%, 94/153) of disagreements occurred in the category of ‘Unsure-quality’.

Examples of good- (A), unsure- (B), and bad-quality (C) heart sounds are shown in Figure 3, including time-frequency spectrum (upper strip) and raw signals (lower strip). Audio files of these examples are also available online.¹⁷ The good-quality recording (Figure 3​AA) clearly showed two major heart sound components (i.e. S1 and S2). Sound energies of S1 were preponderant below 100 Hz, while S2 had energies spreading beyond 100 Hz. In some heartbeats, the third (S3) and/or fourth heart sound (S4) that usually have a low signal amplitude, were also visible. The unsure-quality recording (Figure 3​BB) did not show clear S1 and S2 heart sound components in raw signals, though a few high-amplitude vibrations seemed to appear regularly. Using time-frequency analysis in these unsure-quality signals, S1 showed as low-frequency clusters occurring at a regular interval, followed by S2 in high-frequency regions. The bad-quality recording (Figure 3​CC) did not show any recurring components, neither in raw signals nor in the time-frequency spectrum.

Open in a separate window

Figure 3

Time-frequency colourmap (upper strip) and raw signal (lower strip) of good- (A), unsure- (B), and bad-quality (C) raw heart sound recordings. The time-frequency colourmaps show how heart sound frequency (vertical axis) changes with time (horizontal axis), with heart sound energy ranging from low (in blue) to high (in red). In raw signals, locations of first- (S1), second- (S2), third- (S3), and fourth heart sound (S4) are labelled.

Time of measurement

The influence of increasing user experience on sound quality was evaluated using the Mann–Kendall trend test. The proportion of users who made a good-quality recording significantly increased with the number of attempts to measure heart sound (P<0.001; Figure 4). For the first attempt, ~60% of new users made a good-quality recording, which percentage increased to ~75% in the subsequent two attempts. From the fourth attempt and further, the success rate barely improved and approached ~80%, indicating that most users ‘learned’ how to use the app within the first three attempts.

Open in a separate window

Figure 4

Change of success rate with number of attempts to record heart sound.

Phone version

The relationship between iPhone version and proportion of users who succeeded in making a good-quality recording in one attempt (orange line) and in three attempts (red line) is shown in Figure 5. The proportion significantly increased when more recent iPhones were used for the first attempt (P=0.03), but not for the first three attempts (P=0.29). However, the significance was mainly driven by the lowest success rate in the users of iPhone 7 in both the first attempt and the first three attempts. Actually, the success rate did not change markedly from iPhone versions 8 to 13, fluctuating within 10% (for the first attempt: 55–65%; for the first three attempts: 70–80%). These numbers showed a good hardware support when using iPhone to record heart sounds.

Open in a separate window

Figure 5

IPhone version and success rate. Orange line and dark blue boxes indicate the percentage of heart sound recordings rated with good quality in the first attempt by the users, while red line and light blue boxes in the first three attempts.

Sex and age

Sex was reported in 184 (16%) users, including 92 males and 92 females. Success rate did not differ between males and females in the first attempt (P=0.41; Figure 6​AA) and in the first 3 attempts (P=0.21). Compared with females, the users who did not report their sex had a significantly lower success rate in one attempt (P=0.002) and in three attempts (P=0.003). This might indicate that the ones who reported their sex and/or age might have done so because they were much more confident in their recordings (they had a higher success rate).

Open in a separate window

Figure 6

Changes of percentage of users having a good-quality heart sound recording with sex and age. Orange line and dark blue boxes indicate the percentage of heart sound recordings rated with good quality in the first attempt by the users, while red line and light blue boxes in the first three attempts.

Age (35±14 years; range: 10–69 years) was reported in 194 users. When grouping the users’ age by every decade, in the first attempt, success rate of making a good-quality recording tended to decrease with age (P=0.06; Figure 6​BB). However, this trend diminished when the first three attempts were taken into consideration, indicating good learning capability of more advanced-age users (P=0.70).

Can smartphone be used to record heart sounds?

Nowadays, most smartphones are equipped with a high-quality microphone having a frequency response of 20 Hz–20 kHz. This range is more than sufficient to cover the frequency range of heart sounds (20–200 Hz).¹⁸ The App iStethoscope was the first piece of software developed for auscultation of heart sound.⁸ It was optimized for iPhone 3G devices and also shown to work effectively on 4, 4S, 5, 5C, and 5S iPhones. This finding was supported by another study which tested the accuracy of iPhone microphones using 20 Hz to 20 kHz sounds, and reported that for the early versions of iPhone (iPhone 3Gs to iPhone 5), the difference in sound pressure level was not significant between a professional sound recorder and iPhone microphones.¹⁹ Our study extends these studies by showing that the more recent versions of iPhone from 7 to 13 also work for heart sound measurement, although the success rate of obtaining good-quality recordings tends to be lower for iPhone 7.

The only study on Android phones for heart sound collection reported that heart sounds can be recorded in hospital settings using an app (CPstethoscope) on Samsung and LG phones.⁹ Among the 46 patients enrolled, the researchers were able to distinguish among normal, third heart sound, fourth heart sound, systolic murmur and diastolic murmur. However, in this small-sample-size pilot study, the heart sounds were measured by researchers who had professional knowledge about cardiovascular anatomy.

Our study was a large-scale study that distributed an app among over 1100 non-medical public users. The results showed that the smartphone could not only record the most distinctive components (S1 and S2) of heart sound, but also the low-amplitude and low-frequency extra heart sound components (S3 and S4; examples in Figure 3​AA) that are normally difficult to hear by experienced doctors using stethoscope. Moreover, our study shows good utilization of the app among the first-time users for heart sound measurement. However, we did not investigate the possibility to use the smartphone-recorded heart sounds for clinical diagnosis, because this would require additional personalized medical information and therefore also stricter privacy settings.

Factors affecting heart sound quality

We found that despite a high success rate (~80%) of recording a good-quality heart sound among the users, a few factors still affected the use of our app. The fact that success rate increased with time of measurement, especially in the first three attempts, indicates the possibility of improving the app design to better guide the users to quickly learn how to use the app in the first few attempts in the future. For example, compared with the static user’s guidance in the current app, we may use a video to guide the users how to use the app properly. Regarding hardware, iPhone version does not seem to relate to heart sound quality, indicating similar or even the same microphone sensor used in the series of products. Signal quality is not markedly different between sexes, showing a good understanding of the app in both sexes. The decreased proportion of users who were able to make a good-quality recording with age warranted a better design of the app to guide the old-age users to record heart sounds. However, despite the lower success rate among the elderly for the first attempt of the app, actually these users quickly learned how to correctly use the app in the next two attempts, as evidenced by the higher increase of success rate from the first attempt to the first three attempts compared to their younger counterparts. Yet this finding may need to be confirmed in a larger scale study in the future.

Advantages of using smartphone for recording heart sounds

The huge amount of smartphone users (6.38 billion in 2021) provides a unique opportunity to turn the built-in microphone to an electronic stethoscope for both health education among the public.²⁰ A further possible application, not yet investigated here, is to use the smartphone-built-in microphone for medical purposes in patients. Because almost everybody owns a smartphone, the patients may record their heart sounds at home and during natural conditions as opposed to the outpatient clinic. Since the solution does not incur any extra cost contrary to buying a professional electronic stethoscope, those who are living in underdeveloped or remote areas short of medical resources can use the app for health-related purposes. From this perspective, turning smartphone into an electronic stethoscope provides a solution to reduce health system costs in these areas.

Digital stethoscopes as well as smartphones have the advantage that the signals can be stored and objectively analyzed, while the conventional analogue stethoscopes rely on the well-trained doctors’ ears that possibly introduce interobserver variability in interpretation of the signals.^21,22 The abilities of digital stethoscope may be further enhanced by combining with automatic algorithms including machine learning algorithms. For example, the PASCAL Classifying Heart Sounds Challenge using heart sounds collected from the iPhone-based app (iStethoscope Pro) has stimulated the development of multiple algorithms including convolutional neural network, deep learning algorithm and artificial neural network for automatic segmentation and classification of heart sounds.^23–26

Future perspectives

Our study analyzed an increasingly expanding heart sound dataset of the Echoes app. Our findings support further development of the app for not just public engagement purposes but also for medical purposes. As a low-cost and easy-to-use solution, our app may be deployed on a large scale in areas with insufficient medical resources to help screen and monitor patients with cardiovascular diseases. An example is to use the app to screen for congenital and valvular heart diseases which mostly produce systolic and/or diastolic murmurs but which may be too late to be intervened at an advanced stage. Another potential application is to use the app for heart rate and rhythm managements in patients with chronic cardiovascular diseases such as atrial fibrillation. Atrial fibrillation creates the irregularities of not only rhythm but also amplitude of the heart sounds. In this respect, heart sounds may serve as a valuable tool in supporting electrocardiogram for evaluation of arrhythmias. In particular, our app enables the users’ self-recording of transient arrhythmias in conditions like paroxysmal atrial fibrillation and premature ventricular contraction at home for further hospital consultation.

Since built-in microphone of smartphone is more sensitive to vibrations falling in audible frequency range (≥20 Hz), it may be combined with the built-in inertial measurement unit of the phone for a better measurement of lower-frequency vibrations such as seismocardiogram. For example, the built-in accelerometer may be put on the chest of a patient in supine position to record seismocardiogram.²⁷ However, before applying the app to the patients, further preliminary tests in hospital settings may help improve the usability of the app and establish its role in conjunction with other existing monitoring methods such as electrocardiogram.