Summary

Introduction

The COVID-19 pandemic caused many countries to impose strict restrictions on their citizens' mobility and behaviour to curb the rapid spread of disease, often termed lockdowns. Since relaxing these restrictions, many countries sought to avoid their re-imposition through combinations of non-pharmaceutical interventions¹ and test-and-trace systems. Despite these efforts, many countries have had increases in infection since re-opening and have often re-imposed either regional² or national lockdowns. Regional lockdowns aim to contain the disease while minimising the severe economic effect of national lockdowns.

The effectiveness of regional interventions depends on the early detection of so-called infection hotspots.³ Large-scale, population-based testing can indicate regional hotspots, but at the cost of a delay between testing and actionable results. Moreover, accurately identifying changes in the infection rate requires sufficient testing coverage of a given population,⁴ which can be costly and requires substantial testing capacity. Regional variation in testing access can hamper the ability of public health organisations to detect rapid changes in infection rate. There is a high unmet need for tools and methods that can facilitate the timely and cost-effective identification of infection hotspots to enable policy makers to act with minimal delay.⁵

In this study, we used self-reported, population-wide data, obtained from a mobile application (the COVID Symptom Study app), combined with targeted PCR testing to provide geographical estimates of disease prevalence and incidence. We further show how these estimates can be used to provide timely identification of infection hotspots.

Methods

Results

From March 24 to Sept 29, 2020, a total of 4 644 227 participants signed up to use the app, of which 2 873 726 (61·9%) reported living in England. After excluding a further 4508 (0·1%) participants with invalid age information and 26 486 (0·6%) without any daily assessments logged, we report 2 842 732 users that participated in this study (table). In total, these participants completed 120 192 306 daily assessments, with a median of 16 logs (IQR 3–68) per user, corresponding to logged data for 22·4% of the total possible person-logs across the study period. Compared with the population of England, we had fewer participants aged 18 or younger (196 143 [6·9%] vs 22·5%) and fewer aged 65 or older (349 975 [12·3%] vs 18·4%), more female than male participants (1 739 929 [61·2%] vs 50·6%), and fewer participants living in deprived regions (1 322 394 [46·5%] in the least deprived tercile vs 33·3%; table). Between April 28 and Sept 29, a total of 851 250 invitations for swab tests were sent out. 169 682 (19·9%) of these invitations were responded to by a person reporting a swab test on the app, of which 1912 (1·1%) tested positive. In addition, users reported the results of 689 426 non-invited tests in the app, of which 25 663 (3·7%) were positive. 621 031 (0·5%) of 120 192 306 symptom reports were classified as COVID-19 positive by the symptom model.

In our comparison of England-wide incidence estimates (I_S) with government testing data and the ONS survey¹⁷ (figure 1A), we included two estimates from the ONS: the official reports, released every week, and the results from time-series modelling. The reports represent the best estimate of the ONS at the time of release, whereas the times-series model can evolve, leading to revision of previous estimates in response to new data (appendix p 7). The government values are consistently lower than other estimates because they are not a representative figure for the population. To account for this, we looked at the number of people who reported classic symptoms (fever, loss of smell, and persistent cough) for the first time between July 7 and Aug 5, 2020, and who did not get tested; we found the number to be 50 499 (62%). We used this percentage to scale the government data by a factor of 2·5, our best estimate of the systematic undercounting of new cases.

Open in a separate window

Figure 1

Daily incidence in the UK since May 12, 2020, compared with daily laboratory-confirmed cases and the ONS study (A) and daily prevalence in the UK compared with the ONS and REACT-1 studies (B)

ONS data are taken from the report released on Oct 9, 2020, and ONS report dates are taken as the midpoint for the date range covered by the estimate. For the Symptom Study and REACT-1, the shaded areas represent 95% CIs. For the ONS, the shaded areas and error bars represent 95% credible intervals. ONS=Office for National Statistics. REACT-1=Real-time Assessment of Community Transmission.

Our results predict a steep decline in incidence until the middle of July, a trend in agreement with the government data and ONS estimates. All three estimates show an increase in the number of daily cases from mid-August throughout September; on Sept 28, 2020, we estimated that there were 15 841 (95% CI 14 023–17 885) daily cases. Estimates of incidence per NHS region are in the appendix (p 8), as are maps of our most granular incidence, I_H (appendix p 8).

We also compared our England-wide prevalence estimates, P_A and P_H, with prevalence reported by the ONS and REACT-1 studies (figure 1B). The symptom-based assessments, P_A, indicate a continuous drop in the number of cases from April 1, following the lockdown measures instigated on March 23, which then plateaued in mid-June and began to rise again sharply from early September. The trends observed for P_A agree with data from the ONS survey and the REACT-1 study, although there is some divergence in September, when both P_A and REACT-1 data show a sharper rise in prevalence than the ONS data (figure 1B). It should be noted that P_A captures only symptomatic cases, whereas ONS and REACT-1 data also capture asymptomatic cases, which are thought to account for 40–45% of the total cases.¹⁸ Taking this factor into account, P_A is slightly higher than ONS and REACT-1 data. On Sept 28, 2020, we estimated a prevalence of 0·53% (95% CI 0·45–0·60).

P_H agrees with the trends in the other estimates, and predicts a rise in cases in September at a similar rate to P_A and REACT-1 data (figure 1B). However, the absolute values are consistently lower than other estimates. The recovery model used to calculate P_H is shown in figure 2. Although most users recover in 7–10 days, the curve shows there is a substantial minority who take longer than 3 weeks to recover from COVID-19.

Open in a separate window

Figure 2

Empirical probability density function of days to recovery with a gamma fit (A) and empirical cumulative density function of days to recovery with the same gamma fit (B)

The bars represent empirical findings and the red line is the gamma fit.

The size of our dataset allows us to estimate prevalence for more granular geographical regions than the ONS (appendix p 9). We considered that our estimates of prevalence might be biased by a user's health-seeking behaviour, and we sought to assess the influence of this factor by removing from the analysis all users who reported being sick upon sign-up (appendix p 10).

We compared our estimates of R(t) for England between June 24 and Sept 28, 2020, with the consensus estimates provided by the UK Government Science Pandemic Influenza Modelling group (figure 3).¹⁹ Estimates for each of the NHS regions are in the appendix (p 12). The estimates agree that R(t) has been above 1 from early-mid-September, and we estimate that R(t) in England was 1·17 (95% CI 1·15–1·19) on Sept 28, 2020. The government estimates are much smoother than our estimates, which is probably because they are derived from a consensus of the R(t) estimates from the models produced by many groups.

Open in a separate window

Figure 3

Estimated R(t) for England between June 24 and Sept 28, 2020

The shaded area for Symptom Study data represents 95% credible intervals and for government data represents 95% CIs. UK Government estimates published every 7–12 days from June 12, 2020. R(t)=effective reproduction number.

Concerning hotspot detection in England, ranking based on incidence consistently outperformed the prevalence-based ranking (figure 4). Performance varied over time; the best incidence-based ranking produced recall scores of up to 0·75, with a minimum of 0·35, indicating that we can predict between 7 and 15 of the regions in the UK Government's top 20. On a geographically granular level, on Sept 28, 2020, we detected 15 (75%) of the 20 regions with highest incidence according to government test data. The ranking performed best in late September when cases began to rise, due to a greater difference between regional case numbers. We also compared the agreement of weekly cases in each UTLA between our data and government data against the number of government pillar 2 tests carried out (figure 5). The correlation indicates that the two estimates agree best when the government carries out more tests. This finding means that disagreements between the two rankings could be partially explained by poorer ranking in regions with limited government testing.

Open in a separate window

Figure 4

Performance of our two ranking methods: ranking by prevalence and incidence on two metrics, recall at 20 (A) and the normalised mean reciprocal rank at 20 (B)

Vertical lines represent 95% CIs.

Open in a separate window

Figure 5

Agreement between COVID Symptom Study and UK Government case numbers per week and UTLA, against the number of government pillar 2 tests carried out

UTLA=Upper Tier Local Authority.

Discussion

In this study, we have demonstrated the use of population-wide data reported through the COVID Symptom Study app, drawing on more than 120 million daily reports and about 170 000 invited swab tests from more than 2·8 million users to estimate prevalence, incidence, and R(t). Other digital surveys have been used to provide valuable real-time information about the pandemic, including How We Feel,²⁰ Corona Israel,⁵ the Facebook Survey, and CovidNearYou. However, to our knowledge we are the first to provide national-level disease surveillance, and we have found good agreement with traditional, representative community surveys.

Furthermore, we have used our data to produce geographically granular estimates and a list of potential hotspots. The list consistently flags a relatively high number of regions highlighted by the UK Government testing data. Although we compared our data with that from government testing in our results, it must be noted these cannot be considered ground truth estimates of COVID-19 cases. The government data are an incomplete sampling of new COVID-19 cases; results from our app indicate that only about 40% of those who report classic COVID-19 symptoms go on to receive a test. Furthermore, testing capacity is not uniform across all UTLAs.²¹ Our results indicate that our case estimates agree best with government estimates in areas with high levels of testing per capita, suggesting that our estimates could prove a valuable resource for forecasting in regions with poor testing provision. There are other reasons why our list might differ from the Government list. First, the two methods might have different uptake in some higher-risk groups, such as students in provided accommodation, thus showing different sensitivity to hotspots based on the demographic make-up of a region. The two methods might therefore be complementary, and we suggest that our hotspot detection could be most beneficial as an additional indication of regions where increased testing might be best focused. The modest reliance on PCR tests suggests our approach could prove valuable in countries where testing infrastructure is less developed, although further work is required to assess our approach in other locations.

Other efforts to track the national progression of COVID-19 rely on self-swabbing from community cohorts. Two such efforts exist in England: the ONS study¹⁷ and the REACT-1 study.^{8, 9} These studies have the advantage of being more representative of the population, and their design enables the detection of asymptomatic cases. However, they are smaller than the COVID Symptom Study; the ONS and REACT-1 studies currently report 120 000–175 000 participants in England, whereas the app reports over 2 800 000 users in England. The ability to use self-reported symptom data from this large cohort enables us to make predictions of more geographically granular regions than do either the ONS or REACT-1 studies, allowing us to predict COVID-19 hotspots at the UTLA level. Our estimates should therefore be viewed as independent and complementary to those provided by the ONS and REACT-1 studies.

Several limitations to our work must be acknowledged. The app users are not a representative sample of the wider population for which we aim to make an inference. There is a clear shift in age and gender compared with the general population, our users tend to live in less deprived areas,¹² and we have few users reporting from key sites such as care homes and hospitals. We accounted for some population differences when producing prevalence estimates using specific census adjusted population strata, but the number of invited tests does not allow us to do this when calculating incidence. Differences in reported symptoms across age groups²² would probably lead to different prediction models of COVID-19 positivity, and the performance of the model will vary with the prevalence of other infections with symptoms that overlap with COVID-19, such as influenza. Furthermore, the app population is less ethnically diverse than the general population.²³ Reliance on user self-reporting can also introduce bias into our results; for instance, users who are very sick might be less likely to report than those with mild symptoms. Other sources of error include collider bias²⁴ arising from a user's probability of using the app being dependent on their likelihood of having COVID-19, potentially biasing our estimates of incidence and prevalence. We showed a sensitivity analysis that attempts to understand the effect of health-seeking behaviour, but we acknowledge that there are many other biases that might affect our results—for example, our users might be more risk-averse than the general population—and that our results must be interpreted with this in mind.

We have presented a means of combining app-based symptom reports and targeted testing from over 2·8 million users to estimate incidence, prevalence, and R(t) in England. By integrating symptom reports with PCR test results, we were able to highlight regions which might have concerning increases in COVID-19 cases. This approach could be an effective, complementary way for governments to monitor the spread of COVID-19 and identify potential areas of concern.

Data sharing

Data collected in the COVID Symptom Study smartphone application are being shared with other health researchers through the UK NHS-funded Health Data Research UK (HDRUK) and Secure Anonymised Information Linkage consortium, housed in the UK Secure Research Platform (Swansea, UK). Anonymised data are available to be shared with HDRUK researchers according to their protocols in the public interest. US investigators are encouraged to coordinate data requests through the Coronavirus Pandemic Epidemiology Consortium.

Acknowledgments

Contributors

Declaration of interests

References