Biological Sciences Division
Published April 8, 2022 | Version v1
Journal article Open

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States

Creators

  • 1. University of Massachusetts
  • 2. Centers for Disease Control and Prevention
  • 3. Karlsruhe Institute of Technology
  • 4. In-Q-Tel
  • 5. Heidelberg Institute for Theoretical Studies
  • 6. University of Bern
  • 7. Iowa State University
  • 8. University of Chicago

Description

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.

Data availability

The forecasts from models used in this paper are available from the COVID-19 Forecast Hub GitHub repository (https://github.com/reichlab/covid19-forecast-hub) (4, 34) and the Zoltar forecast archive (https://zoltardata.com/project/44) (35). These are both publicly accessible. The code used to generate all figures and tables in the manuscript is available in a public repository (https://github.com/reichlab/covid19-forecast-evals). All analyses were conducted using the R language for statistical computing (version 4.0.2) (36). We followed the EPIFORGE 2020 guidelines for reporting results from epidemiological forecasting studies (SI Appendix, Table S5) (37).

Files

cramer-et-al-2022-evaluation-of-individual-and-ensemble-probabilistic-forecasts-of-covid-19-mortality-in-the-united.pdf

Files (25.0 MB)

Additional details

Identifiers

DOI
10.1073/pnas.2113561119
Other
oai:uchicago.tind.io:9686

Related works

Is supplement to
https://doi.org/10.1073/pnas.2304076120 (URL)

Funding

CDC
Google
Facebook
National Science Foundation
DMS-2027369
Morris-Singer Foundation
CDC
1U01IP001122
National Institute of General Medical Sciences
R35GM119582
Helmholtz Foundation
Klaus Tschira Foundation
Unknown funder
GM124104
National Science Foundation
III-1812699
Wellcome Trust
210758/Z/18/Z
William W. George Endowment
Virginia C. and Joseph C. Mello Endowments
National Science Foundation
DGE-1650044
National Science Foundation
MRI 1828187
CDC and Council of state and Territorial Epidemiologists
NU38OT000297
The Partnership for Advanced Computing Environment at Georgia Tech
Andrea Laliberte
Joseph C. Mello
Richard "Rick" E. and Charlene Zalesky
Claudia and Paul Raines
CDC
Modeling Infectious Diseases in Healthcare
National Science Foundation
Expeditions
National Science Foundation
CAREER
National Science Foundation
Rapid Response Research
National Science Foundation
Medium
National Science Foundation
Research Traineeship
CDC
MInD program
Oak Ridge National Laboratory
Georgia Tech
Georgia Tech Research Institute
Institute for Health Metrics and Evaluation
The Bill & Melinda Gates Foundation
State of Washington and National Science Foundation
FAIN: 2031096
Iowa State University
Plant Sciences Institute Scholars Program
National Science Foundation
DMS-1916204
National Science Foundation
CCF-1934884
Laurence H. Baker Center for Bioinformatics and Biological Statistics
Johns Hopkins University
National Science Foundation
RAPID
National Science Foundation
RAPID
State of California
US Health and Human Services
US Department of Health Services
US Office of Foreign Disaster Assistance
Johns Hopkins Health System
Johns Hopkins Bloomberg School of Public Health
Johns Hopkins University Modeling and Policy Hub
CDC
5U01CK000538-03
University of Utah
Immunology, Inflammation, & Infectious Disease Initiative
Los Alamos National Lab
Laboratory Directed Research and Development
CDC
COVID Supplement
CSTE
Cooperative Agreement
National Science Foundation
RAPID
National Science Foundation
RAPID
National Science Foundation
RAPID
National Science Foundation
RAPID
Google
Faculty Award
Defense Advanced Research Projects Agency
W31P4Q-21-C-0014
NIGMS
R35GM119582
National Science Foundation
1749854
University of Michigan
National Science Foundation
2035360
National Science Foundation
2035361
Gordon and Betty Moore Foundation
Rockefeller Foundation

UChicago Information

Division(s)
Biological Sciences Division
Department(s)
Ecology and Evolution