Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.
Details
Title
Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States
Author
Cramer, Estee Y. : University of Massachusetts : (https://orcid.org/0000-0003-1373-3177) Ray, Evan L. : University of Massachusetts : (https://orcid.org/0000-0003-4035-0243) Lopez, Velma K. : Centers for Disease Control and Prevention : (https://orcid.org/0000-0003-2926-4010) Bracher, Johannes : Karlsruhe Institute of Technology : (https://orcid.org/0000-0002-3777-1410) Brennan, Andrea : In-Q-Tel Castro Rivadeneira, Alvaro J. : University of Massachusetts Gneiting, Tilmann : Heidelberg Institute for Theoretical Studies : (https://orcid.org/0000-0001-9397-3271) House, Katie H. : University of Massachusetts Huang, Yuxin : University of Massachusetts Jayawardena, Dasuni : University of Massachusetts Kanji, Abdul H. : University of Massachusetts Khandelwal, Ayush : University of Massachusetts Le, Khoa : University of Massachusetts Mühlemann, Anja : University of Bern Niemi, Jarad : Iowa State University : (https://orcid.org/0000-0002-5079-158X) Shah, Apurv : University of Massachusetts Stark, Ariane : University of Massachusetts Wang, Yijin : University of Massachusetts Wattanachit, Nutcha : University of Massachusetts Oidtman, Rachel : University of Chicago : (https://orcid.org/0000-0003-1773-9533)
The forecasts from models used in this paper are available from the COVID-19 Forecast Hub GitHub repository (https://github.com/reichlab/covid19-forecast-hub) (4, 34) and the Zoltar forecast archive (https://zoltardata.com/project/44) (35). These are both publicly accessible. The code used to generate all figures and tables in the manuscript is available in a public repository (https://github.com/reichlab/covid19-forecast-evals). All analyses were conducted using the R language for statistical computing (version 4.0.2) (36). We followed the EPIFORGE 2020 guidelines for reporting results from epidemiological forecasting studies (SI Appendix, Table S5) (37).
Funding Information
CDC Google Facebook National Science Foundation, DMS-2027369 Morris-Singer Foundation CDC, 1U01IP001122 National Institute of General Medical Sciences, R35GM119582 Helmholtz Foundation Klaus Tschira Foundation GM124104 National Science Foundation, III-1812699 Wellcome Trust, 210758/Z/18/Z William W. George Endowment Virginia C. and Joseph C. Mello Endowments National Science Foundation, DGE-1650044 National Science Foundation, MRI 1828187 CDC and Council of state and Territorial Epidemiologists, NU38OT000297 The Partnership for Advanced Computing Environment at Georgia Tech Andrea Laliberte Joseph C. Mello Richard “Rick” E. and Charlene Zalesky Claudia and Paul Raines CDC, Modeling Infectious Diseases in Healthcare, U01CK000531-Supplement National Science Foundation, Expeditions, CCF-1918770 National Science Foundation, CAREER, IIS-2028586 National Science Foundation, Rapid Response Research, IIS-2027862 National Science Foundation, Medium, IIS-1955883 National Science Foundation, Research Traineeship, DGE-1545362 CDC, MInD program Oak Ridge National Laboratory Georgia Tech Georgia Tech Research Institute Institute for Health Metrics and Evaluation The Bill & Melinda Gates Foundation State of Washington and National Science Foundation, FAIN: 2031096 Iowa State University, Plant Sciences Institute Scholars Program National Science Foundation, DMS-1916204 National Science Foundation, CCF-1934884 Laurence H. Baker Center for Bioinformatics and Biological Statistics Johns Hopkins University National Science Foundation, RAPID, 2108526 National Science Foundation, RAPID, 2028604 State of California US Health and Human Services US Department of Health Services US Office of Foreign Disaster Assistance Johns Hopkins Health System Johns Hopkins Bloomberg School of Public Health Johns Hopkins University Modeling and Policy Hub CDC, 5U01CK000538-03 University of Utah, Immunology, Inflammation, & Infectious Disease Initiative, 26798 Los Alamos National Lab, Laboratory Directed Research and Development, 20200700ER CDC, COVID Supplement, HHS-6U01IP001137-01 CSTE, Cooperative Agreement, NU38OT000297 National Science Foundation, RAPID, 2027718 National Science Foundation, RAPID, 2031536 National Science Foundation, RAPID, DMS 2028401 National Science Foundation, RAPID, 2029626 Google, Faculty Award Defense Advanced Research Projects Agency, W31P4Q-21-C-0014 NIGMS, R35GM119582 National Science Foundation, 1749854 University of Michigan National Science Foundation, 2035360 National Science Foundation, 2035361 Gordon and Betty Moore Foundation Rockefeller Foundation