Files
Abstract
This dissertation broadly explores statistical approaches in data assimilation and model selection, with a focus on assessing the scalability, practicality, and trustworthiness of these approaches. Part 1 investigates assimilating sparse, noisy observations over a long-time horizon using a global, data-driven weather model. We demonstrate that our high-resolution estimates derived from the data-driven weather model FourCastNet and the variational data assimilation method 3DVar remain stable across a year-long time horizon when assimilating this sparse, noisy information. Additionally, these estimates serve as effective initial conditions for forecasting tasks by outperforming baselines that do not use our assimilation framework, particularly in a case study of typhoon prediction. Next, part 2 develops the auto-differentiable 3DVar (AD-3DVar) for learning the dynamics of a forecast correction model given sparse, noisy observations over time. This method provides substantially enhanced scalability due to its time and memory efficiency compared to ensemble-based learning approaches, therefore providing a computationally efficient learning mechanism for high-dimensional systems. We demonstrate that we are able to enhance a baseline forecasting model by learning a neural-network-based correction with imperfect observations and outperform a competing baseline. Our approach additionally is able to learn from time-varying observation locations, which the competing method cannot. We perform experiments in a variety of settings, such as varied quality of the baseline forecasting model, varied observability, and varied state dimensionality on the Lorenz '96 system. Lastly, part 3 provides a method to stabilize model selection based on the inflated argmax. Our method adaptively returns a collection of models that all fit the data and is stable such that the returned collection is robust to perturbation of the underlying training data. We demonstrate our advantage in settings known to be unstable, such as regression with highly correlated covariates, dynamical system equation discovery, and graph structure selection. Our method improves trustworthiness by reflecting uncertainties and provides robustness to model misspecification.