Improving Scientific Machine Learning with Algorithmic Insights from Numerical Analysis

Scientific machine learning offers methods of building approximate but efficient solutions to challenging tasks in scientific computing, such as simulating a physical phenomenon or inferring high-dimensional model parameters from data. Algorithms from numerical analysis provide classical computational solutions, but these algorithms are not designed to learn from large datasets, and they can be expensive to run for certain problems and levels of error tolerance. Machine learning solutions offer approximations to these classical methods, often using neural networks, which can learn from large datasets. While general-purpose neural network architectures and training algorithms are frequently a natural first choice for such problems, scientific computing tasks are often ill-conditioned and require a level of accuracy unattainable by general-purpose methods. At the same time, numerical analysis research provides a wealth of insights for principled methods of computing solutions to these problems. This thesis develops state-of-the-art machine learning methods for scientific computing problems by applying insights from contemporary algorithms research in numerical analysis. We consider three different problem settings in scientific computing. The first example uses insights from applied harmonic analysis to derive rotation-invariant random feature models, which provide an attractive alternative to deep neural networks or hand-designed kernels. In another setting, insights from optimization theory and recursive linearization algorithms allow us to design simple yet accurate neural networks for multi-frequency inverse scattering problems in a highly nonlinear regime, when training data is available. In the final setting, this thesis considers a broader class of partial differential equation (PDE)-based inverse problems, in a setting where training data is scarce. In this setting, the thesis contributes develops software and algorithms for accelerating highly-efficient and highly-accurate fast direct solvers of elliptic PDEs on hardware acceleration devices. The final chapter of this thesis develops optimization methods applying these novel algorithm and software tools to multiple PDE-based inverse problems in imaging.