Scientific applications often require massive amounts of compute time and power. With the constantly expanding architecture landscape and growing complexity of application runs, understanding how to improve performance is vital. In this thesis, we will use machine learning in two distinct ways to improve science applications. First, in an effort to understand future performance, we focus on predicting performance across multiple architectures. Cross-architecture metric mapping is heavily studied with hopes of understanding application performance on future or untested machine architectures. Often, there are months or even years spent on porting applications to new architectures, that may or may not provide an increase in performance. Here, we will use a data-driven approach to predict total throughput across different GPU architectures in order to understand future success of these applications. Our goal is to create a general framework that can predict application performance on a target hardware, given performance metrics from a different hardware architecture, without expert input. In this thesis, we propose such a framework and present two case studies of predicting total throughput and IPC between two GPU architectures. Second, we use a data-driven approach and leverage machine learning to understand and improve performance in high-performance computing applications. The goal of this work is to create a streamlined workflow of integrating machine learning surrogates into such applications. Using control theory to automatically and dynamically configure parameters, we can meet accuracy constraints while maximizing performance. This workflow, which we examine in the context of molecular dynamics simulations, will allow for faster and larger simulations by improving overall performance. By replacing typically high-cost functions, such as density functional theory calls or Hartree-Fock calls, with a low-cost machine learning inference call, our proposed workflow can reduce run-time while producing scientifically usable results. We create a decision engine that will automate finding the accuracy and performance trade-off relationship between using a high-fidelity, high-cost function call, and a lower fidelity machine learning inference.