Whether there are imminent physical limits to economic growth has been the focus of a long-standing debate, often (but not exclusively) pitting natural scientists against economists. One side of the debate posits that finite resources constrain human economic growth, the other that productivity improvements and substitution can relax those resource constraints. However, the two sides lack a common framework to resolve their disagreement. We provide this framework and mediate this dispute using a combination of low-order economic and physical models, focusing especially on the waste heat limit. To start, we use a simple accounting identity to show that historical rates of growth in average productivity can support modest economic and population growth (totaling 1.2%). We model the future trajectory of productivity growth by combining a canonical production function (Cobb-Douglas) with a flexible model where population growth drives total-factor-productivity growth. We find remarkable parameter sensitivity: across a plausible range of parameters waste heat either never constrains economic growth in the model or does so in a few hundred years. Interestingly, either decreasing or increasing population growth can dramatically extend the time to heat constraint in the model, the latter due to increases in innovation. Finally, we use a more flexible (constant elasticity of substitution) production function and historical changes in the relative use of different resources to show that humans have the ability to substitute away from resources in shorter supply. Although there are vast uncertainties in this type of projection, our work highlights the potential for an optimistic future for humanity in which economic growth continues on a millennial timescale.