Characterizing Effect Heterogeneity in Multi-Site Trials: An Application to the Evaluation of Charter Schools

Multi-site randomized control trials are increasingly prevalent in the social sciences, particularly for studying the effects of educational interventions. This paper synthesizes and evaluates approaches to char- acterizing how the effects of an intervention vary across sites and individuals, extending beyond the typical approach of estimating an average treatment effect. Working with data from a nationwide evaluation of charter schools, we deploy existing empirical Bayes and fully Bayesian methods to further an understanding of the distribution of site-specific effects. We then use randomization-based methods to understand the distribution of individual-level effects within sites. The paper provides a novel comparison of the application of model-based and randomization-based inferential methods to the study of a multi-site experiment in edu- cation. Ultimately, this paper continues an ongoing push for a focus on heterogeneity in causal inference and introduces methods that grant researchers the ability to study individual-level variation alongside site-level variation.