Privacy and Security in Responsive Environments

Private and secure systems require thorough evaluations to ensure that they are providing adequate protection while maintaining the utility of the underlying system. However, these evaluations frequently neglect to account for the ways in which the defense itself affects the environment in which it is deployed. In this thesis, we investigate this pitfall in the study of three defense techniques: adversarial training, machine unlearning, and differential privacy. In one work, we show the limitations of assuming a fixed adversary when training adversarially robust models and design a theoretically-motivated training scheme that provides protection when adversaries are allowed to modify their attacks to evade existing defenses. In another, we relax the assumption that deletion requests are independent when individuals are allowed to delete their data from AI models and explore how different user behaviors can impact the utility of downstream models. Finally, we examine the US Census Bureau’s implementation of differential privacy in the 2020 census and ask whether the addition of privacy preserving noise will make it more difficult for states to comply with federal redistricting law. These contributions highlight the subtle ways in which private and secure systems alter the environments in which they are deployed.