This dissertation explores compression techniques for neural networks to enable control of resource usage, accelerate training, and increase robustness against adversarial as well as out-of-distribution examples. The convolutional layers are core building blocks of neural network architectures. In general, a convolutional filter applies to the entire frequency spectrum of the input data. Our new band-limiting method artificially constrains the frequency spectra of these filters and data during training and inference. The frequency-domain constraints apply to both the feed-forward and back-propagation steps. Experimental results confirm that band-limited models can effectively control resource usage (GPU and memory). The band-limited method with 50% compression in the frequency domain results in only a 1.5% drop in accuracy for the ReNet-18 model trained on CIFAR-10 data while reducing the GPU memory usage by 40% and the computation time by 30% in comparison to their full-spectra counterparts. The models trained with band-limited layers retain high prediction accuracy and require no modification to existing training algorithms or neural network architectures, unlike other compression schemes. Since band-limited models naturally reject high-frequency noise, they are useful for studying the adversarial robustness of neural networks. I show that band-limited models are actually one instance of a broader family of adversarial defenses called perturbation-based defenses. These defenses, whether random or deterministic, are essentially equivalent in their efficacy, and all share similar weaknesses to adaptive attacks. I also illustrate applications to out-of-distribution robustness of models in terms of their generalization and detection of anomalous inputs in the NLP (Natural Language Processing) domain. The band-limited models have practical applications in a number of time-series and signal processing domains. I present a new solution for a fair wireless co-existence between Wi-Fi Access Points (APs) and LTE-U (Long Term Evolution-Unlicensed) Base Stations (BS) using band-limited neural networks and demonstrate that such networks can accurately learn distinct patterns for the energy distributions in Wi-Fi AP transmissions. Our method results in higher accuracy (close to 99% in all cases) as compared to the existing auto-correlation (AC) and energy detection (ED) approaches.