Deep Generative Models: Design, Improvements and Applications

Deep generative models (DGM) combine the deep neural networks with generative models to learn the underlying generation mechanism of the data of interest. This has emerged as a important approach to extracting knowledge from data in machine learning and artificial intelligence. However, there are many challenges to learning and applying DGMs in different domains despite their promising potential. Therefore, this thesis focuses on understanding, improving, and applying different deep generative models. First, we introduce the underlying principles under different DGMs, including the Variational Auto-Encoder (VAE), the Flow-Based Model, the Generative Adversarial Network (GAN), and the Energy-Based Model (EBM). We also propose a novel counterpart of VAEs: Variational Latent Optimization (VLO), which does not require an encoder structure. Besides, we provide a new angle to understand the generation process of EBMs, build a connection between EBMs and GANs, and design a new approach to improve the sample quality of EBMs from that. Next, we propose two hybrids of DGMs to improve the generation quality of current models. First, we combine Flow-based models and Variational Auto-Encoders to improve the generation quality of Auto-Encoder-based generative models. Second, we borrow the idea of exponential tilting and combine the energy-based model with other likelihood-based generative models to obtain better samples. In the end, we conduct various applications related to modern deep generative models, including using generative models as likelihood-based methods for out-of-distribution (OOD) detection and designing controllable generative models over human faces. We propose a novel OOD-detect score called likelihood regret to help detect OOD samples with VAEs. Also, we propose to add a new structure to current key-points-based face reenactment models and combine them with the 3D morphable model to improve their generation quality and generalization ability.