The strength of the Bayesian paradigm lies in its flexibility through hierarchical modeling and its ability to provide coherent uncertainty quantification. However, the computation costs of classical Bayesian procedures like Markov Chain Monte Carlo (MCMC) can be daunting when confronting big data challenges (large p or large n problems). This thesis innovates Bayesian methodology and theory with the help of modern machine learning techniques, to bring together the best of both worlds. In Chapter 1, I provide a general introduction of the advancement of machine learning methods, challenges in conducting inference with black-box machine learning methods, and an overview of the Bayesian methodology covered in this thesis. In Chapter 2 and Chapter 3, I investigate the integration of machine learning techniques and Bayesian computation in case where the likelihood is implicit or intractable. I developed two summary-free Approximate Bayesian Computation (ABC) approaches. The first approach adopts the "classification trick" to estimate the KL divergence between the simulated and observed data. The second approach directly targets at the posterior distribution by matching the joint distribution of the parameter and the data via conditional generative adversarial networks (cGANs). I study the theoretical guarantees as well as methodology of Bayesian neural networks in Chapter 4 and Chapter 5. Their expressiveness and generalizability has motivated me to deploy deep neural networks inside Bayesian algorithms. This combination not only benefits from the power of neural networks but also retains the inferential potential of the Bayesian probabilistic structure. I tackle the classical problem of variable selection in the context of ensemble tree-based regression in Chapter 6. To encourage more competition among variables, we place a spike- and-slab wrapper outside the sum-of-trees prior and propose to solve the computation with ABC techniques.