The issue of interpretability for machine learning algorithms is vital to the natural science community. If the output of machine learning procedure is fully interpretable, natural scientists can directly brought conclusions from machine learning algorithms applied to their experimental data. In this thesis, I use Bayesian machine learning techniques to develop two new variable selection methods and present two new applications to the control of biological transcription. With respect to these applications, I introduce a class of fully interpretable Deep Neural Networks (DNN) and a hierarchical Bayesian method for parameter inference. The two variable selection algorithms make use of Approximate Bayesian Computation (ABC) for variable selection in the context of Bayesian forests. I proved that the spike-and-slab forest prior is consistent for model-free variable selection and proposed an ABC algorithm that efficiently samples from the posterior. I further improve the speed of this algorithm using Thompson sampling. In the next part of the thesis, I consider interpretability in the context of biological control of transcription. I first introduce a fully interpretable DNN that implements a model for the control of transcription in Drosophila. This DNN has the unusual property that every layer is full interpretable in a sense that specifically implements a prior. I next consider a Bayesian method for parameter inference in a model of stochastic burst of transcription.