We present a theoretical and empirical study of feedforward neural networks using tropical algebra and topological data analysis. We show how examining neural networks through the lens of these two disciplines yields insights into their operation and efficacy. This work is divided into two parts: \emph{Topology of Deep Neural Networks} and \emph{Tropical Geometry of Deep Neural Networks}. Each part is respectively a self-contained analysis of deep neural networks from the perspectives of algebraic topology and of tropical algebra. There is a noteworthy connection between the two parts: One of our conclusions from the first part is that it is important to bound the topological complexity of decision boundaries; the work in the second part, among other things, provides such a bound in terms of the number of linear regions. The first part of this thesis is joint work with Liwen Zhang and Lek-Heng Lim and has appeared as a paper in ICLM conference. The second part is joint work with Andrey Zhitnikov and Lek-Heng Lim and has been submitted for publication.