This thesis consists of four research papers stapled together. In this work, we study moduli spaces of principally polarised abelian varieties of dimension~$g > 1$ with~$p$-torsion structure for prime~$p$. In particular, given a Galois representation~$\overline{\rho}: G_{\mathbb{Q}} \rightarrow \mathrm{GSp}(2g,\mathbb{F}_p)$ with cyclotomic similitude character, we study various rationality aspects of the twist~$\mathcal{A}_g(\overline{\rho})$ of the Siegel modular variety~$\mathcal{A}_g(p)$ of genus~$g$ and level~$p$. Using a description of the cohomology of the compactification~$\mathcal{A}_2^*(3)$ given by Hoffman and Weintraub, we show that the variety~$\mathcal{A}_2(\overline{\rho})$ is not rational in general. When~$\overline{\rho}$ is surjective, the minimal degree of a rational cover is~$6$. Boxer, Calegari, Gee, and Pilloni have shown the existence of a rational cover~$\mathcal{A}_2^w(\overline{\rho})$ of degree~$6$. We find explicit formulae parametrizing the pullback $\mathcal{M}_2^w(\overline{\rho})$ of $\mathcal{A}_2^w(\overline{\rho})$ under the Torelli map $\mathcal{M}_2 \rightarrow \mathcal{A}_2$. This describes the universal family of genus~$2$ curves with a rational Weierstrass point, having fixed~$3$-torsion of Jacobian. This exploits Shioda's work on Mordell-Weil lattices and the invariant theory of the complex reflection group~$C_3 \times \mathrm{Sp}_4(\mathbb{F}_3)$. We also outline how similar results can be obtained for~$(g,p)=(2,2),(3,2),(4,2)$. By making use of the modularity lifting theorem for abelian surfaces proved by Boxer, Calegari, Gee. and Pilloni, we produce some explicit examples of modular abelian surfaces~$A$ with~$\mathrm{End}_{\mathbb{C}}(A) = \mathbb{Z}$. Using the explicit formulae describing families of abelian surfaces with fixed~$3$-torsion, and transferring modularity in the family yields infinitely many such examples. When~$g = 1$ and~$p > 5$, the existence of mod-$p$ Galois representations not arising from elliptic curves over~$\mathbb{Q}$ is known. For~$g > 1$ and~$(g,p) \ne (2,2), (2,3), (3,2)$, we investigate a local obstruction to the existence of rational points on~$\mathcal{A}_g(\overline{\rho})$, and thus construct Galois representations~$\overline{\rho}: G_{\mathbb{Q}} \rightarrow \mathrm{GSp}(2g,\mathbb{F}_p)$ with cyclotomic similitude character, that do not arise from the~$p$-torsion of any~$g$-dimensional abelian variety over~$\mathbb{Q}$. This is accomplished by solving embedding problems with local conditions at suitably chosen auxiliary primes~$l \ne p$, with the help of Galois cohomological machinery.