This dissertation delves into statistical inverse problems with a focus on Bayesian approaches for parameter estimation and uncertainty quantification under sparsity, nonconvexity, and geometric constraints. The dissertation covers innovative methodologies for addressing these challenges across various contexts, including compressed sensing, dynamical systems learning, and parameter estimation of differential equations on Euclidean space and manifolds. The work encompasses various methodologies based on mean-field variational inference, ensemble Kalman methods, Bayesian optimization, and graph Gaussian process to obtain point estimates for the quantity of interest as well as comprehensive uncertainty quantification associated with it. The dissertation effectively showcases how the introduced methods improve computational efficiency and accuracy in parameter estimation and uncertainty analysis across complex models. This is achieved through a blend of theoretical insights and numerical studies, inspired by a wide array of practical scenarios.