In the thesis, we study the problems regarding robustness and model adaptivity with stochastic optimization. First, we formally address two robust concerns. 1. Finite sample cannot well represents the entire population. 2. Data modeling assumptions can be wrong (misspecified). For the first robust concern, we propose an alternative of the popular regularization method based on distributionally robust optimization and clarify their connection and derive finite dimensional computational formulation based on that. For the second robust concern we study Huber’s loss within a modern non-asymptotic setting. We further study the second robust concern with the stochastic gradient descent algorithm and purpose how to amend SGD to defense possibly maliciously outlier attack (which can consider as a stronger version of second robust concern) and justify the statistical optimality. We then study the model adaptivity of training neural network by gradient flow via a dynamic reproducing kernel Hilbert space (RKHS) approach. We show that when reaching any local stationarity, gradient flow learns an adaptive RKHS representation and performs the global least-squares projection onto the adaptive RKHS simultaneously. This approach gives intuition of the benefits of training neural network over only viewing the neural network as a neural tangent kernel.