This dissertation consists of two parts that address problems arising frequently in the analysis of big, high-dimensional dependent data. The first part proposes a new approach for model selection when the response variable contains unit roots with unknown multiplicities at unknown locations. The proposed method, FHTD, is based on the forward stepwise regression technique. Despite of unit roots, high-dimensional predictors, and conditionally heteroscedastic errors, FHTD is shown to select exactly the correct model with probability tending to one. Thus, our approach is applicable to a wide range of data without recourse to any delicate unit-root tests. The second part tackles the computational problem when the data are vertically partitioned and stored across computing nodes. To jointly learn a multivariate linear model, we propose a two-stage relaxed greedy algorithm so that communication between computing nodes is minimized and hence the algorithm is computationally efficient. Throughout, we supply simulation studies and real data examples to demonstrate the performance of the proposed methods.