We introduce a novel family of Gaussian Processes (GPs) kernels, derived from the limit of graph neural networks (GNNs) when their layers become infinitely wide. We start with a prominent example of GNNs, the graph convolutional networks (GCNs), and establish the corresponding GP kernel. We conduct an in-depth analysis of the kernel universality and the behavior in the limit of depth. To address computational efficiency, we propose a low-rank approximation of the covariance matrix, enabling scalable posterior inference with large-scale graph data. Our experiments reveal that the kernel achieves competitive classification and regression performance on diverse graph datasets, while offering significant advantages in computation time and memory efficiency. Expanding on this, we derive various members of this kernel family by extending the equivalence to several interesting GNN architectures. We consider their limits in depth, and demonstrate how to efficiently compose these kernels from the building blocks of GNNs. Our results show that these GNN-based kernels effectively capture rich graph information from graph-structured data, achieving superior performance compared to their GNN counterparts. We further delve into the dynamic evolution of these kernels during the training process. Specifically, we introduce the Graph Neural Tangent Kernel (GNTK), representing the kernel obtained at the end of training. We show that the GNTK can be computed efficiently using our proposed methods and shares similar performance metrics with GNNGP kernels. Finally, we conduct a comprehensive performance analysis, investigating how large GNNs scale with respect to model size, dataset size, and training cost. We derive and empirically validate the scaling laws governing these models, providing useful guidance for designing and training scalable GNNs for various graph applications.