Factor models have been widely used in recent years to model high-dimensional spatio-temporal data. However, the validity of employing factor models in a specific application has received less attention. This article proposes test statistics for testing the symmetry in cross-correlation matrices of a high-dimensional stochastic process implied by exact factor models. A rejection of symmetry indicates that the use of an exact factor model is questionable. Both simulations and real examples are used to demonstrate the applications and to study the finite-sample performance of the proposed test statistics. Empirical results show that the proposed test statistics are effective in identifying cases where exact factor models are not appropriate, providing valuable guidance for choosing factor models in a high-dimensional setting.