Living in cities affords expanded access to various resources, infrastructures, and services at reduced travel costs, which improves social life and promotes systemic gains. However, recent research shows that urban dwellers also experience inequality in accessing urban facilities, which manifests in distinct travel and visitation patterns for residents with different demographic backgrounds. Here, we go beyond simple flawed correlation analysis and reveal prevalent accessibility gaps by quantifying the causal effects of resident demographics on mobility patterns extracted from U.S. residents’ detailed interactions with millions of urban venues. Moreover, to efficiently reveal micro neighborhood-level accessibility gaps, we design a novel Counterfactual RANdom-walks-based Embedding (CRANE) method to learn continuous embedding vectors on urban mobility networks with confounding effects disentangled. Our analysis reveals significant income and racial gaps in mobility frequency and visitation rates to sports and education venues. Besides, bachelor’s degree holders experience greater mobility reduction during the COVID-19 crisis. With extensive experiments on neighborhood-level accessibility prediction and visualizing accessibility gaps with embeddings vectors, we demonstrate that the counterfactual mobility network embeddings can improve the explanatory capacity and robustness of revealed accessibility gaps by extending them from aggregate statistics to individual neighborhoods and allowing for cross-city knowledge transfer. As such, urban mobility networks can reveal consistent accessibility gaps in the U.S., calling for urgent urban design policies to fill in the gaps.