We examine differences in mobility outcomes between residents of highest and lowest socio-economic index (SEI) at the Census block group (CBG) level in nine major US cities prior to and during the COVID-19 pandemic. While low-SEI groups generally traveled shorter distances but visited more city-wide CBGs before the pandemic, high-SEI residents universally reduced their mobility to a greater extent during the pandemic. Although high-SEI residents were making more trips to parks and health-care providers, and fewer subsistence trips to retail stores already before the pandemic, COVID-19 significantly widened these differences thereby exacerbating “mobility gaps” between low-SEI and high-SEI groups. We further examine how such “mobility gaps” can be mitigated by spatial advantages of home locations, controlling for political inclination. We find that living in better transit-served or more walkable neighborhoods generally benefited high-SEI residents more than low-SEI residents, with some variation across cities. This suggests that built environments not only impact mobility outcomes during “normal” times, but also influence how different socio-economic groups are able to adapt during times of crisis.