Health disparities in cognition at older ages may result, at least in part, from cumulative disadvantages from childhood, including characteristics of the family. This paper draws on resource dilution theory to develop hypotheses about the presence of siblings that I test with data from the Health and Retirement Study linked to the 1940 U.S. Census. Using latent growth curve models with reweighting weights by the childhood family and contextual covariates, I estimate the long-term causal influence of sibship. A fully-adjusted model shows that having co-residing siblings in childhood is associated with faster cognitive decline in late adulthood but not with baseline cognitive functioning. The results also show heterogeneities by siblings’ gender composition, with older men with only male siblings showing the worst baseline cognitive function in later life, whereas older women with sisters only face steeper cognitive decline. The paper suggests resource dilution between siblings may cumulatively influence cognitive aging.