Social categorization has vast implications for myriad aspects of human social life, and studying its origins and development can inform our understanding of its pervasive influences across the lifespan. While a growing body of research has found early-emerging social preferences for in-group members, first person social preferences may arise due to familiarity and therefore do not necessarily indicate abstract conceptual reasoning about social categories. In this dissertation, I use third-party violation of expectation looking time studies to investigate infants’ inferences about social groups in the first two years of life. In Part I, I demonstrate that infants expect people who speak the same language to be more likely to affiliate than people who speak different languages, suggesting they may see language as a fundamental marker of social group. In Part II, I ask whether infants use other socially relevant behaviors, such as imitation of actions that may be seen as rituals, to make inferences about third-party social relationships. Here, I find that infants expect people who engage in the same causally-irrelevant actions to be more likely to affiliate than people who engage in different causally-irrelevant actions, and that these effects are not merely due to perceptual similarity. In Part III, I ask whether infants are selective in how they generalize socially relevant attributes across people. Results suggest that although infants’ baseline expectation is that food preferences are generalizable, they withhold generalizing food preferences across people who seem to belong to different social groups. In the general discussion I integrate this work with other broader research from developmental psychology to advocate for a new definition of social categorization that does not rely on first person social preferences and instead makes central inferences about social structure and inductive generalization. Taken together, this work indicates that infants demonstrate early emerging abilities to think about people as members of social groups, and provides novel insights into the origins of social categorization.