Speech is an important signal of identity to others. The way someone speaks doubtless carries various information about that person. Drawing on sociolinguistic theories of listener design and common ground, this study further investigates the intricate role of conversation in social identity perception—positing that how one is spoken to also holds rich information about one's expected knowledge and experiences. Utilizing vignettes of a brief conversation, we empirically showed that speakers’ adaptations in their explanations—basic and neutral—signal the listener's familiarity with the subject matter. By manipulating whether the topic reflected higher or lower social class connotation, we showed that people readily infer the listener’s social class affiliation from how they are spoken to—drawing opposite conclusions across vignettes. The findings demonstrate how subtle linguistic elements can form perceptions of social identity in the eyes of a third party. Employing a nuanced analysis of conversational cues, this study contributes to the broader discourse on social signaling and highlights the complex ways social identities are navigated and negotiated in everyday interactions.