Language modality, the form in which information is conveyed, has been found to influence perspective taking and theory of mind. Testing accidental and intentional harm stories from the harm doer’s perspective, we broaden previous research by considering the role that language modality plays in feeling empathy towards the accidental and intentional harm doer. We presented 277 native English speakers with spoken or written accidental and intentional harm scenarios and measured their empathy towards the agent and the victim, as well as their willingness to forgive the agent, and their desire to socially connect with the agent. For accidental harm, spoken communication, as compared to written communication, resulted in lower empathy towards the accidental harm doer but similar empathy towards the victim, a lower level of forgiveness, and a decreased desire to connect with the agent. For intentional harm, spoken communication resulted in lower empathy towards the agent and higher empathy towards the victim, a lower willingness to forgive the agent, and a decreased desire to connect with the agent. Taken together, the results suggest that listening as compared to reading accidental and intentional harm systematically influences moral reasoning. This could have implications for how people choose to communicate their wrongdoings to a third party when attempting to win people to their side.