Language is traditionally characterized as an arbitrary, symbolic system, made up of discrete, categorical forms. But iconicity and gradience are pervasive in communication. For example, in spoken languages, word forms can be “played with” in iconic gradient ways by varying vowel length, pitch, or speed (e.g., “It’s been a loooooooong day”). These instances of depiction have been called “vocal gesture”. However, little is known about this process in sign languages, wherein both linguistic and gestural representations exist within the same modality . In this work we explore the gradient use of motion in American Sign Language (ASL) and silent gesture. In Study 1 we asked deaf signers of ASL (n = 11) to describe an event manipulated along speed, direction, or path, and observed their use of gradient modification in arbitrary and iconic signs. We found that signers alter the forms of both lexical and depicting signs to enhance meaning. However, the three motion dimensions were not modified equally in lexical signs, suggesting constraints on gradient modification. These constraints may be linguistic in nature, followed only by signers who know ASL or another sign language, or they may be a feature of manual or gestural communication more broadly. To test, in Study 2 we compared the ASL signers to English-speakers (n=11) asked to communicate silently using gesture. As in Study 1, the silent gesturers are given an ASL verb and asked to use it to describe a target event and three variants. We observe whether the target verb is modified to depict the variants. We find both similarities and differences between the groups. The groups are both influenced by the iconicity of these signs, suggesting that intuitions about modification may incorporate some gestural knowledge, since silent gesturers, unfamiliar with ASL, can generate them. However, the groups differed in their overall tendency to modify, as well as their treatment of the different dimensions of movement, indicating that there is more to these modifications than iconicity, and illustrating how iconicity and arbitrariness work together in linguistic system. In Study 3, we test deaf children using ASL (n=38) as well as hearing English-speaking children using silent gesture (n=15) on an adapted version of this paradigm. We find that these intuitions regarding how modification fits within the phonological constraints of ASL emerge early in development for children acquiring a sign language, but these patterns are not shared by children for whom manual communication is not linguistic (silent gesturers).