Sensory inputs from the natural world always come with certain environments, and our visual perception is profoundly influenced by these contexts. Contextual modulation is a ubiquitous phenomenon in visual perception and it underlies different phenomena such as visual saliency and adaptation aftereffects. These perceptual attributes have been linked to contextual modulation of visual neuronal responses: the activity of neurons to a target stimulus can be shaped by its visual environments. Contextual modulation arises early in the visual pathway and is readily detectable in multiple types of retinal ganglion cells including the On-Off direction-selective ganglion cells (DSGCs). On-Off DSGCs are the output neurons of the well-studied direction-selective circuit, and the neural networks wired to these cells are one of the best-known circuits in the retina. However, the neural mechanisms underlying the contextual modulation of DSGCs are not fully understood. Delineating the pattern and mechanism of contextual modulation in DSGC will give us a more comprehensive understanding of the retinal code that DSGC conveys to the brain. Therefore, the central goal of this thesis was to identify how DSGC responses are modulated by visual contexts in both spatial and temporal dimensions, and to investigate how these contextual effects are accomplished at the synaptic and network levels. First, from the spatial perspective, I determined how DSGC responses are modulated by different patterns of surrounding visual stimuli. I found that DSGCs are sensitive to discontinuities of moving contours between the stimuli in the center and the background. Using a combination of synapse-specific genetic manipulations, patch-clamp electrophysiology and connectomic analysis, we identified distinct circuit motifs that are required for spatial contextual modulation of DSGC activity for bright and dark contours. Furthermore, our results revealed a class of wide-field amacrine cells (WACs) with long, straight and unbranched dendrites that may function as "continuity detectors" of moving contours in the retina. Second, from the temporal perspective, I examined the influence of previous visual stimuli on DSGC responses. I found that DSGC responses can be acutely enhanced by prior visual stimulation. Moreover, DSGCs from the dorsal and ventral retina, which receive visual inputs from the lower and upper visual fields respectively, show divergent patterns of sensitization. After a full set of patch-clamp electrophysiology, pharmacology, anatomical reconstruction and simulation computational modeling, our results revealed that glycinergic disinhibition of Off bipolar cell inputs to DSGC plays an essential role in the DSGC sensitization. Furthermore, DSGCs from the dorsal retina gain a sustained membrane depolarization from the Off pathway, which propagates into the On pathway via “crossover dendrites” between layers and sensitizes the subsequent On spiking activity. Overall, these results link the existing circuit motifs mediating direction selectivity to the extensive inner retinal network, highlighting a multilayered circuit architecture that dynamically engages specific microcircuit motifs and synaptic plasticity to process motion information according to visual contexts. Results from my thesis therefore fill the knowledge gaps in the mechanistic understanding of contextual modulation in the visual system, and also lead to a deeper understanding of motion computation in complex natural environments.