We develop a model for correlations of cosmic microwave background (CMB) anisotropy on the largest angular scales, based on standard causal geometrical relationships in slow-roll inflation. Unlike standard models based on quantized field modes, it describes perturbations with nonlocal directional coherence on spherical boundaries of causal diamonds. Causal constraints reduce the number of independent degrees of freedom, impose new angular symmetries, and eliminate cosmic variance for purely angular 2-point correlations. Distortions of causal structure from vacuum fluctuations are modeled as gravitational memory from randomly oriented outgoing and incoming gravitational null shocks, with nonlocally coherent directional displacements on curved surfaces of causal diamonds formed by standard inflationary horizons. The angular distribution is determined by axially symmetric shock displacements on circular intersections of the comoving sphere that represents the CMB photosphere with other inflationary horizons—those centered on it, and those that pass through an observer's world line. Displacements on thin spheres at the end of inflation have a unique angular power spectrum $C_\ell$ that approximates the standard expectation on small angular scales, but differs substantially at large angular scales due to horizon curvature. For a thin sphere, the model predicts a universal angular correlation function $C(\Theta)$ with an exact 'causal shadow' symmetry, $C(\pi/4\lt\Theta\lt3\pi/4) = 0$, and significant large-angle parity violation. We apply a rank statistic to compare models with WMAP and Planck satellite data, and find that a causally-coherent model with no shape parameters or cosmic variance agrees with the measured $C(\Theta)$ better than a large fraction (${\gt}0.9999$) of standard model realizations. Model-independent tests of holographic causal symmetries are proposed.