We prove that in $(3+1)d$ quantum field theories with $Z_N$ symmetry, certain anomalies forbid a symmetry-preserving vacuum state with a gapped spectrum. In particular, this applies to discrete chiral symmetries which are frequently present in gauge theories as we illustrate in examples. Our results also constrain the long-distance behavior of certain condensed matter systems such as Weyl-semimetals and may have applications to crystallographic phases with symmetry protected topological order. These results may be viewed as analogs of the Lieb-Schultz-Mattis theorem for continuum field theories.