There is a long history in the security community of identifying privacy risks and attacks from the exposure of data to third parties. Over time, attacks have become more powerful and effective at extracting private data from ordinary observations. For much of this century, most privacy attacks on different data modalities and computing platforms have grown to rely on techniques driven by statistical machine learning (ML). Nevertheless, this landscape is changing, with the arrival of significantly different AI/ML models and architectures capable of extracting patterns and information once considered out of reach of statistical ML models. Ordinary data once thought to be ``squeezed'' dry are now revealing entirely new and unexpected results through the application of these advanced AI/ML techniques. This in turn requires us to reassess and redefine what we consider possible in the world of privacy attacks. To investigate this direction, this thesis primarily takes a traditional red team approach in the security domain. It first identifies new privacy attacks by integrating and adapting advanced AI/ML algorithms. It then discusses and proposes mitigation to address the newly identified threats. Specifically, the thesis discusses the process of fundamentally redesigning two well-known privacy attacks: keystroke inference and user interface confusion. The attack impacts are significantly enhanced by adapting advanced AI/ML techniques, such as self-supervised and self-attention learning, to the context of security. The above findings highlight the urgent threat posed by enhanced attacks and emphasize the need to study new and robust defenses. Along this line, this thesis also explores a new authentication scheme to defend against AI-enhanced biometric spoofing (e.g., deepfake). The scheme employs carefully crafted autoencoder and convolutional architectures to capture subtle physiological features of human muscles that are difficult to replicate.