Why Reddit LLM Embeddings Fail to Predict Volatility: A Diagnostic Framework

This thesis investigates whether large language model (LLM) embeddings of Reddit's r/WallStreetBets posts can improve forecasts of next-day realized stock volatility beyond a heterogeneous autoregressive (HAR) benchmark. We construct a daily stock-day panel covering 14,080 equities from April 2012 to December 2022 and extract embeddings from both a general-purpose LLM (LLaMA-3.1-8B) and a finance-tuned variant adapted via LoRA on SEC 10-K and 10-Q filings. Using a rolling-window design with eight out-of-sample test years, we find that embeddings achieve a standalone out-of-sample R^2 of approximately 0.15, but combining them with the HAR model yields negligible incremental gains (R^2 improvement < 0.002), establishing a clear empirical ceiling for this data source. To diagnose this failure, we adapt layer-wise linear probing from the NLP interpretability literature to financial prediction, providing what is to our knowledge the first application of this technique to realized volatility forecasting. The layer-wise R^2 curves reveal that volatility-relevant information accumulates monotonically through the transformer's 33 layers, concentrating in the final layers for both model variants. Building on these probes, we develop a diagnostic framework comprising probe-guided residual aggregation, probe-augmented HAR, residual-targeted probes, and reverse prediction. These experiments systematically rule out methodological explanations and uncover a striking puzzle: residual probes confirm that the embedding contains zero orthogonal information beyond HAR, yet reverse prediction shows that HAR features explain only 3\% of the embedding score's variance. The resolution is that the embedding encodes a large body of market-adjacent content, of which only a small fraction correlates with volatility, and that fraction is entirely subsumed by historical price patterns. We further show that domain-specific fine-tuning reorganizes the embedding space into financially coherent clusters but does not improve aggregate prediction, demonstrating that the bottleneck lies in the informational content of the text rather than in the model's representational capacity.