Rethinking AI Systems Through Efficient Model Communication

This shift greatly changes how inference systems should be designed and implemented. In the classic machine learning era, the interface with a model was often a human-readable object, such as text, images, audio, or video, because a human was commonly the final consumer. In modern AI systems, however, interactions with models often do not involve humans at all: model outputs are consumed by other inference requests, models, or software control logic. This opens the possibility of interacting with models through model-native state instead of only human-readable media, such as passing an LLM's key-value (KV) cache between inference components or converting software context into a differentiable loss function that a model can optimize. When forced to communicate through human-readable data, AI systems may repeat computation, increasing response delay, or miss what downstream applications need, reducing decision quality.

This dissertation argues that, in these settings, the right media for interacting with AI models are model-native states: internal or directly model-digestible states that incorporate a model's understanding of the input more directly than human-readable data. Once AI systems communicate through such states, the central systems problem becomes how to store, move, transform, and specialize them without losing the information that makes them useful to the model. I study this problem through two forms of model-native state: the key-value (KV) cache in large language model (LLM) inference, which represents input contexts in a form the model can directly consume, and model behavior specialized to how software uses the model's outputs.

In this thesis, I first focus on KV cache. Traditionally, KV cache was used only within a single inference request and kept entirely in GPU RAM.  Today, however, KV cache increasingly exceeds available GPU memory, creating the need to move it out of the GPU and store it in CPU DRAM or on disk. This shift creates a loading challenge: before a model can use persisted KV cache, the system must fetch it back into GPU RAM, and that transfer can incur substantial latency. I present CacheGen, a system that makes KV cache practical to fetch and load from secondary storage devices by encoding it into compact bitstreams and streaming it. A second challenge is interoperability: because KV cache is a model-specific internal encoding of the input, it cannot be directly reused by another model. I then present DroidSpeak, a system that enables KV cache reuse across different models.

Finally, I show that model-native state is useful beyond LLM KV cache.  ChameleonAPI considers ML APIs that expose generic human-readable labels and scores even though applications care about whether those outputs lead to correct software decisions. ChameleonAPI extracts how an application uses model outputs in control flow and converts that software context into a model-native training objective, allowing the model to internalize which errors matter for the application. Together, these systems demonstrate that the next generation of AI infrastructure should manage not only models, prompts, and outputs, but also the model-native states through which AI systems communicate and reuse computation.