KV-Cache as Attack Surface: Side Channels in Shared LLM Inference Infrastructure
About This Session
Most people think about LLM security at the prompt level. What goes in, what comes out. But there's an entire layer underneath, the KV-cache, and it has some properties that should make anyone running shared inference infrastructure genuinely uncomfortable.
Here's the problem: when a transformer model processes text, it doesn't recompute everything from scratch every time. It caches intermediate computations, the key-value pairs from the attention mechanism, so it can reuse them for efficiency. That's great for latency and cost. It's less great when multiple tenants share the same inference hardware, because those cached states don't always get cleaned up the way you'd expect.
This talk is about what happens in that gap.
We'll get into the mechanics of how KV-caches work, why they exist, and specifically where the isolation assumptions break down in multi-tenant deployments, including managed API services, on-premise inference clusters running vLLM or TensorRT-LLM, and enterprise platforms built on shared GPU pools. Then we'll look at timing-based techniques that let an observer on the same infrastructure infer whether specific content was recently processed, without ever seeing the content directly. If that sounds like a classic cache side-channel attack, that's because it is. The underlying physics haven't changed just because the workload is now a 70-billion parameter language model.
We'll walk through a concrete demonstration on a locally hosted open-weight model, nothing proprietary, nothing that requires a vendor's cooperation to reproduce. The goal is to show this isn't theoretical. It's measurable, it's repeatable, and the conditions for it exist in a lot of production deployments right now.
The second half of the talk pivots to what you can actually do about it. Cache partitioning, flush policies, inference isolation architectures, and, importantly, what questions to ask your infrastructure vendor or cloud provider, because most of them haven't published answers to these questions yet.
Here's the problem: when a transformer model processes text, it doesn't recompute everything from scratch every time. It caches intermediate computations, the key-value pairs from the attention mechanism, so it can reuse them for efficiency. That's great for latency and cost. It's less great when multiple tenants share the same inference hardware, because those cached states don't always get cleaned up the way you'd expect.
This talk is about what happens in that gap.
We'll get into the mechanics of how KV-caches work, why they exist, and specifically where the isolation assumptions break down in multi-tenant deployments, including managed API services, on-premise inference clusters running vLLM or TensorRT-LLM, and enterprise platforms built on shared GPU pools. Then we'll look at timing-based techniques that let an observer on the same infrastructure infer whether specific content was recently processed, without ever seeing the content directly. If that sounds like a classic cache side-channel attack, that's because it is. The underlying physics haven't changed just because the workload is now a 70-billion parameter language model.
We'll walk through a concrete demonstration on a locally hosted open-weight model, nothing proprietary, nothing that requires a vendor's cooperation to reproduce. The goal is to show this isn't theoretical. It's measurable, it's repeatable, and the conditions for it exist in a lot of production deployments right now.
The second half of the talk pivots to what you can actually do about it. Cache partitioning, flush policies, inference isolation architectures, and, importantly, what questions to ask your infrastructure vendor or cloud provider, because most of them haven't published answers to these questions yet.
Speaker
Saloni Garg
Senior Machine Learning Engineer - Adobe
Saloni Garg is a Senior Machine Learning Engineer at Adobe, where she specializes in AI and machine learning. She is also a recognized author in the field, with a forthcoming book on Retrieval-Augmented Generation (RAG) to be published by De Gruyter later this year. Saloni has made significant contributions to the academic community, evidenced by her multiple IEEE publications and a published patent.
