Memory-Time Tradeoffs in LLM Inference: A Practical KV-Cache Experiment

Summary

An architectural KV-cache experiment that retained 16.5% of the baseline cache footprint and increased measured throughput by 27% in a high-concurrency Qwen 0.5B stress test, while revealing limitations of blunt token eviction.

1. Hitting the Memory Limit

If you've spent time hosting LLMs, you quickly realize that the main bottleneck usually isn't compute; it's memory. Engines like vLLM are incredibly fast, but they save every single generated token in the GPU's KV Cache. This takes up roughly ~250MB of VRAM per user for a 10k-token context. On an NVIDIA L4 (24GB), you usually max out at around 40-150 concurrent users before the system runs out of memory and has to start pausing requests.

This got me curious: What actually happens if we just dynamically delete the memory pointers for low-signal tokens while the model is still generating?

2. Prior Work & The Implementation Gap

The idea of dropping tokens isn't new. StreamingLLM (Xiao et al., 2023) showed that keeping "Attention Sinks" (the first few prompt tokens) stops the model from completely breaking. H2O: Heavy-Hitter Oracle (Zhang et al., 2023) also found that you can drop tokens with low "attention mass" and the model still performs pretty well.

The tricky part is actually putting this into a production engine. Sparse eviction creates "holes" in the KV cache, which messes with the contiguous memory requirements of dense matrix operations like FlashAttention. Plus, managing all this memory cleanup in Python usually introduces so much overhead that it negates the benefits. I wanted to see if I could wire up the systems engineering to make this practical.

3. The Setup: A Quick Prototype ("Nitro")

To test this, I hacked together a custom inference pipeline with a few specific components:

1. The Triton Data Plane: I wrote a custom OpenAI Triton kernel that runs a simple gravity-inspired physics scoring directly on the GPU SRAM. It scores tokens on the fly and masks out the filler words before they even hit VRAM.
2. The IPC Bridge: To get the GPU to talk to the CPU scheduler without locking up PyTorch CUDA Graphs, I routed the data through an asynchronous /dev/shm RAM disk.
3. The Control Plane (Linear hash sweep): A hybrid C++/Python garbage collector cleans up the abandoned pointers.
   • A quick note on bottlenecks: My first try used a nested Python loop (O(N³)), which was terribly slow and added a 38% latency penalty due to the GIL. I replaced the nested Python scan with an O(N) sweep using average-O(1) hash membership checks, bringing the overhead down to < 2ms per step.

4. Some Initial Benchmarks

I ran a stress test on a single NVIDIA L4 (24GB) using Qwen 0.5B. I pushed 4.1 million tokens across 2,048 concurrent requests and uncapped vLLM's max_num_seqs limit to see what would happen.

A Neat Side Effect: Throughput Gains

The speed increase was honestly a bit of a surprise. The reduced cache pressure also improved throughput under this stress configuration. I have not yet isolated whether the gain came from reduced scheduler preemption/recomputation, memory pressure, or data movement, so the exact mechanism requires further profiling.

5. Text Quality and Attention

Going into this, I was pretty worried that dropping ~83% of the KV cache would give the model severe amnesia. Interestingly, the text quality actually seemed to improve slightly.

Smaller models tend to get "distracted" over long contexts—their attention spreads too thin across thousands of tokens, which often leads to them getting stuck in repetitive loops. By aggressively filtering the cache to mostly just nouns and core logic, it essentially forced the model to focus. In my sample data, the unique-to-total word ratio went up (0.0972 vs 0.0723), and it fell into repetitive loops significantly less often.

6. What's Next: Fixing the "Swiss Cheese" Problem

While the prototype frees up memory, it currently suffers from internal fragmentation. Because my Triton kernel currently operates by masking things out at the block level, if a 16-slot block has even two good tokens left, the whole block stays allocated. The memory ends up looking a bit like Swiss cheese.

To fix this, the next logical step is cross-block "stitching." I plan to use a Parallel Prefix Sum (cumsum) over the mask to calculate dense destination indices, allowing me to pack surviving tokens tightly together across logical block boundaries. If I can get that working, it should completely eliminate the spatial fragmentation.

Addendum: Re-evaluating the "Semantic Amplifier" Effect

Upon further analysis, the observed reduction in linguistic collapse and increase in lexical diversity is likely an artifact of model scale (Qwen 0.5B) rather than a universal feature of KV eviction.

Small parameter models suffer from high attention entropy—they distribute attention "blurrily" across context, accumulating noise. Truncating the bottom 80% of the cache acted as a blunt regularizer, forcing the model to focus. However, in frontier models (>70B parameters), low-attention tokens often carry subtle, highly specific syntactic weight. Blunt eviction at scale would likely destroy reasoning coherence. This limitation necessitates a shift from blunt deletion to empirical compression (semantic pooling) for future iterations.