Empirical SRAM Gating and "Semantic Ghosts"

While dropping tokens fixed the memory issue, I realized that permanently deleting 80% of the KV cache is a blunt instrument. If a highly specific detail (like an API key) gets swept up in the deletion, the model gets silent amnesia. Also, my previous masking approach left empty "holes" in the physical memory blocks (internal fragmentation).

Instead of trying to stitch those holes together, I'm currently working on a V2 architecture that changes the paradigm from Dropping to Empirical Compression.

The Chunked Prefill Phase

The new approach operates directly inside the Chunked Prefill phase:

1. The Gossip Accumulator: Instead of guessing importance with a separate function, I'm modifying the FlashAttention kernel to sum the actual Softmax probabilities column-wise. The model's own attention dictates the "Gravity" of a token.
2. Dynamic Gating: The threshold is no longer hardcoded; it scales dynamically based on the block's mean attention.
3. The Semantic Ghost: Instead of throwing away the low-attention tokens, I run a Gravity-Weighted FMA (Fused Multiply-Add) reduction on them before Rotary Positional Embeddings (RoPE) are applied. This compresses the discarded tokens into a single "Summary Token" that anchors to the position of the most relevant evicted token.
4. Dense Packing: The block is written back to VRAM densely (e.g., 3 survivors + 1 ghost token), and the CPU is notified exactly how many slots were used, allowing the next sequence to pack directly into the remaining slots.

This essentially creates a closed SRAM ecosystem. Tokens have to mathematically prove their utility to survive into the expensive VRAM. If they fail, they are squashed into a semantic ghost token, ensuring the long-tail context is never truly forgotten, just compressed.