Long Context Has One Villain, and It Is the Cache
It grows with every token, it never shrinks on its own, and it eats your GPU memory until you either buy a bigger card or start throwing tokens overboard. Throwing tokens overboard is the interesting option. The question that actually matters, the one everyone skips, is which ones.
The dumb answer, keep the newest and forget the rest, works great right up until the question depends on the one thing you already forgot, at which point the model just makes something up and says it with total confidence. The research answer is to keep the tokens that actually mattered, using signals like accumulated attention (H2O, Zhang et al. 2023) or a small set of protected attention-sink tokens (StreamingLLM, Xiao et al. 2023).
kvsqueeze puts those policies side by side on a task where forgetting is measurable, and shows exactly how far each one compresses before recall falls off a cliff.
A Needle You Can Actually Watch Get Evicted
The benchmark streams a long sequence through a fixed-budget cache. One early token is the needle a later query needs. A policy only passes if that token is still alive when the query arrives.
• Recency window: keeps the newest tokens, drops the oldest. Strong until the answer ages past the window.
• Heavy hitter: keeps whatever has actually been attended to the most, drops the rest. This is the H2O idea.
• Attention sink: always protects the first few tokens plus a recent window. This is the StreamingLLM fix for the collapse a pure recency window hits.
The Curve, Not Just One Number
A single budget point is a footnote. The real deliverable is the survival curve as you sweep the budget down from generous to brutal. At 80% of full cache every policy looks fine. By 24% the simple policies have flatlined at zero recall while heavy hitter is still holding on.
The Number
At 40% of full cache, heavy hitter holds 57% needle recall. Plain recency manages 15% and is basically checked out mentally by that point. Same memory budget, nearly a 4x gap in what actually survives. Push down to 10% of full cache and heavy hitter is the only policy still recalling anything at all, the last one still awake at the meeting.
7 tests pass in CI across Python 3.9, 3.11, and 3.13, all deterministic.