Why can a neighbor on my inference cluster reconstruct what I just asked the model?
Opportunity
Modern LLM serving frameworks like vLLM share KV-cache blocks across requests with matching prefixes to reduce compute cost. When multiple tenants share the same inference server, an adversary tenant can probe cache-hit and cache-miss latencies to reconstruct another tenant's private prompt. Three independent attacks published in 2025 and 2026, PROMPTPEEK, EarlyBird, and InputSnatch, demonstrate up to 100% prompt reconstruction accuracy against production-grade serving stacks. Providers can defeat this by running fully isolated instances per tenant, but that eliminates all the memory and compute savings that make shared inference economically viable. The KVGov governance paper from August 2026 proposes per-principal cryptographic cache salts as a fix, but no commercial inference provider has deployed it.
Why it matters
Shared inference is the cost model that makes AI accessible at scale, but right now it cannot safely carry any sensitive workload without full tenant isolation.
How I score the opportunity
The Opportunity Score is my own read, not a measurement: how much it hurts, how often it bites, and how little exists to solve it today. Higher means I think it is more worth building.
How much pain it causes when it shows up.
How often people actually run into it.
How little good tooling exists for it today.
More problems worth solving
Why does every AI app forget me the moment I close the tab?
AIWhy is learning a new field still gated by knowing what to ask?
AIWhy can a non-expert not verify what an AI just told them?
AIWhy do we test models on benchmarks but ship them on vibes?
AIWhy do AI agents have no memory of their own mistakes?
AIWhy can't I audit what a model was actually trained on?