Why can I not reproduce the exact output my model gave in production?

Even at temperature zero, the same prompt sent to the same model twice can return different tokens. A January 2026 arXiv paper traced the root cause to dynamic batching: GPU reduction kernels change their internal arithmetic tree when batch size changes, and floating-point addition is not associative, so two requests processed under different batch sizes take different numerical paths and diverge. Serving infrastructure has no obligation to record or expose the batch context that produced a given output. This makes it impossible to replay a failed inference, write a regression test that pins exact behavior, or reconstruct what a model actually said during a production incident. The only solution demonstrated so far imposes a 61 percent throughput cost.