Because model confidence is highest precisely where calibration breaks down. A few concrete reasons: **Distribution shift** Models are calibrated on training data. When the input is genuinely novel or out-of-distribution, the model has no real basis for its confidence, yet the softmax output still sums to 1.0 and some class gets a high probability. The number looks authoritative because the math forces it to. **Confidence reflects familiarity, not correctness** A model assigns high confidence to inputs that *look like* things it has seen before, regardless of whether those inputs were correctly labeled during training. Confident wrong answers are common near decision boundaries that the training set happened to reinforce. **The cases that matter most are the edge cases** High-stakes decisions tend to involve rare events, adversarial inputs, or novel situations, which are exactly the regions where training coverage is sparse. The model is least calibrated there, yet the confidence score gives no signal that you're in that region. **Overconfidence is a known systematic bias** Large neural networks are empirically overconfident. Post-hoc calibration (temperature scaling, Platt scaling) helps on held-out sets but doesn't eliminate the problem, and the calibration guarantee degrades further from the calibration distribution. **Epistemic vs. aleatoric uncertainty are conflated** The score doesn't distinguish between "I am uncertain because the data is inherently ambiguous" and "I am uncertain because I have never seen anything like this." The second case is the dangerous one, and a scalar confidence score is blind to it. The short version: a confidence score is a relative ranking within the training distribution. When you most need it to warn you, you have already left that distribution.