A classifier is calibrated when its predicted probability matches how often it is right. The paper shows that modern deep networks are poorly calibrated, unlike networks from a decade earlier, and studies what causes it.
Findings
- Depth, width, weight decay and batch normalisation all affect calibration.
- Among post-hoc fixes compared on image and document classification, temperature scaling (Platt scaling with a single parameter) works surprisingly well on most datasets.
Limits
- Abstract-level reading only.
- Calibration is fitted on held-out data from the same distribution; the abstract does not address shift.
Relevance
A detector’s or VLM’s confidence cannot be read as a probability without calibration. When a robot fuses scores from several models or teammates, uncalibrated scores mean the most overconfident source wins, an issue for robust-estimation and robust robotics. Cited in GoGet.
Links
- arXiv
- Subfield: Robot learning, Safety and verification