Why optimise for calibration if I can recalibrate later?
If a model produces poorly calibrated probabilities, we can fix them afterwards. Fit the model, pass its outputs through Platt scaling or isotonic regression, and carry on.
So why should we care about calibration while choosing and training the original model?
Well here’s the thing: Recalibration is another modelling problem, with another model, another dataset, and another evaluation procedure.
Sometimes that extra work is justified. But it should not be the default solution to a problem we can easily avoid.
Recalibration involves fitting a new model
Suppose our classifier returns a score of 0.8 for a group of customers, but only 50% of them actually default. A recalibration model learns a mapping from 0.8 to something closer to 0.5.
Platt scaling does this by fitting a sigmoid. Isotonic regression learns a more flexible monotonic mapping. In both cases, we fit parameters from data. In other words, we train another model on top of the first one. Hence, we need to evaluate it properly.
The calibrator becomes part of the system. We need to version it, deploy it, monitor it and retrain it when necessary, alongside the main model. Too much work. And too much room for things going wrong.
The calibrator needs its own data
We cannot fit the calibrator on the same predictions the base model used for training. We need a dedicated calibration set.
In the cleanest setup, we need three datasets:
- a training set to fit the base model,
- a calibration set to fit the recalibration model, and
- an untouched test set to evaluate the complete pipeline.
When data is limited, cross-validation can generate out-of-fold predictions for the calibrator. But it turns out that to evaluate calibration properly, we do need a lot of data. So we are making our lives harder if we now need to split however much data we have into additional sets.
All in all, “I’ll recalibrate later” commits us to training, selecting and evaluating two models instead of one.
The good news is: we don’t have to.
A simpler starting point
If probabilities matter, make them part of model selection from the beginning. Train on the natural class distribution when possible. Choose models that are capable of producing sensible probabilities. Logistic regression and modern gradient boosting methods such as XGBoost, LightGBM and CatBoost optimise log loss, which encourages useful probability estimates.
During hyperparameter tuning and feature selection, evaluate candidates with proper scoring rules such as log loss or Brier score.
Then inspect calibration curves on a sufficiently large, independent test set. The curve is part of deciding which model should reach production.
None of this guarantees perfect calibration. No algorithm does. But it makes poorly calibrated probabilities less likely. And it saves you a ton of work afterwards.
If you liked this article, you’ll also enjoy my new book: Imbalanced Data: Myths, Mistakes and Modern Solutions.
I hope this information was useful!
Wishing you a successful week ahead - see you next Monday! 👋🏻
Sole
Ready to enhance your skills?
Our specializations, courses and books are here to assist you:
​