TunedThresholdClassifierCV: the good, the bad, the ugly
For years, scikit-learn classifiers made one decision for us: predict() converted probabilities into class labels using a de facto cut-off of 0.5. That number seems obvious from stats classes. But in practice, it’s rarely the right one.
A threshold is not a property of the data or the model. It is a decision. And the best decision depends on what we are trying to achieve.
Why adjust the threshold?
Suppose a model estimates that a patient has a 20% probability of disease. Should we classify the patient as positive?
If the next step is an inexpensive screening test, perhaps yes. If the next step is risky surgery, almost certainly not. Based on the same probability, we make different decisions based on the cost. In other words, the cost determines the threshold. Not the model.
The same applies when we optimise a classification metric. Precision, recall, F1-score, balanced accuracy and the entries of the confusion matrix all depend on the cut-off used to convert a continuous model output into a class label. Lowering the threshold usually identifies more positives, but it also creates more false positives. Raising it does the opposite.
There is no reason to expect 0.5 to produce the trade-off we want. In my experiments for Imbalanced Data: Myths, Mistakes and Modern Solutions, the thresholds that maximised balanced accuracy were sometimes close to 0.005. If I had used the de facto threshold of 0.5, I would have underestimated the models and, in some cases, selected the wrong one.
This also explains part of the apparent success of resampling and class weighting. Both can shift the decision boundary towards 0.5. But we can often reach the same operating point by training on the original distribution and adjusting the threshold.
For a long time, predict() meant 0.5
When using scikit-learn, we could always call predict_proba(), apply our own threshold and calculate the metric manually. I know I did this. But this separated the decision rule from the estimator and made the whole process inconvenient.
Scikit-learn changed that with TunedThresholdClassifierCV. The class wraps a binary classifier, evaluates candidate thresholds by cross-validation and selects the one that maximises the metric we provide. After fitting, predict() uses the tuned threshold instead of the de facto 0.5
TunedThresholdClassifierCV in action
Let’s take a look at how TunedThresholdClassifierCV works. We assume we have some data split over a training and a testing set. We train a gradient boosting classifier, and then we want to evaluate the balanced accuracy at the optimal cut-off point. We’d do the following:
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import TunedThresholdClassifierCV
model = TunedThresholdClassifierCV(
GradientBoostingClassifier(random_state=0),
scoring="balanced_accuracy",
cv=5,
)
model.fit(X_train, y_train)
print(model.best_threshold_)
y_pred = model.predict(X_test)
model.best_threshold_ will show you the value of the optimal cut-off, and model.predict() assigns observations to classes based on that value.
|
The model and its decision rule now are shipped together, and we can pass it to hyperparameter tuning or cross-validation functions as we would do for any model.
The good
TunedThresholdClassifierCV solves a real and surprisingly common problem. It makes threshold optimisation part of a scikit-learn workflow. It uses cross-validation by default, accepts scikit-learn scorers and custom business metrics, and exposes the selected value through best_threshold_.
Most importantly, it reminds us that model training and decision-making are different tasks. The classifier estimates a score or probability. The threshold turns that estimate into a class.
The bad
The fitted object returns one best threshold. What it does not give us directly is the dispersion of that threshold across different samples.
That matters because the optimum is an estimate. If we change the observations in a validation fold, the best threshold can change.
And the ugly
By default, TunedThresholdClassifierCV evaluates 100 equi-distant thresholds. That sounds generous until we work with highly imbalanced data.
With imbalance datasets, useful thresholds can accumulate in a tiny region near zero. A grid spread across the full range may then place too few candidates where the optimum actually lies, and we may miss it.
The good thing however, is that we can manually pass hand-picked intervals of thresholds to TunedThresholdClassifierCV to explore the areas we think are more valuable.
Before you go
There is one more thing to consider. TunedThresholdClassifierCV will optimise the metric we ask it to optimise. For a different metric, we’ll need a different threshold, guaranteed.
My recommendation
Use TunedThresholdClassifierCV. It is a welcome addition, and for most projects it is a much better starting point than predict() at 0.5.
If you liked this article, you’ll also enjoy my new book: Imbalanced Data: Myths, Mistakes and Modern Solutions.
I hope this information was useful!
Wishing you a successful week ahead - see you next Monday! 👋🏻
Sole
Ready to enhance your skills?
Our specializations, courses and books are here to assist you:
