Class Weights Do Not Improve Model Performance
Hereβs a pattern I see all the time.
A practitioner trains a model on an imbalanced dataset and obtains a ROC-AUC of 0.670. They then set class_weight="balanced", train the model again, and obtain a ROC-AUC of 0.675.
The second number is higher, so the conclusion seems obvious: class weighting improved the model.
Except we have ignored an important question:
" How precise are those ROC-AUC estimates?"
|
A performance metric calculated on one test set is only an estimate. If the test set contained a slightly different collection of observations, we would obtain a slightly different value.
Therefore, the fact that one ROC-AUC is numerically higher than another does not necessarily mean that one model is genuinely better.
A difference is not necessarily an improvement
Imagine that we obtain the following results:
- No class weights: ROC-AUC = 0.670
- class_weight="balanced": ROC-AUC = 0.675
The observed improvement is 0.005.
If we report only those two numbers, class weighting wins.
But what if the uncertainty around that difference stretches is around 0.04?
Then, those values look more like the following:
- No class weights: ROC-AUC = 0.666 - 0.674
- class_weight="balanced": ROC-AUC = 0.671 - 0.679
Now we can see that the performance intervals overlap substantially. Therefore, the difference in performance is, most likely, not statistically significant. In lay terms: both models perform the same.
Show me the data
Letβs look at this with an example.
The following example creates an imbalanced dataset and trains two logistic regression models. Everything is identical except that one model uses class_weight="balanced".
First, we evaluate both models once on the test set:
import numpy as np
import matplotlib.pyplot as plt
β
β
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
β
β
# Create an imbalanced dataset
X, y = make_classification(
n_samples=2200,
n_features=12,
n_informative=5,
n_redundant=3,
weights=[0.93, 0.07],
class_sep=0.75,
flip_y=0.03,
random_state=3,
)
β
β
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.25,
stratify=y,
random_state=11,
)
β
β
models = {
"No weights": LogisticRegression(max_iter=2000),
"Balanced": LogisticRegression(
max_iter=2000,
class_weight="balanced",
),
}
β
β
probabilities = {}
β
β
for name, model in models.items():
model.fit(X_train, y_train)
probabilities[name] = model.predict_proba(X_test)[:, 1]
β
β
single_scores = {
name: roc_auc_score(y_test, probs)
for name, probs in probabilities.items()
}
β
β
print(single_scores)
|
The results are approximately:
No weights: 0.670
Balanced: 0.675
If we stopped here, we might conclude that class weighting worked.
Letβs not stop here.
|
Estimating the uncertainty
We can bootstrap the test set to see how much the ROC-AUC changes when the evaluation sample changes.
Importantly, we sample the same observations for both models during each bootstrap iteration. This gives us a paired comparison: for every sample, we calculate not only the two ROC-AUC values, but also their difference.
rng = np.random.default_rng(0)
n_bootstraps = 2000
β
β
bootstrap_scores = {
"No weights": [],
"Balanced": [],
}
bootstrap_differences = []
β
β
for _ in range(n_bootstraps):
idx = rng.integers(0, len(y_test), len(y_test))
β
β
# ROC-AUC requires both classes
if len(np.unique(y_test[idx])) < 2:
continue
β
β
auc_no_weights = roc_auc_score(
y_test[idx],
probabilities["No weights"][idx],
)
auc_balanced = roc_auc_score(
y_test[idx],
probabilities["Balanced"][idx],
)
β
β
bootstrap_scores["No weights"].append(auc_no_weights)
bootstrap_scores["Balanced"].append(auc_balanced)
bootstrap_differences.append(
auc_balanced - auc_no_weights
)
|
We can now plot the ROC-AUC estimates and their 95% bootstrap intervals. The second panel shows the paired difference directly.
names = list(models)
means = [
np.mean(bootstrap_scores[name])
for name in names
]
β
β
intervals = [
np.percentile(bootstrap_scores[name], [2.5, 97.5])
for name in names
]
β
β
errors = np.array([
[mean - low, high - mean]
for mean, (low, high) in zip(means, intervals)
]).T
plt.figure(figsize=(5, 4))
plt.errorbar(
names,
means,
yerr=errors,
fmt="o",
capsize=6,
color="#c44e7b",
)
β
β
plt.ylabel("ROC-AUC")
plt.title("Performance with 95% bootstrap intervals")
plt.grid(axis="y", alpha=0.25)
plt.tight_layout()
plt.show()
|
The weighted model still has a slightly higher point estimate. But the confidence intervals overlap:
So the correct conclusion is not that class weighting improved performance.
The correct conclusion is that this experiment does not provide convincing evidence that either model performs better.
This is not an isolated result
For my book, Imbalanced Data: Myths, Mistakes and Modern Solutions, I investigated this question across 37 benchmark datasets.
I trained random forests and modern implementations of gradient boosting machines on the original class distribution. I then trained otherwise identical models using class weights derived from the imbalance ratio.
Across the datasets, the ROC-AUC of the weighted models remained essentially unchanged. The observed differences fell within the estimated uncertainty: class weighting did not make the models inherently better at separating positive from negative observations.
But class weighting did change something.
It distorted the estimated probabilities.
Class weights alter the effective class distribution seen during training. Consequently, the resulting probabilities are calibrated to the weighted distribution rather than the original one. In my experiments, this was reflected in higher Brier scores for the weighted models.
The same classification performance could be achieved by training on the original distribution and adjusting the decision threshold. Class weighting mostly shifted the decision boundary towards 0.5; it did not add new discriminatory information.
Conclusion
Never conclude that a modelling intervention worked because one metric increased on one evaluation sample.
Estimate the uncertainty around the comparison using cross-validation or bootstrapping. Ideally, examine the paired differences, not just two isolated error bars.
If the intervals overlap, the evidence does not establish an improvement.
And remember that class_weight="balanced" is not harmless. It may leave ranking performance unchanged while making the predicted probabilities harder to interpret.
If you liked this article, youβll also enjoy my new book: Imbalanced Data: Myths, Mistakes and Modern Solutions.
I hope this information was useful!
Wishing you a successful week ahead - see you next Monday! ππ»
Sole
Ready to enhance your skills?
Our specializations, courses and books are here to assist you:
β