Welcome!

Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, or
method to help you stay sharp in the field. I hope you find it useful!

My New Book is Out!

Imbalanced Data, Myths, Mistakes and Modern Solutions

Most advice about imbalanced data is incomplete, or simply wrong. This book challenges common assumptions and provides practical, evidence-based guidance for tackling class imbalance and building better machine learning models.

Class Weights Do Not Improve Model Performance

Here’s a pattern I see all the time.

A practitioner trains a model on an imbalanced dataset and obtains a ROC-AUC of 0.670. They then set class_weight="balanced", train the model again, and obtain a ROC-AUC of 0.675.

The second number is higher, so the conclusion seems obvious: class weighting improved the model.

Except we have ignored an important question:

A performance metric calculated on one test set is only an estimate. If the test set contained a slightly different collection of observations, we would obtain a slightly different value.

Therefore, the fact that one ROC-AUC is numerically higher than another does not necessarily mean that one model is genuinely better.

A difference is not necessarily an improvement

Imagine that we obtain the following results:

• No class weights: ROC-AUC = 0.670
• class_weight="balanced": ROC-AUC = 0.675

The observed improvement is 0.005.

If we report only those two numbers, class weighting wins.

But what if the uncertainty around that difference stretches is around 0.04?

Then, those values look more like the following:

• No class weights: ROC-AUC = 0.666 - 0.674
• class_weight="balanced": ROC-AUC = 0.671 - 0.679

Now we can see that the performance intervals overlap substantially. Therefore, the difference in performance is, most likely, not statistically significant. In lay terms: both models perform the same.

Show me the data

Let’s look at this with an example.

The following example creates an imbalanced dataset and trains two logistic regression models. Everything is identical except that one model uses class_weight="balanced".

First, we evaluate both models once on the test set:

Estimating the uncertainty

We can bootstrap the test set to see how much the ROC-AUC changes when the evaluation sample changes.

Importantly, we sample the same observations for both models during each bootstrap iteration. This gives us a paired comparison: for every sample, we calculate not only the two ROC-AUC values, but also their difference.

We can now plot the ROC-AUC estimates and their 95% bootstrap intervals. The second panel shows the paired difference directly.

The weighted model still has a slightly higher point estimate. But the confidence intervals overlap:

So the correct conclusion is not that class weighting improved performance.

The correct conclusion is that this experiment does not provide convincing evidence that either model performs better.

This is not an isolated result

For my book, Imbalanced Data: Myths, Mistakes and Modern Solutions, I investigated this question across 37 benchmark datasets.

I trained random forests and modern implementations of gradient boosting machines on the original class distribution. I then trained otherwise identical models using class weights derived from the imbalance ratio.

Across the datasets, the ROC-AUC of the weighted models remained essentially unchanged. The observed differences fell within the estimated uncertainty: class weighting did not make the models inherently better at separating positive from negative observations.

But class weighting did change something.

It distorted the estimated probabilities.

Class weights alter the effective class distribution seen during training. Consequently, the resulting probabilities are calibrated to the weighted distribution rather than the original one. In my experiments, this was reflected in higher Brier scores for the weighted models.

The same classification performance could be achieved by training on the original distribution and adjusting the decision threshold. Class weighting mostly shifted the decision boundary towards 0.5; it did not add new discriminatory information.

Conclusion

Never conclude that a modelling intervention worked because one metric increased on one evaluation sample.

Estimate the uncertainty around the comparison using cross-validation or bootstrapping. Ideally, examine the paired differences, not just two isolated error bars.

If the intervals overlap, the evidence does not establish an improvement.

And remember that class_weight="balanced" is not harmless. It may leave ranking performance unchanged while making the predicted probabilities harder to interpret.

If you liked this article, you’ll also enjoy my new book: Imbalanced Data: Myths, Mistakes and Modern Solutions.

I hope this information was useful!

Wishing you a successful week ahead - see you next Monday! 👋🏻

Sole

Ready to enhance your skills?

Our specializations, courses and books are here to assist you:

• ​Clustering and Dimensionality Reduction (course)
• ​Forecasting with Machine Learning (course)
• ​Feature Selection for Machine Learning with Python (book)

You are receiving this email because you subscribed to our newsletter, signed up on our website, purchased or downloaded any products from us.

Follow us on social media

​​​

Copyright (C) 2026 Train in Data. All rights reserved.

​
​Unsubscribe · Preferences​