Welcome!

Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, or
method to help you stay sharp in the field. I hope you find it useful!

Have you added it to your reading list yet?

Imbalanced Data, Myths, Mistakes and Modern Solutions

Imbalanced data isn't a problem. How you handle it is.

My book, Imbalanced Data: Myths, Mistakes and Modern Solutions, cuts through the common misconceptions and brings together practical, evidence-based approaches you can actually apply.

Less guesswork. Better decisions. Stronger models.

Class imbalance makes metrics volatile

When discussing class imbalance, we usually ask whether it makes machine learning models perform worse.

But class imbalance creates another problem that receives much less attention:

It makes model performance more difficult to measure.

Imagine that we have 20,000 observations. That sounds like a reasonably large dataset.

With an imbalance ratio of 100:1, however, only 199 observations belong to the minority class. At 1000:1, that number goes down to 19.

Those 19 observations carry most of the information we have about the model’s ability to identify and rank the minority class. Therefore, although our dataset contains thousands of rows, most of them belong to the majority class, so we are left with little data to judge the model’s performance on the class we care about the most.

And small samples produce volatile metrics.

Why ROC-AUC becomes more volatile

ROC-AUC measures the probability that a randomly selected positive observation receives a higher score than a randomly selected negative observation.

To estimate that probability, we need examples from both classes.

When the positive class contains only a handful of observations, each one has a substantial influence on the result. If one difficult positive example falls into a validation fold, the ROC-AUC may decrease considerably. If that observation falls into the training data instead, the validation ROC-AUC may be much higher.

Same model. Same underlying problem. Different split, different result.

This means that wider error bars do not necessarily indicate that the model has become unstable or incapable of learning from imbalanced data. They can indicate that we do not have enough minority observations to measure its performance precisely.

Show me the data

I ran a simulation using logistic regression and XGBoost.

I created two datasets with 20,000 observations, the same feature structure and the same class separation. The only difference was the imbalance ratio: 100:1 versus 1000:1.

I evaluated both models using repeated stratified cross-validation and ROC-AUC. I also compared three-fold with five-fold cross-validation.

The following figure shows the mean ROC-AUC for both models. The error bars represent plus or minus one standard deviation across the validation folds.

In both panels, the error bars become much wider when the imbalance ratio increases from 100:1 to 1000:1.

The following table shows the results after cross-validation 5 folds:

Small test samples lead to large error bars

This problem is not unique to class imbalance.

In Cross-validation failure: small sample sizes lead to large error bars, Gaël Varoquaux investigated the reliability of performance estimates in predictive modelling.

Using simulations and neuroimaging datasets, he showed that small evaluation samples produce large differences between cross-validation performance and performance on new data. In his experiments, a sample size of 100 could lead to errors of approximately plus or minus 10 percentage points.

Most importantly, he showed that this variability is not necessarily caused by the machine learning model. Similar behaviour appeared even with a predictor defined independently of the training data. The volatility could therefore be traced back to sampling noise in the evaluation set.

That is closely related to what we see with class imbalance.

We may have 20,000 observations in total, but at an imbalance ratio of 1000:1, we have only 19 minority examples. For the part of the metric that depends on those observations, our sample is tiny.

No evaluation procedure can fully recover information that is not present in the data.

You can read the complete author-posted version of Varoquaux’s article here.

Give the metric more observations

The most direct way to reduce this volatility is to increase the number of minority observations used for evaluation.

One option is to collect more data. This is the best solution because it increases the information available for both training and evaluation.

Of course, it is not always possible.

A second option is to leave a larger proportion of the available data in the test set. A larger test set contains more minority observations and should provide a more precise final performance estimate.

The trade-off is that fewer observations remain for training.

A similar decision appears in cross-validation. Decreasing the number of folds increases the size of each validation fold.

In the experiment I shared earlier, three-fold cross-validation placed approximately six minority observations in each validation fold at 1000:1. Five-fold cross-validation placed only three or four.

The three-fold design produced lower ROC-AUC dispersion for both models. At 1000:1, the standard deviation decreased from 0.176 to 0.142 for logistic regression and from 0.138 to 0.111 for XGBoost.

But once again, there is a trade-off. With three folds, each model is trained on two-thirds of the data. With five folds, it is trained on four-fifths. Choices…

Larger validation folds give us more information for evaluation, but less information for fitting each model.

​
​

Conclusions

When evaluating a model on imbalanced data, do not look only at the total sample size. Look at how many minority observations actually reach each validation or test set.

And do not report the mean metric alone. Examine its dispersion. If the error bars are wide, a small difference between models may say more about the observations that landed in each split than about the models themselves.

Where possible, collect more minority observations. Otherwise, consider allocating more data to evaluation, either through a larger test set or through fewer cross-validation folds.

Neither choice is free. We need to balance the amount of information used to train the model against the amount used to evaluate it.

The code for this issue is in this Github repository. Please give it a star if you found it useful.

If you liked this article, you’ll also enjoy my new book: Imbalanced Data: Myths, Mistakes and Modern Solutions.

I hope this information was useful!

Wishing you a successful week ahead - see you next Monday! 👋🏻

Sole

Ready to enhance your skills?

Our specializations, courses and books are here to assist you:

• ​Clustering and Dimensionality Reduction (course)
• ​Forecasting with Machine Learning (course)
• ​Feature Selection for Machine Learning with Python (book)

​​​

Copyright (C) 2026 Train in Data. All rights reserved.

​
​Unsubscribe · Preferences​