Models are not sensitive to class imbalance
Still today, many blogs and articles online, as well as generative AI models, will tell you that machine learning models are not able to cope with imbalanced datasets and will therefore show poor performance. Hence, you need to do something about the imbalance if you want to accurately predict the minority class, which is usually the one you care about.
That might have been true 26 years ago, when we did not have ensemble methods and relied more heavily on individual decision trees and KNNs. But after the introduction of random forests and gradient-boosting machines, that is no longer the world we live in.
In fact, if the classes are linearly separable, their relative proportions will not prevent even a linear model such as logistic regression from separating them properly. For more complex patterns, where the classes are not linearly separable, ensemble models are perfectly capable of discriminating between them, even when the data is highly imbalanced.
Show me the data
To test this, I ran a simulation with imbalance ratios ranging from 10:1 to 5000:1, with a decent class separation.
For each ratio, I trained several commonly used classifiers, including logistic regression, random forests, XGBoost, LightGBM, CatBoost, and two gradient-boosting implementations from scikit-learn.
I trained these models with hyperparameters tuning, which is what we would do in practice anyway.
I then measured their ability to discriminate between the two classes using ROC-AUC with cross-validation, to obtain a measure of performance dispersion.
As we can see in the following image, the performance of the models did not decrease as we increased the imbalance ratio. If the classes can be separated by the features we have in the data, the models will perfectly do so, regardless of the class imbalance.
In the following images we see the performance of the most commonly used ensemble methods (random forest, XGBoost, Catboost, LightGBM) alongside 2 GBM implementations from scikit-learn and logistic regression. The imbalance ratio increases towards the right (x-axis). Performance is measured with the ROC-AUC. As you can see, apart from sklearn's GBM, all models achieve similar performance regardless of the class imbalance degree.
The widening error bands (shown as the shaded area surrounding the curves) reflect the increasing difficulty of assessing performance when the positive class is very rare. With fewer positive examples available for evaluation, the ROC-AUC estimate naturally becomes less precise. This is a limitation of the evaluation sample, not evidence that the models can no longer discriminate between the classes.
From my simulations, only Scikit-learn’s traditional gradient-boosting model seemed to be susceptible to imbalance, and only when the imbalance became really extreme.
Leo Breiman explains that in random forests, each tree is trained on a bootstrap sample of the original data. When the imbalance is extreme, some of those samples may contain very few, or even no minority-class observations. As a result, the corresponding trees have little opportunity to learn the minority class and may perform poorly when predicting it. Hence, we predicted, imbalance ratio may affect random forest performance. But in fact, he went on to test that, and he saw that when using the ROC as evaluation metric, he didn’t see performance degradation, and neither did we, when the hyperparameters are tuned properly.
Check out Leo’s discussion here:
Chen, C., Liaw, A., & Breiman, L. (2004). Using random forest to learn imbalanced data (Technical Report No. 666). Department of Statistics, University of California, Berkeley. https://statistics.berkeley.edu/sites/default/files/tech-reports/666.pdf
Conclusion
In short, when you have imbalanced datasets, reach out for any of these powerful classifiers, tune their hyperparameters, and evaluate them appropriately, using ROC curves and ROC-AUC or scoring functions like log loss and Brier score. With this framework, if your features are able to separate the data, so will your models.
You can find the code for the experiment described here at this link. If you like the repo, please give it a star.
If you liked this article, consider buying my new book: Imbalanced Data: Myths, Mistakes and Modern Solutions.
I hope this information was useful!
Wishing you a successful week ahead - see you next Monday! 👋🏻
Sole
While you are at it, check out our courses:
Stop aimless internet browsing. Start learning today with meticulously crafted courses offering a robust curriculum, fostering skill development with steadfast focus and efficiency.
