Undersampling does not improve model performance
When faced with an imbalanced dataset, one of the most common recommendations is to balance the classes before training the model.
The simplest way to do this is undersampling: remove observations from the majority class so that its size becomes closer to that of the minority class.
It sounds reasonable. If the model sees both classes in similar proportions, surely it will pay more attention to the rare one.
But there is a rather uncomfortable detail: We are throwing away data we went through the trouble of collecting. And that’s rarely a good idea.
What undersampling methods actually do
Random undersampling removes majority-class observations at random until a chosen class ratio is reached, often 1:1.
Cleaning methods are more selective. They use rules based on nearest neighbours, class overlap, or instance hardness to decide which majority-class observations look redundant, noisy, or difficult, and remove those instead.
For example, Edited Nearest Neighbours removes majority observations whose nearest neighbours disagree with their class. Tomek Links removes majority observations that form very close opposite-class pairs. Condensed Nearest Neighbours tries to retain only the observations needed by a one-nearest-neighbour classifier.
Why did undersampling seem like a good idea?
Let’s take a look at the history.
Many of these methods were introduced long before the ensemble models we routinely use today. Condensed Nearest Neighbours appeared in 1968. Edited Nearest Neighbours followed in 1972. Tomek Links, Repeated ENN, and AllKNN arrived in 1976. One-Sided Selection was proposed in 1997, Neighbourhood Cleaning Rule in 2001, and NearMiss in 2003.
At the time, researchers commonly worked with nearest-neighbour classifiers, linear machines, and individual decision trees. Computing resources were limited, datasets were smaller, and simplifying the training set could make both computational and statistical sense.
Then random forests and gradient-boosting machines changed the modelling landscape. Yet the old recommendation survived: balance the data first
So I put it to the test.
Show me the data
For my book, Imbalanced Data: Myths, Mistakes and Modern Solutions, I evaluated undersampling across 37 datasets with different sizes and imbalance ratios.
I combined the undersampling methods with widely used ensemble models: random forests, scikit-learn’s gradient boosting machine, CatBoost, XGBoost, and LightGBM. I evaluated discrimination with ROC-AUC and the quality of the predicted probabilities with the Brier score. I also accounted for variability instead of treating every tiny numerical difference as an improvement.
The following figure zooms in on random undersampling for five representative datasets. The datasets contain between 50,000 and 280,000 observations, with imbalance ratios ranging from 3:1 to almost 600:1.
As we can see in the image, random undersampling does not make the models better at discriminating between the classes, and it deteriorates the calibration of their outputs. The result is remarkably consistent.
Look at the left-hand panels. The green ROC-AUC values are usually similar to, and sometimes lower than, their blue counterparts. Undersampling did not give the models new information with which to rank positive observations above negative ones.
Now look at the right-hand panels. The Brier scores increase sharply after undersampling. In other words, the predicted probabilities become less accurate.
If undersampling does not work, what should we do instead?
Train a strong classifier on the original class distribution. Evaluate it with a metric that matches the question you care about and report the uncertainty around that estimate. If you need a different trade-off between false positives and false negatives, adjust the decision threshold after training.
You can find the complete experiments in this GitHub repository. Please give it a star if you find it useful.
If you liked this article, you’ll also enjoy my new book: Imbalanced Data: Myths, Mistakes and Modern Solutions.
I hope this information was useful!
Wishing you a successful week ahead - see you next Monday! 👋🏻
Sole
Ready to enhance your skills?
Our specializations, courses and books are here to assist you:
