Myth: ROC-AUC is sensitive to class imbalance
Far too often, we hear that the ROC curve and ROC-AUC are unsuitable for evaluating models trained on imbalanced data. Apparently, imbalance has a personal vendetta against ROC-AUC.
The argument usually comes in two flavours:
- ROC-AUC is sensitive to class imbalance.
- ROC-AUC is overly optimistic on imbalanced datasets, so it cannot discriminate between models.
Both claims are myths. In this issue, I’ll tackle the first one. In the next, we’ll look at the “too optimistic” argument.
ROC-AUC is insensitive to class prevalence
Class prevalence (another way of saying class distribution) is the proportion of positive and negative examples in the data. When we have 2 classes and one is severely outnumbered, we usually speak of the prevalence of class 1, i.e., the minority class. So if we have a balanced dataset, the class prevalence is 0.5, if we have an imbalanced dataset, it could be anything smaller than that, for example, 0.1 when we have 10 observations of the majority per minority class example.
ROC-AUC measures how well a model ranks the two classes. One useful interpretation is:
ROC-AUC is the probability that a randomly selected positive observation receives a higher score than a randomly selected negative observation.
The ROC curve plots the true positive rate against the false positive rate:
- True positive rate: TP / (TP + FN)
- False positive rate: FP / (FP + TN)
Think of ROC as giving the model two separate tests:
- Among the actual positives, how many did it identify?
- Among the actual negatives, how many did it mistakenly flag?
Because each class is judged separately, having many more negatives than positives does not automatically change either result. The imbalance changes how many people take each of the former tests, not the model’s success rate within each test.
Show me the numbers
Let’s look at this with an example. I trained a logistic regression and an XGBoost model on a balanced dataset with decent class separation. Then, I evaluated the models using the ROC-AUC on different test sets with different class prevalences.
The model is the same, so it has the same capacity of ranking observations.
As we see in the following image, increasing the imbalance ratio (i.e., decreasing the class prevalence) does not affect the ROC-AUC. If we take an observation at random, the model is equally likely to assign it a higher probability of being of class 1 if it is indeed of class 1, regardless of the class imbalance.
In short, the ROC-AUC is insensitive to class prevalence. Same model, different class distributions, same ROC-AUC.
Average precision is sensitive to class prevalence
The mainstream claim is that because ROC curves are unsuitable for imbalance datasets (we’ve seen they are not), we should go for PR curves instead. Now, interestingly, PR curves and hence the PR-AUC (also called average precision) are sensitive to class prevalence. Here’s why:
Precision is: TP / (TP + FP)
Unlike the rates used by the ROC curve, precision mixes the two classes in one denominator. As the positive class becomes rarer, false positives make up a larger fraction of predicted positives, even when the model’s true positive and false positive rates have not changed. Precision therefore falls, and so does the precision–recall curve and its summary; average precision or PR-AUC.
Take a look at the following image, where I evaluate the same model, trained on a balanced dataset, on different test sets with decreasing class prevalence (i.e., increasing class imbalance ratios):
ROC-AUC and PR-AUC baselines
You probably know that the ROC-AUC baseline is 0.5, that is, the performance of a random guessing model will result in a ROC-AUC of 0.5.
Less known is that the baseline of the PR-AUC is the class imbalance. So for a balanced dataset, the average precision is 0.5, for an imbalance dataset the class precision is less than that: With one positive for every 10 negatives, the baseline is about 0.091. At 5,000:1, it is about 0.0002.
This moves our reference point. While a good model should have a ROC-AUC greater than 0.5, regardless of class imbalance. A good model should have a PR-curve greater than the baseline for imbalance datasets. In short, we can’t assess a model trained on one class distribution, on a different class distribution with PR curves, because we’ll be comparing pears with bananas.
Does this matter in practice?
Usually, less than this debate suggests.
In the overwhelming majority of projects, the imbalance ratio does not change, or at least that’s our assumption. We train on a given class distribution. We evaluate those models on the same distribution. And we assume that after deployment, we’ll continue to see the same distribution.
That means that we normally compare candidate models on a validation or test set with the same class prevalence as the train set. That dataset has one class distribution, so every model faces the same prevalence. If prevalence makes a metric look numerically better or worse, it does so for all the models being compared.
In short, no worries, we can evaluate the models with either metric.
This distinction matters when the class distribution changes after a model is deployed. Imagine a model that predicts an infectious disease. If that disease becomes more common five years from now, its prevalence will increase. To compare the model’s performance then with its performance today, we need a metric that is not mechanically affected by that change in prevalence.
Conclusion
This is not an argument that ROC-AUC is the only metric worth using. Metric choice should reflect the decision we need to make. It is simply a demonstration of a narrower point: ROC-AUC is not mechanically sensitive to class prevalence.
So from now on, you can use it with confidence, regardless of your data imbalance.
If you liked this article, consider supporting me by buying my new book: Imbalanced Data: Myths, Mistakes and Modern Solutions.
I hope this information was useful!
Wishing you a successful week ahead - see you next Monday! 👋🏻
Sole
While you are at it, check out our courses:
Stop aimless internet browsing. Start learning today with meticulously crafted courses offering a robust curriculum, fostering skill development with steadfast focus and efficiency.
