Welcome!

Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, or
method to help you stay sharp in the field. I hope you find it useful!

The ROC Curve Myth for Imbalanced Datasets

One myth I hear far too often is that the ROC curve is not a good metric for evaluating models trained on imbalanced datasets because it tends to produce overly optimistic ROC-AUC values.

This is not true. Last Monday we saw that ROC curves are insensitive to class prevalence: same model, different class prevalence = same ROC-AUC.

Today, we’ll debunk the myth that ROC-curves are overly optimistic.

ROC-Curves reflect the ability of the model to separate the classes correctly

While writing my new book Imbalanced Data: Myths, Mistakes and Modern Solutions I ran several experiments, many to debunk common myths about working with imbalance data, and many others to put undersampling and oversampling to the test.

Today, I’ll show you that this myth, that the ROC-curve is overly optimistic, is false.

I trained several ensemble models, including random forests, XGBoost, Catboost and lightGBM across 37 datasets with different degrees of class imbalance, and evaluated them using ROC-curves. Here’s what I saw.

For many datasets, I found ROC curves close to the top-left corner of the plot with AUC values close to 1. However, this behaviour was far from universal.

Take a look at the following figure. Some datasets (e.g., car_eval_4 and protein_homo) produce near-perfect ROC curves, whereas others (e.g., coil_2000 and scene) yield much lower AUC values despite being imbalanced:

The shape of the ROC curve therefore reflects how well the model separates the classes, not the prevalence of the minority class.

High ROC-AUC values, like those observed for the protein homology dataset, were not an artefact of class imbalance. Instead, they reflected that the model genuinely achieved strong class separation.

To investigate further, I plotted pairwise combinations of six features from the protein homology dataset, colouring observations according to their class label.

Take a look at the following image now. In most panels, the red and blue points occupy distinct regions of the feature space, with only limited overlap.

This indicates that the classes are already well separated by the original features, making the prediction task inherently easier and explaining why most models achieve ROC-AUC values close to one.

So bottom line: ROC curves are not overly optimistic by default. Instead, the truly reflect the capacity of the model to discriminate among classes.

Hence, if your models achieve a high ROC-AUC, that’s something to be happy about rather than worry over.

If you liked this article, consider supporting me by buying my new book: Imbalanced Data: Myths, Mistakes and Modern Solutions.

I hope this information was useful!

Wishing you a successful week ahead - see you next Monday! 👋🏻

Sole

Ready to enhance your skills?

Our specializations, courses and books are here to assist you:

• ​Clustering and Dimensionality Reduction (course)
• ​Forecasting with Machine Learning (course)
• ​Feature Selection for Machine Learning with Python (book)