The ROC Curve Myth for Imbalanced Datasets


Welcome!

Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, or
method to help you stay sharp in the field. I hope you find it useful!


The ROC Curve Myth for Imbalanced Datasets

One myth I hear far too often is that the ROC curve is not a good metric for evaluating models trained on imbalanced datasets because it tends to produce overly optimistic ROC-AUC values.

This is not true. Last Monday we saw that ROC curves are insensitive to class prevalence: same model, different class prevalence = same ROC-AUC.

Today, we’ll debunk the myth that ROC-curves are overly optimistic.

ROC-Curves reflect the ability of the model to separate the classes correctly

While writing my new book Imbalanced Data: Myths, Mistakes and Modern Solutions I ran several experiments, many to debunk common myths about working with imbalance data, and many others to put undersampling and oversampling to the test.

Today, I’ll show you that this myth, that the ROC-curve is overly optimistic, is false.

I trained several ensemble models, including random forests, XGBoost, Catboost and lightGBM across 37 datasets with different degrees of class imbalance, and evaluated them using ROC-curves. Here’s what I saw.

For many datasets, I found ROC curves close to the top-left corner of the plot with AUC values close to 1. However, this behaviour was far from universal.

Take a look at the following figure. Some datasets (e.g., car_eval_4 and protein_homo) produce near-perfect ROC curves, whereas others (e.g., coil_2000 and scene) yield much lower AUC values despite being imbalanced:

The shape of the ROC curve therefore reflects how well the model separates the classes, not the prevalence of the minority class.

High ROC-AUC values, like those observed for the protein homology dataset, were not an artefact of class imbalance. Instead, they reflected that the model genuinely achieved strong class separation.

To investigate further, I plotted pairwise combinations of six features from the protein homology dataset, colouring observations according to their class label.

Take a look at the following image now. In most panels, the red and blue points occupy distinct regions of the feature space, with only limited overlap.

This indicates that the classes are already well separated by the original features, making the prediction task inherently easier and explaining why most models achieve ROC-AUC values close to one.

So bottom line: ROC curves are not overly optimistic by default. Instead, the truly reflect the capacity of the model to discriminate among classes.

Hence, if your models achieve a high ROC-AUC, that’s something to be happy about rather than worry over.

If you liked this article, consider supporting me by buying my new book: Imbalanced Data: Myths, Mistakes and Modern Solutions.

I hope this information was useful!

Wishing you a successful week ahead - see you next Monday! 👋🏻

Sole


Ready to enhance your skills?

Our specializations, courses and books are here to assist you:


Hi…I’m Sole
​
I’m the main instructor at Train in Data. My work as a data scientist includes creating and implementing machine learning models for evaluating insurance claims, managing credit risk, and detecting fraud. In 2018, I was honoured with a Data Science Leaders' award, and in 2019 and again in 2024, I was acknowledged as one of LinkedIn's voices in data science and analytics.

You are receiving this email because you subscribed to our newsletter, signed up on our website, purchased or downloaded any products from us.


Follow us on social media

Copyright (C) 2026 Train in Data. All rights reserved.

​
​Unsubscribe · Preferences​

Train in Data

by Soledad Galli, PhD | Data scientist | Python Developer | Best-selling instructor | Book author | 👉 www.trainindata.com

Read more from Train in Data
Your hyperparameter search is probably bigger than it needs to be

Welcome! Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, ormethod to help you stay sharp in the field. I hope you find it useful! Something exciting is coming this Black Friday! 👀 November is here, and that means Black Friday is just around the corner! Have you had your eye on one of our courses for a while, or been waiting for the right moment to get started? Well, we have something coming up that you might want to look out for! We're preparing...

After months with Claude and ChatGPT, here is where each one wins

Welcome! Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, ormethod to help you stay sharp in the field. I hope you find it useful! Our new course is out! Master Tree Ensemble Models: from Decision Trees to Gradient Boosting! XGBoost, LightGBM, and CatBoost are everywhere, and generative AI can now set them up in minutes. But can you explain what they are doing, why they make certain decisions, or whether the generated configuration makes sense?...

Grid or random search? You might be using the wrong one

Welcome! Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, ormethod to help you stay sharp in the field. I hope you find it useful! Coming soon: Master Tree Ensemble Models. From Decision Trees to Gradient Boosting! We’re developing a comprehensive new course on the models that dominate machine learning for tabular data, starting with decision trees and progressing through powerful ensemble methods. The main focus will be on the all-time favourites:...