Your hyperparameter search is probably bigger than it needs to be


Welcome!

Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, or
method to help you stay sharp in the field. I hope you find it useful!


Something exciting is coming this Black Friday! πŸ‘€

November is here, and that means Black Friday is just around the corner!

Have you had your eye on one of our courses for a while, or been waiting for the right moment to get started? Well, we have something coming up that you might want to look out for!

We're preparing something special for you, so keep an eye out over the coming weeks.

πŸ‘€ Black Friday is coming. Stay tuned!

More details coming soon.


Your hyperparameter search is probably bigger than it needs to be

Say we want to tune 4 hyperparameters of a gradient boosting model and try 5 values for each. That gives us 625 combinations, and with five fold cross validation, 3,125 model fits. At one minute per fit, the search would run for just over 52 hours.

Most of us accept that cost because we assume the best model hides somewhere in that grid. But here's the thing: when we look closely at the results of a search, a large part of the search space returns models that perform similarly.

Show me the data

I ran a random search with 60 hyperparameter combinations on scikit-learn's HistGradientBoostingClassifier. I tuned the learning rate, the number of leaves, the minimum number of samples per leaf, the regularisation and the fraction of features used at each split.

I evaluated every combination with five fold cross validation and ROC-AUC, on a simulated dataset with 5,000 observations.

Then I sorted the combinations by their mean ROC-AUC and plotted them together with their standard deviation across folds:

The best combination showed a ROC-AUC of 0.930 Β± 0.015. But 39 of the 60 combinations had a mean score within one standard deviation of it.

Among those 39, the learning rate ranged from 0.01 to 0.47 and the trees had anything from 7 to 62 leaves.

What does this mean? That there isn’t a unique set of hyperparameters to return a good performing model. There are several. So we don’t need to examine them all so exhaustively.

Search wide first, then go deep

Instead of one huge search, we can work in stages. We start with a short, coarse search to discard the regions that clearly perform worse, and then spend our computing budget on a deeper search in the promising region.

The good thing is that we can do this automatically with successive halving or early stopping.

Successive halving

Successive halving trains every hyperparameter combination on a small part of the data, keeps the best third, and gives the survivors more data, until only a few remain. Here's a minimal example, where I narrowed the number of leaves to 16 or more after the first search:

from scipy.stats import loguniform, randint

from sklearn.ensemble import HistGradientBoostingClassifier

from sklearn.experimental import enable_halving_search_cv

from sklearn.model_selection import HalvingRandomSearchCV

​
​

param_distributions = {

"learning_rate": loguniform(0.01, 0.5),

"max_leaf_nodes": randint(16, 64),

"min_samples_leaf": randint(5, 200),

}

​
​

search = HalvingRandomSearchCV(

HistGradientBoostingClassifier(max_iter=200, early_stopping=False, random_state=0),

param_distributions,

n_candidates=60,

factor=3,

resource="n_samples",

min_resources=500,

scoring="roc_auc",

cv=5,

random_state=0,

)

search.fit(X, y)

print(search.best_params_)

The search trained 60 candidates on 500 observations, 20 on 1,500 and 7 on 4,500. It took 93 seconds, while a plain random search over the same 60 candidates took 154.

The two searches picked different winners. The one chosen by successive halving ranked fourth in the full random search, with a ROC-AUC of 0.927 Β± 0.010 against 0.928 for the top one, but well within the error rate.

Early stopping

Boosting models add one tree at a time, so the number of trees doesn't need a search of its own. We set a generous maximum and let the model stop when the score on a validation set no longer improves:

model = HistGradientBoostingClassifier(

learning_rate=0.05,

max_iter=2000,

early_stopping=True,

validation_fraction=0.1,

n_iter_no_change=20,

random_state=0,

)

model.fit(X_train, y_train)

print(model.n_iter_)

The model stopped after 92 trees instead of 2,000 and trained about ten times faster. With cross validation, it reached a ROC-AUC of 0.924 Β± 0.019, against 0.929 Β± 0.014 for the model that used all 2,000 trees.

HistGradientBoosting sets aside 10% of the training data to decide when to stop, so the data we use to score the model has no say in it. With XGBoost and LightGBM we pass our own validation set, so it's worth keeping it separate from the one we use for evaluation.

For early stopping is a good idea to test the main hyperparameters manually to understand where the optimal values lie, and then move on to early stopping.

To wrap up

Next time you run a search, plot the mean and standard deviation of every candidate from cv_results_ before picking a winner. If many of them overlap, we can stop chasing small differences and narrow the search to the region that matters.

If you'd like to learn how to set up successive halving, define search spaces and read the results of a search with confidence, join me in Master Hyperparameter Tuning for Tabular Learning.

And if you'd like to understand how boosting builds its trees and how to make the most of early stopping in XGBoost, LightGBM and CatBoost, check out our new course Master Tree Ensemble Models: From Decision Trees to Gradient Boosting.​

I hope this information was useful!

Wishing you a successful week ahead - see you next Monday! πŸ‘‹πŸ»

Sole


Ready to enhance your skills?

Our specializations, courses and books are here to assist you:


Hi…I’m Sole
​
I’m a Python developer, AI educator, and developer advocate. I’m the creator and maintainer of Feature-engine and the main instructor at Train in Data.
​
I share practical Python lessons, open-source insights, and ideas to help our community build better software and grow as developers and data scientists.

You are receiving this email because you subscribed to our newsletter, signed up on our website, purchased or downloaded any products from us.


Follow us on social media

Copyright (C) 2026 Train in Data. All rights reserved.

​
​Unsubscribe Β· Preferences​

Train in Data

by Soledad Galli, PhD | Data scientist | Python Developer | Best-selling instructor | Book author | πŸ‘‰ www.trainindata.com

Read more from Train in Data

Welcome! Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, ormethod to help you stay sharp in the field. I hope you find it useful! Black Friday is around the corner! November is here, and that means Black Friday is just around the corner! Have you had your eye on one of our courses for a while, or been waiting for the right moment to get started? In 2 weeks, you can get hold of our courses and books at a discounted price! πŸ‘€ Black Friday is coming....

After months with Claude and ChatGPT, here is where each one wins

Welcome! Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, ormethod to help you stay sharp in the field. I hope you find it useful! Our new course is out! Master Tree Ensemble Models: from Decision Trees to Gradient Boosting! XGBoost, LightGBM, and CatBoost are everywhere, and generative AI can now set them up in minutes. But can you explain what they are doing, why they make certain decisions, or whether the generated configuration makes sense?...

Grid or random search? You might be using the wrong one

Welcome! Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, ormethod to help you stay sharp in the field. I hope you find it useful! Coming soon: Master Tree Ensemble Models. From Decision Trees to Gradient Boosting! We’re developing a comprehensive new course on the models that dominate machine learning for tabular data, starting with decision trees and progressing through powerful ensemble methods. The main focus will be on the all-time favourites:...