Your hyperparameter search is probably bigger than it needs to be
Say we want to tune 4 hyperparameters of a gradient boosting model and try 5 values for each. That gives us 625 combinations, and with five fold cross validation, 3,125 model fits. At one minute per fit, the search would run for just over 52 hours.
Most of us accept that cost because we assume the best model hides somewhere in that grid. But here's the thing: when we look closely at the results of a search, a large part of the search space returns models that perform similarly.
Show me the data
I ran a random search with 60 hyperparameter combinations on scikit-learn's HistGradientBoostingClassifier. I tuned the learning rate, the number of leaves, the minimum number of samples per leaf, the regularisation and the fraction of features used at each split.
I evaluated every combination with five fold cross validation and ROC-AUC, on a simulated dataset with 5,000 observations.
Then I sorted the combinations by their mean ROC-AUC and plotted them together with their standard deviation across folds:
The best combination showed a ROC-AUC of 0.930 Β± 0.015. But 39 of the 60 combinations had a mean score within one standard deviation of it.
Among those 39, the learning rate ranged from 0.01 to 0.47 and the trees had anything from 7 to 62 leaves.
What does this mean? That there isnβt a unique set of hyperparameters to return a good performing model. There are several. So we donβt need to examine them all so exhaustively.
Search wide first, then go deep
Instead of one huge search, we can work in stages. We start with a short, coarse search to discard the regions that clearly perform worse, and then spend our computing budget on a deeper search in the promising region.
The good thing is that we can do this automatically with successive halving or early stopping.
Successive halving
Successive halving trains every hyperparameter combination on a small part of the data, keeps the best third, and gives the survivors more data, until only a few remain. Here's a minimal example, where I narrowed the number of leaves to 16 or more after the first search:
from scipy.stats import loguniform, randint
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import HalvingRandomSearchCV
β β
param_distributions = {
"learning_rate": loguniform(0.01, 0.5),
"max_leaf_nodes": randint(16, 64),
"min_samples_leaf": randint(5, 200),
}
β β
search = HalvingRandomSearchCV(
HistGradientBoostingClassifier(max_iter=200, early_stopping=False, random_state=0),
param_distributions,
n_candidates=60,
factor=3,
resource="n_samples",
min_resources=500,
scoring="roc_auc",
cv=5,
random_state=0,
)
search.fit(X, y)
print(search.best_params_)
|
The search trained 60 candidates on 500 observations, 20 on 1,500 and 7 on 4,500. It took 93 seconds, while a plain random search over the same 60 candidates took 154.
The two searches picked different winners. The one chosen by successive halving ranked fourth in the full random search, with a ROC-AUC of 0.927 Β± 0.010 against 0.928 for the top one, but well within the error rate.
Early stopping
Boosting models add one tree at a time, so the number of trees doesn't need a search of its own. We set a generous maximum and let the model stop when the score on a validation set no longer improves:
model = HistGradientBoostingClassifier(
learning_rate=0.05,
max_iter=2000,
early_stopping=True,
validation_fraction=0.1,
n_iter_no_change=20,
random_state=0,
)
model.fit(X_train, y_train)
print(model.n_iter_)
|
The model stopped after 92 trees instead of 2,000 and trained about ten times faster. With cross validation, it reached a ROC-AUC of 0.924 Β± 0.019, against 0.929 Β± 0.014 for the model that used all 2,000 trees.
HistGradientBoosting sets aside 10% of the training data to decide when to stop, so the data we use to score the model has no say in it. With XGBoost and LightGBM we pass our own validation set, so it's worth keeping it separate from the one we use for evaluation.
For early stopping is a good idea to test the main hyperparameters manually to understand where the optimal values lie, and then move on to early stopping.
To wrap up
Next time you run a search, plot the mean and standard deviation of every candidate from cv_results_ before picking a winner. If many of them overlap, we can stop chasing small differences and narrow the search to the region that matters.
If you'd like to learn how to set up successive halving, define search spaces and read the results of a search with confidence, join me in Master Hyperparameter Tuning for Tabular Learning.
And if you'd like to understand how boosting builds its trees and how to make the most of early stopping in XGBoost, LightGBM and CatBoost, check out our new course Master Tree Ensemble Models: From Decision Trees to Gradient Boosting.β
I hope this information was useful!
Wishing you a successful week ahead - see you next Monday! ππ»
Sole
Ready to enhance your skills?
Our specializations, courses and books are here to assist you: