Which hyperparameters should you tune?
Anyone who opens the XGBoost documentation for the first time finds a long list of hyperparameters, from the learning rate and the depth of the trees to regularisation terms, sampling ratios and a few options whose purpose is far from obvious.
Faced with that list, it is hard to know where to start.
Most people deal with it in one of two ways. Some put every hyperparameter they can find into one huge search and leave it running for hours, while others copy the search space from a blog post or a Kaggle notebook and hope it suits their data.
Both approaches are understandable, especially when we are just starting out. But the first one burns a lot of computing time, and the second one leaves us with a model tuned for somebody else's problem.
Only a few hyperparameters matter
The good news is that research has shown that only a handful of hyperparameters drive most of the performance. In their classic paper on random search, Bergstra and Bengio suggested that searches over many hyperparameters succeed because most of them barely affect the result.
Van Rijn and Hutter studied hyperparameter importance across 100 datasets and found that, for random forests and AdaBoost, the same hyperparameters tend to stay the most important across datasets, and that focusing the search on them gives better results.
Probst, Boulesteix and Bischl benchmarked six popular algorithms on 38 datasets to measure the tunability of their hyperparameters, precisely to help practitioners decide which ones are worth tuning, and whether tuning is worth the time at all.
This means we can run smaller, faster searches and still find very good models. If you'd like to see how, I cover search spaces and search strategies in detail in my course Master Hyperparameter Tuning for Tabular Learning.
The hyperparameters that matter for tree ensembles
For gradient boosting machines like XGBoost and LightGBM, these are the hyperparameters worth starting with:
- Learning rate and number of trees. These two work together, because a smaller learning rate needs more trees to reach the same performance. A common approach is to fix a small learning rate and let early stopping find the number of trees.
- Tree complexity. The maximum depth in XGBoost, or the number of leaves in LightGBM, together with the minimum number of observations per leaf, controls how much each tree can learn, and it is our main defence against overfitting.
Random forests are much more forgiving, and they often perform well with their default values. When we do tune them, these are the ones that matter most:
- max_features, the number of features considered at each split, which is usually the most influential.
- min_samples_leaf or max_depth, which control how deep and specific each tree becomes.
- max_samples, the fraction of observations used to build each tree, which can help with large datasets.
The number of trees behaves differently from the rest, because adding trees tends to improve performance until it plateaus. So it is better treated as a computing budget than as something to tune.
Why each of these hyperparameters matters, and how they shape the trees and the final model, is exactly what we cover in our coming course, Master Tree Ensemble Models: From Decision Trees to Gradient Boosting, which we'll launch in November.
To wrap up
Tuning becomes much easier once we know which few hyperparameters deserve our attention, because our searches get smaller, faster and easier to interpret. If you'd like to learn how to define search spaces, choose the right search strategy and evaluate the results with confidence, join me in Master Hyperparameter Tuning for Tabular Learning. I look forward to seeing you there.
I hope this information was useful!
Wishing you a successful week ahead - see you next Monday! 👋🏻
Sole
Ready to enhance your skills?
Our courses and books are here to assist you:
