How to Choose the Right Hyperparameter Search
Every machine learning model comes with settings that shape how it learns. A random forest has a number of trees and a maximum depth. A gradient boosting model has a learning rate and a number of boosting rounds. We call these settings hyperparameters.
Hyperparameter optimisation is the process of trying different values and selecting the combination that performs best on validation data.
Most people use grid search or random search. If you have used Hyperopt or Optuna, you probably also heard of TPE. So which method should you use, why, and when?
Here are practical guidelines for choosing a strategy that fits your model, your search space, and your computing budget.
The two methods you will use most often
Grid search tries every combination from a list of values. Imagine tuning the depth of a decision tree with the values 3, 5, and 10, and the minimum number of samples per leaf with the values 5 and 20. Grid search trains all six combinations.
This works beautifully when you have one or two hyperparameters and only a few sensible values. Linear models, logistic regression, and K nearest neighbours often fit this situation.
Random search samples combinations from ranges or lists. Give it a budget of 30 trials and it explores 30 combinations of hyperparameters. The advantage over grid search is that ti samples from hyperparameter distributions instead of manually picked values.
Random search helps when several hyperparameters matter or when values such as the learning rate come from a continuous range.
Random search is a strong default for random forests, Extra Trees, gradient boosting, XGBoost, and CatBoost. It explores broad spaces efficiently and runs many trials in parallel.
Make grid and random search faster with multi-fidelity
Grid and random search give every configuration the full training budget. That means, for every hyperparameter combination, they train the model till the end.
Multi-fidelity methods begin with a small budget and increase it for the most promising hyperparameter combinations.
Successive Halving, for example, trains every combination on a small sample of the data or for a few boosting rounds. It keeps the most promising groups, gives them more resources, and repeats the process till it finds the winning configuration.
This approach is especially useful when models train for many epochs, use hundreds of boosting rounds, or work with a large dataset. You still choose configurations with grid or random search. Successive Halving decides how much training time each configuration receives.
By stopping unpromising configurations early, it saves a lot of computing time, accelerating the search substantially. If you are not using successive halving, you are probably wasting a lot of time 😊
Where TPE enters the picture
TPE stands for Tree structured Parzen Estimator. It is an adaptive search method. Instead of choosing hyperparameter values at random, it chooses them from more promising areas.
TPE separates previous trials into a stronger group and a weaker group. It then looks for parameter values that appear frequently in the stronger group and less frequently in the weaker one. As more results arrive, its suggestions become more informed.
TPE is useful when each trial takes meaningful time or the search space contains many parameters. By selecting hyperparameters that are promising, with less trials, so the theory goes, it can find better values.
In my personal opinion, TPE became popular before the multi-fidelity models were described. For tabular data, I’d recommend you stick to random search and make it faster with successive halving.
So, which method should you choose?
👉 Choose grid search when you have one or two hyperparameters, a small set of candidate values, and quick model fits.
👉 Choose random search for more complex scikit-learn models or the sklearn compatible versions of XGBoost or lightGBM.
👉 Use Successive Halving to make random search faster, particularly for tree based ensemble methods.
👉 Choose TPE when trials are costly, the space is large, or the parameters follow conditional rules. Its learning process can find strong regions with fewer trials.
If you liked this article, you’ll also enjoy my recently updated course: Master Hyperparameter Tuning for Tabular Learning.
I hope this information was useful!
Wishing you a successful week ahead - see you next Monday! 👋🏻
Sole
Ready to enhance your skills?
Our courses and books are here to assist you:
