Grid or random search? You might be using the wrong one
Are you using grid search when you need a continuous distribution? Passing a grid to random search? Running a long random search for just two or three hyperparameters?
Your search may be doing more work while exploring fewer useful values than you intended.
Are you using grid search for a continuous range?
Grid search evaluates every combination of the values you specify. Two hyperparameters with three values each give nine combinations. For example, if your grid is:
param_grid = {
"max_depth": [2, 3, 4],
"n_estimators": [100, 500, 1000],
}
|
Grid search will train and evaluate nine models: (2, 100), (2, 500), (2, 1000), (3, 100), (3, 500), (3, 1000), (4, 100), (4, 500), (4, 1000).
But here's the thing: a grid tests only the values you choose. It cannot explore what lies between them. For an integer hyperparameter with a small useful range, like max_depth, that's usually fine. But for n_estimators, we never test anything between 100 and 500, or between 500 and 1000, and the best value may sit there.
The problem gets worse for hyperparameters that are continuous by nature, such as the learning rate or a regularization coefficient. For these, consider random search and define the search space with continuous distributions instead of lists. That way you can explore many more values within your chosen ranges:
from scipy.stats import loguniform, randint, uniform
param_distributions = {
"max_depth": [3, 4, 5],
"n_estimators": randint(100, 1000),
"learning_rate": loguniform(1e-3, 0.3),
"subsample": uniform(0.5, 0.5), # from 0.5 to 1.0
}
|
Now n_estimators can take any integer between 100 and 999, learning_rate any value between 0.001 and 0.3 sampled on a log scale, and subsample any value between 0.5 and 1.0. Each candidate is a fresh draw, so with n_iter=30 you test 30 different learning rates rather than the three you would have hand-picked for a grid.
Are you passing a grid to random search?
Then you are still limited to the grid points. You lose the benefit of exploring values between them.
Multiply the number of values per hyperparameter to get the total number of combinations, then compare it with n_iter. If n_iter is equal to or larger than that total, scikit-learn caps the search at the grid size, and you are running grid search with extra steps. Use GridSearchCV instead. If you only want a subset of the combinations, set n_iter below the total.
Are you using random search for two or three hyperparameters?
With only a few candidate values, a small grid is often the simpler choice. If your hyperparameters are continuous, however, and you don’t know where the optimal value lies, then you need to set up a distribution.
Look at the parameter n_iter. For 2-3 hyperparameters, you may not need to train 50 models, with 20 you may already find the optimal values and save a lot of time.
Could you speed up your searches?
If you are running random search on a lot of hyperparameters, you could speed the search dramatically by using successive halving.
Successive halving starts candidates with a small resource budget, discards weaker performers, and gives the survivors more resources. Like this, while training many configurations, the weaker are stopped earlier, and you save time and computing resources.
Want to learn how to put it into practice? Check out my new course.
I hope this information was useful!
Wishing you a successful week ahead - see you next Monday! 👋🏻
Sole
Ready to enhance your skills?
Our specializations, courses and books are here to assist you:
