Does the number of trees hurt ensemble performance?


Welcome!

Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, or
method to help you stay sharp in the field. I hope you find it useful!


Black Friday is around the corner!

November is here, and that means Black Friday is just around the corner!

Have you had your eye on one of our courses for a while, or been waiting for the right moment to get started? In 2 weeks, you can get hold of our courses and books at a discounted price!

👀 Black Friday is coming. Stay tuned!

Does the number of trees hurt ensemble performance?

Bagging and boosting combine predictions from multiple models, often decision trees. In bagging, models are trained in parallel and their predictions are averaged. In boosting, models are trained sequentially, with each one trying to correct errors made by the previous models.

Does adding more trees improve an ensemble’s performance? If so, up to what point? Can adding trees make performance worse?

To investigate, I trained random forest, AdaBoost, and gradient boosting classifiers on five datasets.

Does more trees hurt model performance?

According to Leo Breiman, who introduced random forests, adding trees should not hurt the model's performance. Even if individual trees overfit, averaging their predictions smooths out that effect.

That’s what I saw in these five datasets: adding more trees did not reduce random forests’ test performance.

The two boosting models behaved differently. On three of the five datasets, test performance improved up to a point, much like it did for random forests. On the other two, performance declined as more trees were added.

In short:

  • Adding trees did not hurt random forest performance.
  • AdaBoost’s test performance declined on two datasets: churn and diabetes.
  • Gradient boosting’s test performance declined on two datasets: credit and diabetes, with a larger drop than AdaBoost.

The models’ sensitivity to the number of trees, from highest to lowest, was:

Gradient boosting > AdaBoost > random forest

Do the models overfit?

To investigate overfitting, I compared each model’s performance on the training set with its performance on the test set.

Random forests and gradient boosting reached a training ROC-AUC of 1 on all five datasets, indicating that they fit the training data very closely. The performance on the test set was lower.

AdaBoost also overfits to the training set, but to a lesser extent, with smaller performance differences between train and test in all 5 datasets. Interestingly, it did not overfit to the training set for the credit data.

To wrap up

For these datasets, adding trees to a random forest did not hurt test performance. Averaging many trees makes the ensemble insensitive to the number of trees.

Boosting was more sensitive. Because each tree is trained to address what the earlier trees missed, adding trees can eventually hurt test performance. The number of trees is therefore an important hyperparameter to tune.

If you'd like to understand how boosting builds its trees and how to make the most of early stopping in XGBoost, LightGBM and CatBoost, check out our new course Master Tree Ensemble Models: From Decision Trees to Gradient Boosting.​

I hope this information was useful!

See you next Monday! 👋🏻

Sole


Ready to enhance your skills?

Our specializations, courses and books are here to assist you:


Hi…I’m Sole
​
I’m a Python developer, AI educator, and developer advocate. I’m the creator and maintainer of Feature-engine and the main instructor at Train in Data.
​
I share practical Python lessons, open-source insights, and ideas to help our community build better software and grow as developers and data scientists.

You are receiving this email because you subscribed to our newsletter, signed up on our website, purchased or downloaded any products from us.


Follow us on social media

Copyright (C) 2026 Train in Data. All rights reserved.

​
​Unsubscribe · Preferences​

Train in Data

by Soledad Galli, PhD | Data scientist | Python Developer | Best-selling instructor | Book author | 👉 www.trainindata.com

Read more from Train in Data
Your hyperparameter search is probably bigger than it needs to be

Welcome! Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, ormethod to help you stay sharp in the field. I hope you find it useful! Something exciting is coming this Black Friday! 👀 November is here, and that means Black Friday is just around the corner! Have you had your eye on one of our courses for a while, or been waiting for the right moment to get started? Well, we have something coming up that you might want to look out for! We're preparing...

After months with Claude and ChatGPT, here is where each one wins

Welcome! Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, ormethod to help you stay sharp in the field. I hope you find it useful! Our new course is out! Master Tree Ensemble Models: from Decision Trees to Gradient Boosting! XGBoost, LightGBM, and CatBoost are everywhere, and generative AI can now set them up in minutes. But can you explain what they are doing, why they make certain decisions, or whether the generated configuration makes sense?...

Grid or random search? You might be using the wrong one

Welcome! Every Monday, I’ll drop a no-fluff, straight-to-the-point tip on a data science skill, tool, ormethod to help you stay sharp in the field. I hope you find it useful! Coming soon: Master Tree Ensemble Models. From Decision Trees to Gradient Boosting! We’re developing a comprehensive new course on the models that dominate machine learning for tabular data, starting with decision trees and progressing through powerful ensemble methods. The main focus will be on the all-time favourites:...