Performance metrics shape the model
We usually think of performance metrics as tools to evaluate a model. We train the model, calculate a metric, and use the result to select the best model.
But the metric plays a much bigger role.
When we use it to tune hyperparameters or select features, it helps determine which model we keep, which variables enter that model, and how much influence each variable has. In this way, the metric shapes both the final predictions and the information we extract from the model.
Let’s take a look at this with an example.
One model, three objectives
I ran an experiment with logistic regression on five thousand observations and ten features. Three features carried original information, five were redundant, and the classes overlapped moderately, with a seventy to thirty (70:30) class distribution.
I used the same training and test sets, pipeline, five cross validation folds, and hyperparameter grid. Only the optimisation metric changed. I tuned one model for accuracy, one for ROC AUC, and one for log loss.
Each metric selected a different regularisation.
- Accuracy selected C 0.01 with equal L1 and L2 regularisation.
- ROC AUC selected C 0.001 with L2 regularisation.
- Log loss selected C 0.1 with L1 regularisation.
The results are shown here:
The estimator and the data remained the same, while the metric used for optimisation resulted in three different fitted models.
Different models, similar performance
I evaluated all three models on the same held out test set, using accuracy, ROC-AUC, and log loss for every model.
Their predictive performance was remarkably similar as shown on this table:
If we looked only at these results, we could easily conclude that the tuning objective made little difference. All three models separate the classes at a similar level, and all three provide similar predictions on new observations.
The coefficients tell another story
The picture changed when I inspected the coefficients of the logistic regression. Some features received a stronger role under one objective and a weaker role under another.
The intercept also changed, from minus 0.967 for the ROC-AUC model, to minus 1.137 for the accuracy model, and minus 1.264 for the log loss model.
For logistic regression, coefficients are often used to discuss the direction and relative importance of the relationships in the data. Different coefficients can therefore lead us to different conclusions about which variables matter most and how strongly they influence the prediction.
The metric is part of the model
A performance metric does more than measure performance. During model selection, it defines what better means and results in a particular model.
Accuracy rewards correct class decisions, ROC-AUC rewards ranking, and log loss rewards reliable probabilities. Each metric asks a different question, so each can produce a different model, even when their final performance looks similar.
This also affects the coefficients, feature importance, and conclusions we extract.
Metric selection therefore belongs at the beginning of the modelling process, guided by the purpose of the model and the information we want from it.
References
You can find the code for the experiment described here at this link. If you like the repo, please give it a star.
If you liked this article, you’ll also enjoy my new book: Imbalanced Data: Myths, Mistakes and Modern Solutions.
I hope this information was useful!
Wishing you a successful week ahead - see you next Monday! 👋🏻
Sole
Ready to enhance your skills?
Our specializations, courses and books are here to assist you:
