In a recent video, I covered Random Forests and Neural Nets as part of the codecentric.ai Bootcamp.
In the most recent video, I covered Gradient Boosting and XGBoost.
You can find the video on YouTube and the slides on slides.com. Both are again in German with code examples in Python.
But below, you find the English version of the content, plus code examples in R for caret, xgboost and h2o. :-)
Like Random Forest, Gradient Boosting is another technique for performing supervised machine learning tasks, like classification and regression. The implementations of this technique can have different names, most commonly you encounter Gradient Boosting machines (abbreviated GBM) and XGBoost. XGBoost is particularly popular because it has been the winning algorithm in a number of recent Kaggle competitions.
Similar to Random Forests, Gradient Boosting is an ensemble learner. This means it will create a final model based on a collection of individual models. The predictive power of these individual models is weak and prone to overfitting but combining many such weak models in an ensemble will lead to an overall much improved result. In Gradient Boosting machines, the most common type of weak model used is decision trees - another parallel to Random Forests.
How Gradient Boosting works
Let’s look at how Gradient Boosting works. Most of the magic is described in the name: “Gradient” plus “Boosting”.
Boosting builds models from individual so called “weak learners” in an iterative way. In the Random Forests part, I had already discussed the differences between Bagging and Boosting as tree ensemble methods. In boosting, the individual models are not built on completely random subsets of data and features but sequentially by putting more weight on instances with wrong predictions and high errors. The general idea behind this is that instances, which are hard to predict correctly (“difficult” cases) will be focused on during learning, so that the model learns from past mistakes. When we train each ensemble on a subset of the training set, we also call this Stochastic Gradient Boosting, which can help improve generalizability of our model.
The gradient is used to minimize a loss function, similar to how Neural Nets utilize gradient descent to optimize (“learn”) weights. In each round of training, the weak learner is built and its predictions are compared to the correct outcome that we expect. The distance between prediction and truth represents the error rate of our model. These errors can now be used to calculate the gradient. The gradient is nothing fancy, it is basically the partial derivative of our loss function - so it describes the steepness of our error function. The gradient can be used to find the direction in which to change the model parameters in order to (maximally) reduce the error in the next round of training by “descending the gradient”.
In Neural nets, gradient descent is used to look for the minimum of the loss function, i.e. learning the model parameters (e.g. weights) for which the prediction error is lowest in a single model. In Gradient Boosting we are combining the predictions of multiple models, so we are not optimizing the model parameters directly but the boosted model predictions. Therefore, the gradients will be added to the running training process by fitting the next tree also to these values.
Because we apply gradient descent, we will find learning rate (the “step size” with which we descend the gradient), shrinkage (reduction of the learning rate) and loss function as hyperparameters in Gradient Boosting models - just as with Neural Nets. Other hyperparameters of Gradient Boosting are similar to those of Random Forests:
the number of iterations (i.e. the number of trees to ensemble),
the number of observations in each leaf,
tree complexity and depth,
the proportion of samples and
the proportion of features on which to train on.
Gradient Boosting Machines vs. XGBoost
XGBoost stands for Extreme Gradient Boosting; it is a specific implementation of the Gradient Boosting method which uses more accurate approximations to find the best tree model. It employs a number of nifty tricks that make it exceptionally successful, particularly with structured data. The most important are
1.) computing second-order gradients, i.e. second partial derivatives of the loss function (similar to Newton’s method), which provides more information about the direction of gradients and how to get to the minimum of our loss function. While regular gradient boosting uses the loss function of our base model (e.g. decision tree) as a proxy for minimizing the error of the overall model, XGBoost uses the 2nd order derivative as an approximation.
2.) And advanced regularization (L1 &amp; L2), which improves model generalization.
XGBoost has additional advantages: training is very fast and can be parallelized / distributed across clusters.
Code in R
Here is a very quick run through how to train Gradient Boosting and XGBoost models in R with caret, xgboost and h2o.
Data
First, data: I’ll be using the ISLR package, which contains a number of datasets, one of them is College.
Statistics for a large number of US Colleges from the 1995 issue of US News and World Report.
library(tidyverse)
library(ISLR)
ml_data %
glimpse()
## Observations: 777
## Variables: 18
## $ Private Yes, Yes, Yes, Yes, Yes, Yes, Yes, Yes, Yes, Yes, ...
## $ Apps 1660, 2186, 1428, 417, 193, 587, 353, 1899, 1038, ...
## $ Accept 1232, 1924, 1097, 349, 146, 479, 340, 1720, 839, 4...
## $ Enroll 721, 512, 336, 137, 55, 158, 103, 489, 227, 172, 4...
## $ Top10perc 23, 16, 22, 60, 16, 38, 17, 37, 30, 21, 37, 44, 38...
## $ Top25perc 52, 29, 50, 89, 44, 62, 45, 68, 63, 44, 75, 77, 64...
## $ F.Undergrad 2885, 2683, 1036, 510, 249, 678, 416, 1594, 973, 7...
## $ P.Undergrad 537, 1227, 99, 63, 869, 41, 230, 32, 306, 78, 110,...
## $ Outstate 7440, 12280, 11250, 12960, 7560, 13500, 13290, 138...
## $ Room.Board 3300, 6450, 3750, 5450, 4120, 3335, 5720, 4826, 44...
## $ Books 450, 750, 400, 450, 800, 500, 500, 450, 300, 660, ...
## $ Personal 2200, 1500, 1165, 875, 1500, 675, 1500, 850, 500, ...
## $ PhD 70, 29, 53, 92, 76, 67, 90, 89, 79, 40, 82, 73, 60...
## $ Terminal 78, 30, 66, 97, 72, 73, 93, 100, 84, 41, 88, 91, 8...
## $ S.F.Ratio 18.1, 12.2, 12.9, 7.7, 11.9, 9.4, 11.5, 13.7, 11.3...
## $ perc.alumni 12, 16, 30, 37, 2, 11, 26, 37, 23, 15, 31, 41, 21,...
## $ Expend 7041, 10527, 8735, 19016, 10922, 9727, 8861, 11487...
## $ Grad.Rate 60, 56, 54, 59, 15, 55, 63, 73, 80, 52, 73, 76, 74...
Gradient Boosting in caret
The most flexible R package for machine learning is caret. If you go to the Available Models section in the online documentation and search for “Gradient Boosting”, this is what you’ll find:
Model
method Value
Type
Libraries
Tuning Parameters
eXtreme Gradient Boosting
xgbDART
Classification, Regression
xgboost, plyr
nrounds, max_depth, eta, gamma, subsample, colsample_bytree, rate_drop, skip_drop, min_child_weight
eXtreme Gradient Boosting
xgbLinear
Classification, Regression
xgboost
nrounds, lambda, alpha, eta
eXtreme Gradient Boosting
xgbTree
Classification, Regression
xgboost, plyr
nrounds, max_depth, eta, gamma, colsample_bytree, min_child_weight, subsample
Gradient Boosting Machines
gbm_h2o
Classification, Regression
h2o
ntrees, max_depth, min_rows, learn_rate, col_sample_rate
Stochastic Gradient Boosting
gbm
Classification, Regression
gbm, plyr
n.trees, interaction.depth, shrinkage, n.minobsinnode
A table with the different Gradient Boosting implementations, you can use with caret. Here I’ll show a very simple Stochastic Gradient Boosting example:
library(caret)
# Partition into training and test data
set.seed(42)
index