<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/gradient-boosting-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://datascienceplus.com/gradient-boosting-in-r/"> R Programming – DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><b>Boosting</b> is another famous ensemble learning technique in which we are not concerned with reducing the variance of learners like in <b>Bagging</b> where our aim is to reduce the high variance of learners by averaging lots of models fitted on bootstrapped data samples generated with replacement from training data, so as to avoid overfitting.</p>
<p>Another major difference between both the techniques is that in Bagging the various models which are generated are independent of each other and have equal weightage .Whereas Boosting is a sequential process in which each next model which is generated is added so as to improve a bit from the previous model.Simply saying each of the model that is added to mix is added so as to improve on the performance of the previous collection of models.In Boosting we do weighted averaging. </p>
<p>Both the ensemble techniques are common in terms of generating lots of models on training data and using their combined power to increase the accuracy of the final model which is formed by combining them.</p>
<p>But Boosting is more towards <b>Bias</b> i.e simple learners or more specifically <b>Weak learners</b>. Now a weak learner is a learner which always learns something i.e does better than chance and also has error rate less then 50%.The best example of a weak learner is a Decision tree.This is the reason we generally use ensemble technique on decision trees to improve its accuracy and performance.</p>
<p>In Boosting each tree or Model is grown or trained using the <b>hard</b> examples.By hard I mean all the training examples \( (x_i,y_i) \) for which a previous model produced incorrect output \(Y\).Boosting boosts the performance of a simple base-learner by iteratively shifting the focus towards problematic training observations that are difficult to predict.Now that information from the previous model is fed to the next model.And the thing with boosting is that every new tree added to the mix will do better than the previous tree because it will learn from the mistakes of the previous models and try not to repeat them.Hence by this technique it will eventually convert a weak learner to a strong learner which is better and more accurate in generalization for unseen test examples.</p>
<p>An important thing to remember in boosting is that the base learner which is being boosted should not be a complex and complicated learner which has high variance for e.g a neural network with lots of nodes and high weight values.For such learners boosting will have inverse effects.</p>
<p>So I will explain Boosting with respect to decision trees in this tutorial because they can be regarded as weak learners most of the times.We will generate a <b>gradient boosting model</b>.</p>
<p>Gradient boosting generates learners using the same general boosting learning process. It first builds learner to predict the values/labels of samples, and calculate the loss (the difference between the outcome of the first learner and the real value). It will build a second learner to predict the loss after the first step. The step continues to learn the third, forth… until certain threshold.Gradient boosting identifies hard examples by calculating large residuals-\( (y_{actual}-y_{pred} ) \) computed in the previous iterations.</p>
<h2>Implementing Gradient Boosting</h2>
<p>Let’s use <code>gbm</code> package in R to fit gradient boosting model.</p>
<pre>
require(gbm)
require(MASS)#package with the boston housing dataset

#separating training and test data
train=sample(1:506,size=374)
</pre>
<p>We will use the Boston housing data to predict the median value of the houses.</p>
<pre>
Boston.boost=gbm(medv ~ . ,data = Boston[train,],distribution = &quot;gaussian&quot;,n.trees = 10000,
                  shrinkage = 0.01, interaction.depth = 4)
Boston.boost

summary(Boston.boost) #Summary gives a table of Variable Importance and a plot of Variable Importance

<em>gbm(formula = medv ~ ., distribution = &quot;gaussian&quot;, data = Boston[-train, ], n.trees = 10000, interaction.depth = 4, shrinkage = 0.01)
A gradient boosted model with gaussian loss function.
10000 iterations were performed.
There were 13 predictors of which 13 had non-zero influence.
>summary(Boston.boost)
var rel.inf
rm rm 36.96963915
lstat lstat 24.40113288
dis dis 10.67520770
crim crim 8.61298346
age age 4.86776735
black black 4.23048222
nox nox 4.06930868
ptratio ptratio 2.21423811
tax tax 1.73154882
rad rad 1.04400159
indus indus 0.80564216
chas chas 0.28507720
zn zn 0.09297068
</em>
</pre>
<p>The above Boosted Model is a Gradient Boosted Model which generates 10000 trees and the shrinkage parametet (\lambda= 0.01\) which is also a sort of learning Rate. Next parameter is the interaction depth which is the total splits we want to do.So here each tree is a small tree with only 4 splits.<br>
The summary of the Model gives a <b>feature importance plot</b>.In the above list is on the top is the most important variable and at last is the least important variable.<br>
And the 2 most important features which explain the maximum variance in the Data set is <code>lstat</code> i.e lower status of the population (percent) and <code>rm</code> which is average number of rooms per dwelling.</p>
<p>Variable importance plot–<br>
<a rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/08/gbm.png?ssl=1" target="_blank"><img sizes="(max-width: 701px) 100vw, 701px" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/08/gbm.png?w=450&ssl=1" alt="" class="wp-image-8089" data-recalc-dims="1"></a></p>
<h3>Plotting the Partial Dependence Plot</h3>
<p>The partial Dependence Plots will tell us the relationship and dependence of the variables \(X_i\) with the Response variable \(Y\).</p>
<pre>
#Plot of Response variable with lstat variable
plot(Boston.boost,i=&quot;lstat&quot;) 
#Inverse relation with lstat variable

plot(Boston.boost,i=&quot;rm&quot;) 
#as the average number of rooms increases the the price increases

</pre>
<p><a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/08/Rplot1.png?ssl=1" target="_blank"><img sizes="(max-width: 655px) 100vw, 655px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/08/Rplot1.png?w=450&ssl=1" alt="" class="wp-image-8090" data-recalc-dims="1"></a></p>
<p><a rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/08/Rplot2.png?ssl=1" target="_blank"><img sizes="(max-width: 790px) 100vw, 790px" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/08/Rplot2.png?w=450&ssl=1" alt="" class="wp-image-8091" data-recalc-dims="1"></a></p>
<p>The above plot simply shows the relation between the variables in the x-axis and the mapping function \(f(x)\) on the y-axis.First plot shows that <code>lstat</code> is negatively correlated with the response <code>mdev</code>, whereas the second one shows that <code>rm</code> is somewhat directly related to <code>mdev</code>.</p>
<pre>
cor(Boston$lstat,Boston$medv)#negetive correlation coeff-r

cor(Boston$rm,Boston$medv)#positive correlation coeff-r

<em>> cor(Boston$lstat,Boston$medv)
[1] -0.7376627
> cor(Boston$rm,Boston$medv)
[1] 0.6953599
</em>
</pre>
<h3>Prediction on Test Set</h3>
<p>We will compute the Test Error as a function of number of Trees.</p>
<pre>
n.trees = seq(from=100 ,to=10000, by=100) #no of trees-a vector of 100 values 

#Generating a Prediction matrix for each Tree
predmatrix&lt;-predict(Boston.boost,Boston[-train,],n.trees = n.trees)
dim(predmatrix) #dimentions of the Prediction Matrix

#Calculating The Mean squared Test Error
test.error&lt;-with(Boston[-train,],apply( (predmatrix-medv)^2,2,mean))
head(test.error) #contains the Mean squared test error for each of the 100 trees averaged

#Plotting the test error vs number of trees

plot(n.trees , test.error , pch=19,col=&quot;blue&quot;,xlab=&quot;Number of Trees&quot;,ylab=&quot;Test Error&quot;, main = &quot;Perfomance of Boosting on Test Set&quot;)

#adding the RandomForests Minimum Error line trained on same data and similar parameters
abline(h = min(test.err),col=&quot;red&quot;) #test.err is the test error of a Random forest fitted on same data
legend(&quot;topright&quot;,c(&quot;Minimum Test error Line for Random Forests&quot;),col=&quot;red&quot;,lty=1,lwd=1)


<em>dim(predmatrix)
[1] 206 100
head(test.error)
100 200 300 400 500 600 26.428346 14.938232 11.232557 9.221813 7.873472 6.911313 </em>
</pre>
<p><a rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/08/Rplot.png?ssl=1" target="_blank"><img sizes="(max-width: 707px) 100vw, 707px" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/08/Rplot.png?w=450&ssl=1" alt="" class="wp-image-8095" data-recalc-dims="1"></a></p>
<p>In the above plot the red line represents the least error obtained from training a Random forest with same data and same parameters and number of trees.Boosting outperforms Random Forests on same test dataset with lesser Mean squared Test Errors.</p>
<h2>Conclusion</h2>
<p>From the above plot we can notice that if boosting is done properly by selecting appropriate tuning parameters such as shrinkage parameter \(lambda\) ,the number of splits we want and the number of trees \(n\), then it can generalize really well and convert a weak learner to strong learner. Ensembling techniques are really well and tend to outperform a single learner which is prone to either overfitting or underfitting or generate thousands or hundreds of them,then combine them to produce a better and stronger model.</p>
<p>Hope you guys liked the article, make sure to like and share. Cheers!!.</p>
<ul></ul>
<p><strong>
<p>Related Post</p>
<p>
<ol>
<li><a rel="nofollow" href="https://datascienceplus.com/radial-kernel-support-vector-classifier/" title="Permanent Link toRadial kernel Support Vector Classifier" target="_blank">Radial kernel Support Vector Classifier</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/random-forests-in-r/" title="Permanent Link toRandom Forests in R" target="_blank">Random Forests in R</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/network-analysis-of-game-of-thrones/" title="Permanent Link toNetwork analysis of Game of Thrones" target="_blank">Network analysis of Game of Thrones</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/structural-changes-in-global-warming/" title="Permanent Link toStructural Changes in Global Warming" target="_blank">Structural Changes in Global Warming</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/deep-learning-with-r/" title="Permanent Link toDeep Learning with R" target="_blank">Deep Learning with R</a></li>
</ol></strong>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://datascienceplus.com/gradient-boosting-in-r/"> R Programming – DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/S2WjmzQNzzM" height="1"></a></div>