<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/random-forests-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://datascienceplus.com/random-forests-in-r/"> R Programming – DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Ensemble Learning is a type of Supervised Learning Technique in which the basic idea is to generate multiple Models on a training dataset and then simply combining(average) their Output Rules or their Hypothesis \( H_x \) to generate a Strong Model which performs very well and does not overfits and which balances the Bisa-Variance Tradeoff too.</p>
<p>The idea is that instead of producing a single complicated and complex Model which might have a high variance which will lead to Overfitting or might be too simple and have a high bias which leads to Underfitting, we will generate lots of Models by training on Training Set and at the end combine them. Such a technique is Random Forest which is a popular Ensembling technique is used to improve the predictive performance of Decision Trees by reducing the variance in the Trees by averaging them. Decision Trees are considered very simple and easily interpretable as well as understandable Modelling techniques, but a major drawback in them is that they have a poor predictive performance and poor Generalization on Test Set. They are also referred to as Weak Learners which are Learners which always perform better than chance and have an error less than 50 %.</p>
<h2>Random Forests</h2>
<p>Random Forests are similar to a famous Ensemble technique called Bagging but have a different tweak in it. In Random Forests the idea is to decorrelate the several trees which are generated on the different bootstrapped samples from training Data.And then we simply reduce the Variance in the Trees by averaging them.<br>
Averaging the Trees helps us to reduce the variance and also improve the Perfomance of Decision Trees on Test Set and eventually avoid Overfitting.</p>
<p>The idea is to build lots of Trees in such a way to make the Correlation between the Trees smaller.</p>
<p>Another major difference is that we only consider a Random subset of predictors \( m \) each time we do a split on training examples.Whereas usually in Trees we find all the predictors while doing a split and choose best amongst them. Typically \( m=\sqrt{p} \) where \(p\) are the number of predictors.</p>
<p>Now it seems crazy to throw away lots of predictors, but it makes sense because the effect of doing so is that each tree uses different predictors to split data at various times.</p>
<p><i>So by doing this trick of throwing away Predictors, we have de-correlated the Trees and the resulting average seems a little better.</i></p>
<h2>Implementation in R</h2>
<p>loading the required packages</p>
<pre>
require(randomForest)
require(MASS)#Package which contains the Boston housing dataset
attach(Boston)
set.seed(101)

dim(Boston)
<em>## [1] 506 14
</em>
</pre>
<h3>Saperating Training and Test Sets</h3>
<pre>
#training Sample with 300 observations
train=sample(1:nrow(Boston),300)
?Boston  #to search on the dataset
</pre>
<p>We are going to use variable ′medv′ as the Response variable, which is the Median Housing Value. We will fit 500 Trees.</p>
<h3>Fitting the Random Forest</h3>
<p>We will use all the Predictors in the dataset.</p>
<pre>
Boston.rf=randomForest(medv ~ . , data = Boston , subset = train)
Boston.rf
<em>## ## Call:
## randomForest(formula = medv ~ ., data = Boston, subset = train) ## Type of random forest: regression
## Number of trees: 500
## No. of variables tried at each split: 4
## ## Mean of squared residuals: 12.62686
## % Var explained: 84.74
</em>
</pre>
<p>The above Mean Squared Error and Variance explained are calculated using <i>Out of Bag Error</i> Estimation.In this \(\frac23\) of Training data is used for training and the remaining \( \frac13 \) is used to Validate the Trees. Also, the number of variables randomly selected at each split is 4.</p>
<p>Plotting the Error vs Number of Trees Graph.</p>
<pre>
plot(Boston.rf)
</pre>
<p><a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/07/download-5.png?ssl=1" target="_blank"><img sizes="(max-width: 490px) 100vw, 490px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/07/download-5-490x350.png?w=450&ssl=1" alt="" class="wp-image-7866" data-recalc-dims="1"></a></p>
<p>This plot shows the Error and the Number of Trees.We can easily notice that how the Error is dropping as we keep on adding more and more trees and average them.</p>
<h3>Now we can compare the Out of Bag Sample Errors and Error on Test set</h3>
<p>The above Random Forest model chose Randomly 4 variables to be considered at each split. We could now try all possible 13 predictors which can be found at each split.</p>
<pre>
oob.err=double(13)
test.err=double(13)

#mtry is no of Variables randomly chosen at each split
for(mtry in 1:13) 
{
  rf=randomForest(medv ~ . , data = Boston , subset = train,mtry=mtry,ntree=400) 
  oob.err[mtry] = rf$mse[400] #Error of all Trees fitted
  
  pred&lt;-predict(rf,Boston[-train,]) #Predictions on Test Set for each Tree
  test.err[mtry]= with(Boston[-train,], mean( (medv - pred)^2)) #Mean Squared Test Error
  
  cat(mtry,&quot; &quot;) #printing the output to the console
  
}
<em>## 1 2 3 4 5 6 7 8 9 10 11 12 13
</em>
</pre>
<p>Test Error</p>
<pre>
test.err
<em>## [1] 26.06433 17.70018 16.51951 14.94621 14.51686 14.64315 14.60834
## [8] 15.12250 14.42441 14.53687 14.89362 14.86470 15.09553
</em>
</pre>
<p>Out of Bag Error Estimation</p>
<pre>
oob.err
<em>## [1] 19.95114 13.34894 13.27162 12.44081 12.75080 12.96327 13.54794
## [8] 13.68273 14.16359 14.52294 14.41576 14.69038 14.72979
</em>
</pre>
<p>What happens is that we are growing 400 trees for 13 times i.e for all 13 predictors.</p>
<h3>Plotting both Test Error and Out of Bag Error</h3>
<pre>
matplot(1:mtry , cbind(oob.err,test.err), pch=19 , col=c(&quot;red&quot;,&quot;blue&quot;),type=&quot;b&quot;,ylab=&quot;Mean Squared Error&quot;,xlab=&quot;Number of Predictors Considered at each Split&quot;)
legend(&quot;topright&quot;,legend=c(&quot;Out of Bag Error&quot;,&quot;Test Error&quot;),pch=19, col=c(&quot;red&quot;,&quot;blue&quot;))
</pre>
<p><a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/07/download-1-1.png?ssl=1" target="_blank"><img sizes="(max-width: 490px) 100vw, 490px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/07/download-1-1-490x350.png?w=450&ssl=1" alt="" class="wp-image-7868" data-recalc-dims="1"></a></p>
<p>Now what we observe is that the Red line is the Out of Bag Error Estimates and the Blue Line is the Error calculated on Test Set. Both curves are quite smooth and the error estimates are somewhat correlated too. The Error Tends to be minimized at around \( mtry = 4 \) .</p>
<p>On the Extreme Right Hand Side of the above plot, we considered all possible 13 predictors at each Split which is only Bagging.</p>
<h2>Conclusion</h2>
<p>Now in this article, I gave a simple overview of Random Forests and how they differ from other Ensemble Learning Techniques and also learned how to implement such complex and Strong Modelling Technique in R with a simple package <code>randomForest</code>.Random Forests are a very Nice technique to fit a more Accurate Model by averaging Lots of Decision Trees and reducing the Variance and avoiding Overfitting problem in Trees. Decision Trees themselves are poor performance wise, but when used with Ensembling Techniques like Bagging, Random Forests etc, their predictive performance is improved a lot.Now obviously there are various other packages in R which can be used to implement Random Forests in R.</p>
<p>I hope the tutorial is enough to get you started with implementing Random Forests in R or at least understand the basic idea behind how this amazing Technique works.</p>
<p>Thanks for reading the article and make sure to like and share it.</p>
<ul></ul>
<p><strong>
<p>Related Post</p>
<p>
<ol>
<li><a rel="nofollow" href="https://datascienceplus.com/network-analysis-of-game-of-thrones/" title="Permanent Link toNetwork analysis of Game of Thrones" target="_blank">Network analysis of Game of Thrones</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/structural-changes-in-global-warming/" title="Permanent Link toStructural Changes in Global Warming" target="_blank">Structural Changes in Global Warming</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/deep-learning-with-r/" title="Permanent Link toDeep Learning with R" target="_blank">Deep Learning with R</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/unsupervised-learning-and-text-mining-of-emotion-terms-using-r/" title="Permanent Link toUnsupervised Learning and Text Mining of Emotion Terms Using R" target="_blank">Unsupervised Learning and Text Mining of Emotion Terms Using R</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/using-mca-and-variable-clustering-in-r-for-insights-in-customer-attrition/" title="Permanent Link toUsing MCA and variable clustering in R for insights in customer attrition" target="_blank">Using MCA and variable clustering in R for insights in customer attrition</a></li>
</ol></strong>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://datascienceplus.com/random-forests-in-r/"> R Programming – DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/BgfAC6nCnr0" height="1"></a></div>