<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/how-random-forests-improve-simple-regression-trees/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://insightr.wordpress.com/2017/09/23/how-random-forests-improve-simple-regression-trees/"> R – insightR</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p><strong>By Gabriel Vasconcelos</strong></p>
</div>
<h2>Regression Trees</h2>
<p>In this post I am going to discuss some features of Regression Trees an Random Forests. Regression Trees are know to be very unstable, in other words, a small change in your data may drastically change your model. The Random Forest uses this instability as an advantage through bagging (you can see details about bagging <a rel="nofollow" href="https://insightr.wordpress.com/2017/05/22/bagging-the-perfect-solution-for-model-instability/" target="_blank">here</a>) resulting on a very stable model.</p>
<p><span></span></p>
<p>The first question is how a Regression Tree works. Suppose, fore example, that we have the number of points scored by a set of basketball players and we want to relate it to the player’s weight an height. The Regression Tree will simply split the height-weight space and assign a number of points to each partition. The figure below shows two different representations for a small tree. In the left we have the tree itself and in the right how the space is partitioned (the blue line shows the first partition and the red lines the following partitions). The numbers in the end of the tree (and in the partitions) represent the value of the response variable. Therefore, if a basketball player is higher than 1.85 meters and weights more than 100kg it is expected to score 27 points (I invented this data =] ).</p>
<p><img alt="" src="https://i1.wp.com/i.imgur.com/6HhbRVL.png?w=450&ssl=1" data-recalc-dims="1"></p>
<p> </p>
<p>You might be asking how I chose the partitions. In general, in each node the partition is chosen through a simple optimization problem to find the best pair variable-observation based on how much the new partition reduces the model error.</p>
<p>What I want to illustrate here is how unstable a Regression Tree can be. The package <code>tree</code> has some examples that I will follow here with some small modifications. The example uses computer CPUs data and the objective is to build a model for the CPU performance based on some characteristics. The data has 209 CPU observations that will be used to estimate two Regression Trees. Each tree will be estimate from a random re-sample with replacement. Since the data comes from the same place, it would be desirable to have similar results on both models.</p>
<p> </p>
<pre>library(ggplot2)
library(reshape2)
library(tree)
library(gridExtra)
data(cpus, package = &quot;MASS&quot;) # = Load Data

# = First Tree
set.seed(1) # = Seed for Replication
tree1 = tree(log(perf) ~ syct + mmin + mmax + cach + chmin + chmax,
             data = cpus[sample(1:209, 209, replace = TRUE), ])
plot(tree1);  text(tree1)
</pre>
<p><img alt="plot of chunk unnamed-chunk-28" src="https://i1.wp.com/i.imgur.com/7JyV7eU.png?w=456&ssl=1" data-recalc-dims="1"></p>
<pre># = Second Tree
set.seed(10)
tree2 = tree(log(perf) ~ syct + mmin + mmax + cach + chmin + chmax,
             data = cpus[sample(1:209,209, replace=TRUE), ])
plot(tree2);  text(tree2)
</pre>
<p><img alt="plot of chunk unnamed-chunk-28" src="https://i1.wp.com/i.imgur.com/sbjLGew.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>As you can see, the two trees are different from the start. We can use some figures to verify. First let us calculate the predictions of each model in the real data (not the re-sample). The first figure is a scatterplot of both predictions and the second figure shows their boxplots. Although the scatterplot shows some relation between the two predictions, it is far from good.</p>
<pre># = Calculate predictions
pred = data.frame(p1 = predict(tree1, cpus) ,p2 = predict(tree2, cpus))
# = Plots
g1 = ggplot(data = pred) + geom_point(aes(p1, p2))
g2 = ggplot(data = melt(pred)) + geom_boxplot(aes(variable, value))
grid.arrange(g1, g2, ncol = 2)
</pre>
<p><img alt="plot of chunk unnamed-chunk-29" src="https://i2.wp.com/i.imgur.com/TojzUzR.png?w=456&ssl=1" data-recalc-dims="1"></p>
<h2>Random Forest</h2>
<p>As mentioned before, the Random Forest solves the instability problem using bagging. We simply estimate the desired Regression Tree on many bootstrap samples (re-sample the data many times with replacement and re-estimate the model) and make the final prediction as the average of the predictions across the trees. There is one small (but important) detail to add. The Random Forest adds a new source of instability to the individual trees. Every time we calculate a new optimal variable-observation point to split the tree, we do not use all variables. Instead, we randomly select 2/3 of the variables. This will make the individual trees even more unstable, but as I mentioned <a rel="nofollow" href="https://insightr.wordpress.com/2017/05/22/bagging-the-perfect-solution-for-model-instability/" target="_blank">here</a>, bagging benefits from instability.</p>
<p>The question now is: how much improvement do we get from the Random Forest. The following example is a good illustration. I broke the CPUs data into a training sample (first 150 observations) and a test sample (remaining observations) and estimated a Regression Tree and a Random Forest. The performance is compared using the mean squared error.</p>
<pre>library(randomForest)
# = Regression Tree
tree_fs = tree(log(perf) ~ syct + mmin + mmax + cach + chmin + chmax,
               data = cpus[1:150, ])

# = Random Forest
set.seed(1) # = Seed for replication
rf = randomForest(log(perf) ~ syct + mmin + mmax + cach + chmin + chmax,
                  data=cpus[1:150, ], nodesize = 10, importance = TRUE)

# = Calculate MSE
mse_tree = mean((predict(tree_fs, cpus[-c(1:150), ]) - log(cpus$perf)[-c(1:150)])^2)
mse_rf = mean((predict(rf, cpus[-c(1:150), ]) - log(cpus$perf[-c(1:150)]))^2)

c(rf = mse_rf, tree = mse_tree)
</pre>
<pre>##        rf      tree
## 0.2884766 0.5660053
</pre>
<p>As you can see, the Regression Tree has an error twice as big as the Random Forest. The only problem is that by using a combination of trees any kind of interpretation becomes really hard. Fortunately, there are importance measures that allow us to at least know which variables are more relevant in the Random Forest. In our case, both importance measures pointed to the cache size as the most important variable.</p>
<pre>importance(rf)
</pre>
<pre>##        %IncMSE IncNodePurity
## syct  22.60512     22.373601
## mmin  19.46153     21.965340
## mmax  24.84038     27.239772
## cach  27.92483     33.536185
## chmin 13.77196     13.352793
## chmax 17.61297      8.379306
</pre>
<p>Finally, we can see how the model error decreases as we increase the number of trees in the Random Forest with the following code:</p>
<pre>plot(rf)
</pre>
<p><img alt="plot of chunk unnamed-chunk-32" src="https://i0.wp.com/i.imgur.com/hv6YmMR.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>If you liked this post, you can find more details on Regression Trees and Random forest in the book Elements of Statistical learning, which can be downloaded direct from the authors page <a rel="nofollow" href="https://web.stanford.edu/~hastie/ElemStatLearn/" target="_blank">here</a>.</p>
<p> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/insightr.wordpress.com/693/" target="_blank"><img border="0" alt="" src="http://feeds.wordpress.com/1.0/comments/insightr.wordpress.com/693/"></a> <img border="0" alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" data-recalc-dims="1"></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://insightr.wordpress.com/2017/09/23/how-random-forests-improve-simple-regression-trees/"> R – insightR</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/_YpuUOt1zs8" height="1"></a></div>