<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/machine-learning-basics-random-forest/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://shirinsplayground.netlify.com/2018/10/ml_basics_rf/"> Shirin's playgRound</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>A few colleagues of mine and I from <a rel="nofollow" href="https://www.codecentric.de/kuenstliche-intelligenz/" target="_blank">codecentric.ai</a> are currently working on developing a free online course about machine learning and deep learning. As part of this course, I am developing a <a rel="nofollow" href="https://www.youtube.com/codecentricAI" target="_blank">series of videos</a> about machine learning basics – the first video in this series was about Random Forests.</p>
<p>You can find the video on <a rel="nofollow" href="https://youtu.be/ieF_QjVUNEQ" target="_blank">YouTube</a> but as of now, it is only available in German. Same goes for the <a rel="nofollow" href="https://codecentric.slides.com/shiringlander/ml_basics_rf" target="_blank">slides</a>, which are also currently German only.</p>
<p>I did however translate my script:</p>
<hr>
<div>
<img src="https://i2.wp.com/shiring.github.io/netlify_images/ml_basics_rf.png?w=456&ssl=1" data-recalc-dims="1">
</div>
<p>Random Forest (RF) is one of the many machine learning algorithms used for supervised learning, this means for learning from labelled data and making predictions based on the learned patterns. RF can be used for both classification and regression tasks.</p>
<div>
<h2>Decision trees</h2>
<p>RF is based on decision trees. In machine learning decision trees are a technique for creating predictive models. They are called decision <strong>trees</strong> because the prediction follows several branches of “if… then…” decision splits – similar to the branches of a tree. If we imagine that we start with a sample, which we want to predict a class for, we would start at the bottom of a tree and travel up the trunk until we come to the first split-off branch. This split can be thought of as a feature in machine learning, let’s say it would be “age”; we would now make a decision about which branch to follow: “if our sample has an age bigger than 30, continue along the left branch, else continue along the right branch”. This we would do until we come to the next branch and repeat the same decision process until there are no more branches before us. This endpoint is called a leaf and in decision trees would represent the final result: a predicted class or value.</p>
<p>At each branch, the feature thresholds that best split the (remaining) samples locally is found. The most common metrics for defining the “best split” are <strong>gini impurity</strong> and <strong>information gain</strong> for classification tasks and <strong>variance reduction</strong> for regression.</p>
<p>Single decision trees are very easy to visualize and understand because they follow a method of decision-making that is very similar to how we humans make decisions: with a chain of simple rules. However, they are not very robust, i.e. they don’t generalize well to unseen samples. Here is where Random Forests come into play.</p>
</div>
<div>
<h2>Ensemble learning</h2>
<p>RF makes predictions by combining the results from many individual decision trees – so we cal them a <strong>forest</strong> of decision trees. Because RF combines multiple models, it falls under the category of ensemble learning. Other ensemble learning methods are gradient boosting and stacked ensembles.</p>
</div>
<div>
<h2>Combining decision trees</h2>
<p>There are two main ways for combining the outputs of multiple decision trees into a random forest:</p>
<ol>
<li>Bagging, which is also called Bootstrap aggregation (used in Random Forests)</li>
<li>Boosting (used in Gradient Boosting Machines)</li>
</ol>
<p>Bagging works the following way: decision trees are trained on randomly sampled subsets of the data, while sampling is being done with replacement. Bagging is the default method used with Random Forests. A big advantage of bagging over individual trees is that it decrease the variance of the model. Individual trees are very prone to overfitting and are very sensitive to noise in the data. As long as our individual trees are not correlated, combining them with bagging will make them more robust without increasing the bias. The part about correlation is important, though! We remove (most of) the correlation by randomly sampling subsets of data and training the different decision trees on this subsets instead of on the entire dataset. In addition to randomly sampling instances from our data, RF also uses <strong>feature bagging</strong>. With feature bagging, at each split in the decision tree only a random subset of features is considered. This technique reduces correlation even more because it helps reduce the impact of very strong predictor variables (i.e. features that have a very strong influence on predicting the target or response variable).</p>
<p>Boosting works similarly but with one major difference: the samples are weighted for sampling so that samples, which were predicted incorrectly get a higher weight and are therefore sampled more often. The idea behind this is that difficult cases should be emphasized during learning compared to easy cases. Because of this difference bagging can be easily paralleled, while boosting is performed sequentially.</p>
<p>The final result of our model is calculated by averaging over all predictions from these sampled trees or by majority vote.</p>
<div>
<img alt="https://commons.wikimedia.org/wiki/File:Random_forest_diagram_complete.png" src="https://i0.wp.com/upload.wikimedia.org/wikipedia/commons/7/76/Random_forest_diagram_complete.png?w=456&ssl=1" data-recalc-dims="1">
<p><a rel="nofollow" href="https://commons.wikimedia.org/wiki/File:Random_forest_diagram_complete.png" target="_blank">https://commons.wikimedia.org/wiki/File:Random_forest_diagram_complete.png</a></p>
</div>
</div>
<div>
<h2>Hyperparameters to be tuned</h2>
<p>Hyperparameters are the arguments that can be set before training and which define how the training is done. The main hyperparameters in Random Forests are</p>
<ul>
<li>The number of decision trees to be combined</li>
<li>The maximum depth of the trees</li>
<li>The maximum number of features considered at each split</li>
<li>Whether bagging/bootstrapping is performed with or without replacement</li>
</ul>
</div>
<div>
<h2>Training Random Forest models</h2>
<p>Random Forest implementations are available in many machine learning libraries for R and Python, like <code>caret</code> (R, imports the <code>randomForest</code> and other RF packages), Scikit-learn (Python) and H2O (R and Python).</p>
<p>Examples in R can be found here: <a rel="nofollow" href="https://shirinsplayground.netlify.com/2018/06/intro_to_ml_workshop_heidelberg/" target="_blank">https://shirinsplayground.netlify.com/2018/06/intro_to_ml_workshop_heidelberg/</a>.</p>
</div>
<div>
<h2>Other tree-based machine learning algorithms</h2>
<p>The pros of Random Forests are that they are a relatively fast and powerful algorithm for classification and regression learning. Calculations can be parallelized and perform well on many problems, even with small datasets and the output returns prediction probabilities.</p>
<p>Downsides of Random Forests are that they are black-boxes, meaning that we can’t interpret the decisions made by the model because they are too complex. RF are also somewhat prone to overfitting and they tend to be bad at predicting underrepresented classes in unbalanced datasets.</p>
<p>Other tree-based algorithms are (Extreme) Gradient Boosting and Rotation Forests.</p>
<hr>
</div>
<div>
<h1>Video</h1>
<p><iframe width="450" allowfullscreen src="https://www.youtube.com/embed/ieF_QjVUNEQ"><br>
</iframe>
</div>
<div>
<h1>Slides</h1>
<p><iframe allowfullscreen src="https://codecentric.slides.com/shiringlander/ml_basics_rf/embed" width="450"><br>
</iframe>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://shirinsplayground.netlify.com/2018/10/ml_basics_rf/"> Shirin's playgRound</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/Y9i63FIXuyY" height="1"></a></div>