<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/does-money-buy-happiness-after-all-machine-learning-with-one-rule/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://shiring.github.io/machine_learning/2017/04/23/one_r"> Shirin's playgRound</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>This week, I am exploring <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/OneR/vignettes/OneR.html">Holger K. von Jouanne-Diedrich’s OneR package</a> for machine learning. I am running an example analysis on world happiness data and compare the results with other machine learning models (decision trees, random forest, gradient boosting trees and neural nets).</p>
<p></p>
<h2>Conclusions</h2>
<p>All in all, based on this example, I would confirm that OneR models do indeed produce sufficiently accurate models for setting a good baseline. OneR was definitely faster than random forest, gradient boosting and neural nets. However, the latter were more complex models and included cross-validation.</p>
<p>If you prefer an easy to understand model that is very simple, OneR is a very good way to go. You could also use it as a starting point for developing more complex models with improved accuracy.</p>
<p>When looking at feature importance across models, the feature OneR chose – Economy/GDP per capita – was confirmed by random forest, gradient boosting trees and neural networks as being the most important feature. This is in itself an interesting conclusion! Of course, this correlation does not tell us that there is a direct causal relationship between money and happiness, but we can say that a country’s economy is the best individual predictor for how happy people tend to be.</p>
<hr>
<p></p>
<h2>OneR</h2>
<p>OneR has been developed for the purpose of creating machine learning models that are easy to interpret and understand, while still being as accurate as possible. It is based on the one rule classification algorithm from Holte (1993), which is basically a decision tree cut at the first level.</p>
<blockquote>
<p><a target="_blank" rel="nofollow" href="http://www.mlpack.org/papers/ds.pdf">R.C. Holte (1993). Very simple classification rules perform well on most commonly used datasets. Machine Learning. 11:63-91.</a></p>
</blockquote>
<p>While the original algorithm has difficulties in handling missing values and numeric data, the package provides enhanced functionality to handle those cases better, e.g. introducing a separate class for NA values and the <em>optbin()</em> function to find optimal splitting points for each feature. The main function of the package is OneR, which finds an optimal split for each feature and only use the most important feature with highest training accuracy for classification.</p>
<p></p>
<p>I installed the latest stable version of the <strong>OneR</strong> package from <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=OneR">CRAN</a>.</p>
<div>
<pre><code><span>library</span><span>(</span><span>OneR</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<h2>The dataset</h2>
<p>I am using the <a target="_blank" rel="nofollow" href="https://www.kaggle.com/unsdsn/world-happiness"><strong>World Happiness Report 2016</strong> from Kaggle</a>.</p>
<div>
<pre><code><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>data_16</span><span> </span><span>&lt;-</span><span> </span><span>read.table</span><span>(</span><span>&quot;world-happiness/2016.csv&quot;</span><span>,</span><span> </span><span>sep</span><span> </span><span>=</span><span> </span><span>&quot;,&quot;</span><span>,</span><span> </span><span>header</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>data_15</span><span> </span><span>&lt;-</span><span> </span><span>read.table</span><span>(</span><span>&quot;world-happiness/2015.csv&quot;</span><span>,</span><span> </span><span>sep</span><span> </span><span>=</span><span> </span><span>&quot;,&quot;</span><span>,</span><span> </span><span>header</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<p>In the 2016 data there are upper and lower CI for the happiness score given, while in the 2015 data we have standard errors. Because I want to combine data from the two years, I am using only columns that are in both datasets.</p>
<div>
<pre><code><span>common_feats</span><span> </span><span>&lt;-</span><span> </span><span>colnames</span><span>(</span><span>data_16</span><span>)[</span><span>which</span><span>(</span><span>colnames</span><span>(</span><span>data_16</span><span>)</span><span> </span><span>%in%</span><span> </span><span>colnames</span><span>(</span><span>data_15</span><span>))]</span><span>
</span><span># features and response variable for modeling
</span><span>feats</span><span> </span><span>&lt;-</span><span> </span><span>setdiff</span><span>(</span><span>common_feats</span><span>,</span><span> </span><span>c</span><span>(</span><span>&quot;Country&quot;</span><span>,</span><span> </span><span>&quot;Happiness.Rank&quot;</span><span>,</span><span> </span><span>&quot;Happiness.Score&quot;</span><span>))</span><span>
</span><span>response</span><span> </span><span>&lt;-</span><span> </span><span>&quot;Happiness.Score&quot;</span><span>
</span><span># combine data from 2015 and 2016
</span><span>data_15_16</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>select</span><span>(</span><span>data_15</span><span>,</span><span> </span><span>one_of</span><span>(</span><span>c</span><span>(</span><span>feats</span><span>,</span><span> </span><span>response</span><span>))),</span><span>
</span><span>select</span><span>(</span><span>data_16</span><span>,</span><span> </span><span>one_of</span><span>(</span><span>c</span><span>(</span><span>feats</span><span>,</span><span> </span><span>response</span><span>))))</span><span>
</span></code></pre>
</div>
<p>The response variable <strong>happiness score</strong> is on a numeric scale. OneR could also perform regression but here, I want to compare classification tasks. For classifying happiness, I create three bins for low, medium and high values of the happiness score. In order to not having to deal with unbalanced data, I am using the <em>bin()</em> function from OneR with <code>method = &quot;content&quot;</code>. For plotting the cut-points, I am extracting the numbers from the default level names.</p>
<div>
<pre><code><span>data_15_16</span><span>$</span><span>Happiness.Score.l</span><span> </span><span>&lt;-</span><span> </span><span>bin</span><span>(</span><span>data_15_16</span><span>$</span><span>Happiness.Score</span><span>,</span><span> </span><span>nbins</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;content&quot;</span><span>)</span><span>
</span><span>intervals</span><span> </span><span>&lt;-</span><span> </span><span>paste</span><span>(</span><span>levels</span><span>(</span><span>data_15_16</span><span>$</span><span>Happiness.Score.l</span><span>),</span><span> </span><span>collapse</span><span> </span><span>=</span><span> </span><span>&quot; &quot;</span><span>)</span><span>
</span><span>intervals</span><span> </span><span>&lt;-</span><span> </span><span>gsub</span><span>(</span><span>&quot;\\(|]&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>intervals</span><span>)</span><span>
</span><span>intervals</span><span> </span><span>&lt;-</span><span> </span><span>gsub</span><span>(</span><span>&quot;,&quot;</span><span>,</span><span> </span><span>&quot; &quot;</span><span>,</span><span> </span><span>intervals</span><span>)</span><span>
</span><span>intervals</span><span> </span><span>&lt;-</span><span> </span><span>as.numeric</span><span>(</span><span>unique</span><span>(</span><span>strsplit</span><span>(</span><span>intervals</span><span>,</span><span> </span><span>&quot; &quot;</span><span>)[[</span><span>1</span><span>]]))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>data_15_16</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>()</span><span> </span><span>+</span><span>
</span><span>geom_density</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>Happiness.Score</span><span>),</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;blue&quot;</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;blue&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.4</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_vline</span><span>(</span><span>xintercept</span><span> </span><span>=</span><span> </span><span>intervals</span><span>[</span><span>2</span><span>])</span><span> </span><span>+</span><span>
</span><span>geom_vline</span><span>(</span><span>xintercept</span><span> </span><span>=</span><span> </span><span>intervals</span><span>[</span><span>3</span><span>])</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/machine_learning/2017/04/23/one_r_files/figure-markdown_github/unnamed-chunk-5-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Now I am removing the original happiness score column from the data for modeling and rename the factor levels of the response variable.</p>
<div>
<pre><code><span>data_15_16</span><span> </span><span>&lt;-</span><span> </span><span>select</span><span>(</span><span>data_15_16</span><span>,</span><span> </span><span>-</span><span>Happiness.Score</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Happiness.Score.l</span><span> </span><span>=</span><span> </span><span>plyr</span><span>::</span><span>revalue</span><span>(</span><span>Happiness.Score.l</span><span>,</span><span> </span><span>c</span><span>(</span><span>&quot;(2.83,4.79]&quot;</span><span> </span><span>=</span><span> </span><span>&quot;low&quot;</span><span>,</span><span> </span><span>&quot;(4.79,5.89]&quot;</span><span> </span><span>=</span><span> </span><span>&quot;medium&quot;</span><span>,</span><span> </span><span>&quot;(5.89,7.59]&quot;</span><span> </span><span>=</span><span> </span><span>&quot;high&quot;</span><span>)))</span><span>
</span></code></pre>
</div>
<p>Because there are only 9 features in this small dataset, I want to explore them all individually before modeling. First, I am plotting the only categorical variable: Region.</p>
<p>This plots shows that there are a few regions with very strong biases in happiness: People in Western Europe, Australia, New Zealand, North America, Latin American and the Caribbean tend to me in the high happiness group, while people in sub-saharan Africa and Southern Asia tend to be the least happiest.</p>
<div>
<pre><code><span>data_15_16</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>Region</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>Happiness.Score.l</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;dodge&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.7</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>axis.text.x</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>angle</span><span> </span><span>=</span><span> </span><span>45</span><span>,</span><span> </span><span>vjust</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>hjust</span><span> </span><span>=</span><span> </span><span>1</span><span>),</span><span>
</span><span>plot.margin</span><span> </span><span>=</span><span> </span><span>unit</span><span>(</span><span>c</span><span>(</span><span>0</span><span>,</span><span> </span><span>0</span><span>,</span><span> </span><span>0</span><span>,</span><span> </span><span>1.5</span><span>),</span><span> </span><span>&quot;cm&quot;</span><span>))</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/machine_learning/2017/04/23/one_r_files/figure-markdown_github/unnamed-chunk-7-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>The remaining quantitative variables show happiness biases to varying degrees: e.g. low health and life expectancy is strongly biased towards low happiness, economic factors, family and freedom show a bias in the same direction, albeit not as strong.</p>
<div>
<pre><code><span>data_15_16</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>x</span><span>,</span><span> </span><span>y</span><span>,</span><span> </span><span>Economy..GDP.per.Capita.</span><span>:</span><span>Dystopia.Residual</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>y</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>Happiness.Score.l</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_histogram</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.7</span><span>)</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>~</span><span> </span><span>x</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>4</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i2.wp.com/shiring.github.io/machine_learning/2017/04/23/one_r_files/figure-markdown_github/unnamed-chunk-8-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>While OneR could also handle categorical data, in this example, I only want to consider the quantitative features to show the differences between OneR and other machine learning algorithms.</p>
<div>
<pre><code><span>data_15_16</span><span> </span><span>&lt;-</span><span> </span><span>select</span><span>(</span><span>data_15_16</span><span>,</span><span> </span><span>-</span><span>Region</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<h2>Modeling</h2>
<p>The algorithms I will compare to OneR will be run via the <strong>caret</strong> package.</p>
<div>
<pre><code><span># configure multicore
</span><span>library</span><span>(</span><span>doParallel</span><span>)</span><span>
</span><span>cl</span><span> </span><span>&lt;-</span><span> </span><span>makeCluster</span><span>(</span><span>detectCores</span><span>())</span><span>
</span><span>registerDoParallel</span><span>(</span><span>cl</span><span>)</span><span>
</span><span>library</span><span>(</span><span>caret</span><span>)</span><span>
</span></code></pre>
</div>
<p>I will also use caret’s <em>createDataPartition()</em> function to partition the data into training (70%) and test sets (30%).</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>42</span><span>)</span><span>
</span><span>index</span><span> </span><span>&lt;-</span><span> </span><span>createDataPartition</span><span>(</span><span>data_15_16</span><span>$</span><span>Happiness.Score.l</span><span>,</span><span> </span><span>p</span><span> </span><span>=</span><span> </span><span>0.7</span><span>,</span><span> </span><span>list</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>train_data</span><span> </span><span>&lt;-</span><span> </span><span>data_15_16</span><span>[</span><span>index</span><span>,</span><span> </span><span>]</span><span>
</span><span>test_data</span><span> </span><span>&lt;-</span><span> </span><span>data_15_16</span><span>[</span><span>-</span><span>index</span><span>,</span><span> </span><span>]</span><span>
</span></code></pre>
</div>
<p></p>
<h3>OneR</h3>
<p>OneR only accepts categorical features. Because we have numerical features, we need to convert them to factors by splitting them into appropriate bins. While the original OneR algorithm splits the values into ever smaller factors, this has been changed in this R-implementation with the argument of preventing overfitting. We can either split the data into pre-defined numbers of buckets (by length, content or cluster) or we can use the <em>optbin()</em> function to obtain the optimal number of factors from pairwise logistic regression or information gain.</p>
<div>
<pre><code><span># default method length
</span><span>data_1</span><span> </span><span>&lt;-</span><span> </span><span>bin</span><span>(</span><span>train_data</span><span>,</span><span> </span><span>nbins</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;length&quot;</span><span>)</span><span>
</span><span># method content
</span><span>data_2</span><span> </span><span>&lt;-</span><span> </span><span>bin</span><span>(</span><span>train_data</span><span>,</span><span> </span><span>nbins</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;content&quot;</span><span>)</span><span>
</span><span># method cluster
</span><span>data_3</span><span> </span><span>&lt;-</span><span> </span><span>bin</span><span>(</span><span>train_data</span><span>,</span><span> </span><span>nbins</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;cluster&quot;</span><span>)</span><span>
</span><span># optimal bin number logistic regression
</span><span>data_4</span><span> </span><span>&lt;-</span><span> </span><span>optbin</span><span>(</span><span>formula</span><span> </span><span>=</span><span> </span><span>Happiness.Score.l</span><span> </span><span>~</span><span>.</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;logreg&quot;</span><span>)</span><span>
</span><span># optimal bin number information gain
</span><span>data_5</span><span> </span><span>&lt;-</span><span> </span><span>optbin</span><span>(</span><span>formula</span><span> </span><span>=</span><span> </span><span>Happiness.Score.l</span><span> </span><span>~</span><span>.</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;infogain&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<p>This is how the data looks like following discretization:</p>
<ul>
<li>Default method</li>
</ul>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/machine_learning/2017/04/23/one_r_files/figure-markdown_github/unnamed-chunk-14-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<ul>
<li>5 bins with <code>method = &quot;content</code></li>
</ul>
<p><img alt="" src="https://i2.wp.com/shiring.github.io/machine_learning/2017/04/23/one_r_files/figure-markdown_github/unnamed-chunk-15-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<ul>
<li>3 bins with <code>method = &quot;cluster</code></li>
</ul>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/machine_learning/2017/04/23/one_r_files/figure-markdown_github/unnamed-chunk-16-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<ul>
<li>optimal bin number according to logistic regression</li>
</ul>
<p><img alt="" src="https://i2.wp.com/shiring.github.io/machine_learning/2017/04/23/one_r_files/figure-markdown_github/unnamed-chunk-17-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<ul>
<li>optimal bin number according to information gain</li>
</ul>
<p><img alt="" src="https://i1.wp.com/shiring.github.io/machine_learning/2017/04/23/one_r_files/figure-markdown_github/unnamed-chunk-18-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h3>Model building</h3>
<p>Now I am running the OneR models. During model building, the chosen attribute/feature with highest accuracy along with the top 7 features decision rules and accuracies are printed. Unfortunately, this information is not saved in the model object; this would have been nice in order to compare the importance of features across models later on.</p>
<p>Here, all five models achieved highest prediction accuracy with the feature Economy GDP per capita.</p>
<div>
<pre><code><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>5</span><span>)</span><span> </span><span>{</span><span>
</span><span>data</span><span> </span><span>&lt;-</span><span> </span><span>get</span><span>(</span><span>paste0</span><span>(</span><span>&quot;data_&quot;</span><span>,</span><span> </span><span>i</span><span>))</span><span>
</span><span>print</span><span>(</span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>OneR</span><span>(</span><span>formula</span><span> </span><span>=</span><span> </span><span>Happiness.Score.l</span><span> </span><span>~</span><span>.</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>data</span><span>,</span><span> </span><span>verbose</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>))</span><span>
</span><span>assign</span><span>(</span><span>paste0</span><span>(</span><span>&quot;model_&quot;</span><span>,</span><span> </span><span>i</span><span>),</span><span> </span><span>model</span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## 
##     Attribute                     Accuracy
## 1 * Economy..GDP.per.Capita.      63.96%  
## 2   Health..Life.Expectancy.      59.91%  
## 3   Family                        57.21%  
## 4   Dystopia.Residual             51.8%   
## 5   Freedom                       49.55%  
## 6   Trust..Government.Corruption. 45.5%   
## 7   Generosity                    41.89%  
## ---
## Chosen attribute due to accuracy
## and ties method (if applicable): '*'
## 
## 
## Call:
## OneR(data = data, formula = Happiness.Score.l ~ ., verbose = TRUE)
## 
## Rules:
## If Economy..GDP.per.Capita. = (-0.00182,0.365] then Happiness.Score.l = low
## If Economy..GDP.per.Capita. = (0.365,0.73]     then Happiness.Score.l = low
## If Economy..GDP.per.Capita. = (0.73,1.09]      then Happiness.Score.l = medium
## If Economy..GDP.per.Capita. = (1.09,1.46]      then Happiness.Score.l = high
## If Economy..GDP.per.Capita. = (1.46,1.83]      then Happiness.Score.l = high
## 
## Accuracy:
## 142 of 222 instances classified correctly (63.96%)
## 
## 
##     Attribute                     Accuracy
## 1 * Economy..GDP.per.Capita.      64.41%  
## 2   Health..Life.Expectancy.      60.81%  
## 3   Family                        59.91%  
## 4   Trust..Government.Corruption. 55.41%  
## 5   Freedom                       53.15%  
## 5   Dystopia.Residual             53.15%  
## 7   Generosity                    41.44%  
## ---
## Chosen attribute due to accuracy
## and ties method (if applicable): '*'
## 
## 
## Call:
## OneR(data = data, formula = Happiness.Score.l ~ ., verbose = TRUE)
## 
## Rules:
## If Economy..GDP.per.Capita. = (-0.00182,0.548] then Happiness.Score.l = low
## If Economy..GDP.per.Capita. = (0.548,0.877]    then Happiness.Score.l = low
## If Economy..GDP.per.Capita. = (0.877,1.06]     then Happiness.Score.l = medium
## If Economy..GDP.per.Capita. = (1.06,1.28]      then Happiness.Score.l = medium
## If Economy..GDP.per.Capita. = (1.28,1.83]      then Happiness.Score.l = high
## 
## Accuracy:
## 143 of 222 instances classified correctly (64.41%)
## 
## 
##     Attribute                     Accuracy
## 1 * Economy..GDP.per.Capita.      63.51%  
## 2   Health..Life.Expectancy.      62.16%  
## 3   Family                        54.5%   
## 4   Freedom                       50.45%  
## 4   Dystopia.Residual             50.45%  
## 6   Trust..Government.Corruption. 43.24%  
## 7   Generosity                    36.49%  
## ---
## Chosen attribute due to accuracy
## and ties method (if applicable): '*'
## 
## 
## Call:
## OneR(data = data, formula = Happiness.Score.l ~ ., verbose = TRUE)
## 
## Rules:
## If Economy..GDP.per.Capita. = (-0.00182,0.602] then Happiness.Score.l = low
## If Economy..GDP.per.Capita. = (0.602,1.1]      then Happiness.Score.l = medium
## If Economy..GDP.per.Capita. = (1.1,1.83]       then Happiness.Score.l = high
## 
## Accuracy:
## 141 of 222 instances classified correctly (63.51%)
## 
## 
##     Attribute                     Accuracy
## 1 * Economy..GDP.per.Capita.      63.96%  
## 2   Health..Life.Expectancy.      62.16%  
## 3   Family                        58.56%  
## 4   Freedom                       51.35%  
## 5   Dystopia.Residual             50.9%   
## 6   Trust..Government.Corruption. 46.4%   
## 7   Generosity                    40.09%  
## ---
## Chosen attribute due to accuracy
## and ties method (if applicable): '*'
## 
## 
## Call:
## OneR(data = data, formula = Happiness.Score.l ~ ., verbose = TRUE)
## 
## Rules:
## If Economy..GDP.per.Capita. = (-0.00182,0.754] then Happiness.Score.l = low
## If Economy..GDP.per.Capita. = (0.754,1.12]     then Happiness.Score.l = medium
## If Economy..GDP.per.Capita. = (1.12,1.83]      then Happiness.Score.l = high
## 
## Accuracy:
## 142 of 222 instances classified correctly (63.96%)
## 
## 
##     Attribute                     Accuracy
## 1 * Economy..GDP.per.Capita.      67.12%  
## 2   Health..Life.Expectancy.      65.77%  
## 3   Family                        61.71%  
## 4   Trust..Government.Corruption. 56.31%  
## 5   Dystopia.Residual             55.41%  
## 6   Freedom                       50.9%   
## 7   Generosity                    43.69%  
## ---
## Chosen attribute due to accuracy
## and ties method (if applicable): '*'
## 
## 
## Call:
## OneR(data = data, formula = Happiness.Score.l ~ ., verbose = TRUE)
## 
## Rules:
## If Economy..GDP.per.Capita. = (-0.00182,0.68] then Happiness.Score.l = low
## If Economy..GDP.per.Capita. = (0.68,1.24]     then Happiness.Score.l = medium
## If Economy..GDP.per.Capita. = (1.24,1.83]     then Happiness.Score.l = high
## 
## Accuracy:
## 149 of 222 instances classified correctly (67.12%)
</code></pre>
</div>
<p></p>
<h3>Model evaluation</h3>
<p>The function <em>eval_model()</em> prints confusion matrices for absolute and relative predictions, as well as accuracy, error and error rate reduction. For comparison with other models, it would have been convenient to be able to extract these performance metrics directly from the eval_model object, instead of only the confusion matrix and values of correct/all instances and having to re-calculate performance metrics again manually.</p>
<div>
<pre><code><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>5</span><span>)</span><span> </span><span>{</span><span>
</span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>get</span><span>(</span><span>paste0</span><span>(</span><span>&quot;model_&quot;</span><span>,</span><span> </span><span>i</span><span>))</span><span>
</span><span>eval_model</span><span>(</span><span>predict</span><span>(</span><span>model</span><span>,</span><span> </span><span>test_data</span><span>),</span><span> </span><span>test_data</span><span>$</span><span>Happiness.Score.l</span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## 
## Confusion matrix (absolute):
##           Actual
## Prediction high low medium Sum
##     high     23   0     11  34
##     low       1  26     10  37
##     medium    7   5     10  22
##     Sum      31  31     31  93
## 
## Confusion matrix (relative):
##           Actual
## Prediction high  low medium  Sum
##     high   0.25 0.00   0.12 0.37
##     low    0.01 0.28   0.11 0.40
##     medium 0.08 0.05   0.11 0.24
##     Sum    0.33 0.33   0.33 1.00
## 
## Accuracy:
## 0.6344 (59/93)
## 
## Error rate:
## 0.3656 (34/93)
## 
## Error rate reduction (vs. base rate):
## 0.4516 (p-value = 2.855e-09)
## 
## 
## Confusion matrix (absolute):
##           Actual
## Prediction high low medium Sum
##     high     19   0      1  20
##     low       3  28     14  45
##     medium    9   3     16  28
##     Sum      31  31     31  93
## 
## Confusion matrix (relative):
##           Actual
## Prediction high  low medium  Sum
##     high   0.20 0.00   0.01 0.22
##     low    0.03 0.30   0.15 0.48
##     medium 0.10 0.03   0.17 0.30
##     Sum    0.33 0.33   0.33 1.00
## 
## Accuracy:
## 0.6774 (63/93)
## 
## Error rate:
## 0.3226 (30/93)
## 
## Error rate reduction (vs. base rate):
## 0.5161 (p-value = 1.303e-11)
## 
## 
## Confusion matrix (absolute):
##           Actual
## Prediction high low medium Sum
##     high     23   0     11  34
##     low       0  25      7  32
##     medium    8   6     13  27
##     Sum      31  31     31  93
## 
## Confusion matrix (relative):
##           Actual
## Prediction high  low medium  Sum
##     high   0.25 0.00   0.12 0.37
##     low    0.00 0.27   0.08 0.34
##     medium 0.09 0.06   0.14 0.29
##     Sum    0.33 0.33   0.33 1.00
## 
## Accuracy:
## 0.6559 (61/93)
## 
## Error rate:
## 0.3441 (32/93)
## 
## Error rate reduction (vs. base rate):
## 0.4839 (p-value = 2.116e-10)
## 
## 
## Confusion matrix (absolute):
##           Actual
## Prediction high low medium Sum
##     high     23   0     11  34
##     low       2  26     11  39
##     medium    6   5      9  20
##     Sum      31  31     31  93
## 
## Confusion matrix (relative):
##           Actual
## Prediction high  low medium  Sum
##     high   0.25 0.00   0.12 0.37
##     low    0.02 0.28   0.12 0.42
##     medium 0.06 0.05   0.10 0.22
##     Sum    0.33 0.33   0.33 1.00
## 
## Accuracy:
## 0.6237 (58/93)
## 
## Error rate:
## 0.3763 (35/93)
## 
## Error rate reduction (vs. base rate):
## 0.4355 (p-value = 9.799e-09)
## 
## 
## Confusion matrix (absolute):
##           Actual
## Prediction high low medium Sum
##     high     21   0      3  24
##     low       0  26      8  34
##     medium   10   5     20  35
##     Sum      31  31     31  93
## 
## Confusion matrix (relative):
##           Actual
## Prediction high  low medium  Sum
##     high   0.23 0.00   0.03 0.26
##     low    0.00 0.28   0.09 0.37
##     medium 0.11 0.05   0.22 0.38
##     Sum    0.33 0.33   0.33 1.00
## 
## Accuracy:
## 0.7204 (67/93)
## 
## Error rate:
## 0.2796 (26/93)
## 
## Error rate reduction (vs. base rate):
## 0.5806 (p-value = 2.761e-14)
</code></pre>
</div>
<p>Because I want to calculate performance measures for the different classes separately and like to have a more detailed look at the prediction probabilities I get from the models, I prefer to obtain predictions with <code>type = &quot;prob</code>. While I am not looking at it here, this would also allow me to test different prediction thresholds.</p>
<div>
<pre><code><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>5</span><span>)</span><span> </span><span>{</span><span>
</span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>get</span><span>(</span><span>paste0</span><span>(</span><span>&quot;model_&quot;</span><span>,</span><span> </span><span>i</span><span>))</span><span>
</span><span>pred</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>model</span><span> </span><span>=</span><span> </span><span>paste0</span><span>(</span><span>&quot;model_&quot;</span><span>,</span><span> </span><span>i</span><span>),</span><span>
</span><span>sample_id</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>nrow</span><span>(</span><span>test_data</span><span>),</span><span>
</span><span>predict</span><span>(</span><span>model</span><span>,</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>actual</span><span> </span><span>=</span><span> </span><span>test_data</span><span>$</span><span>Happiness.Score.l</span><span>)</span><span>
</span><span>pred</span><span>$</span><span>prediction</span><span> </span><span>&lt;-</span><span> </span><span>colnames</span><span>(</span><span>pred</span><span>)[</span><span>3</span><span>:</span><span>5</span><span>][</span><span>apply</span><span>(</span><span>pred</span><span>[,</span><span> </span><span>3</span><span>:</span><span>5</span><span>],</span><span> </span><span>1</span><span>,</span><span> </span><span>which.max</span><span>)]</span><span>
</span><span>pred</span><span>$</span><span>correct</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>pred</span><span>$</span><span>actual</span><span> </span><span>==</span><span> </span><span>pred</span><span>$</span><span>prediction</span><span>,</span><span> </span><span>&quot;correct&quot;</span><span>,</span><span> </span><span>&quot;wrong&quot;</span><span>)</span><span>
</span><span>pred</span><span>$</span><span>pred_prob</span><span> </span><span>&lt;-</span><span> </span><span>NA</span><span>
</span><span>for</span><span> </span><span>(</span><span>j</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>nrow</span><span>(</span><span>pred</span><span>))</span><span> </span><span>{</span><span>
</span><span>pred</span><span>[</span><span>j</span><span>,</span><span> </span><span>&quot;pred_prob&quot;</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>max</span><span>(</span><span>pred</span><span>[</span><span>j</span><span>,</span><span> </span><span>3</span><span>:</span><span>5</span><span>])</span><span>
</span><span>}</span><span>
</span><span>if</span><span> </span><span>(</span><span>i</span><span> </span><span>==</span><span> </span><span>1</span><span>)</span><span> </span><span>{</span><span>
</span><span>pred_df</span><span> </span><span>&lt;-</span><span> </span><span>pred</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>pred_df</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>pred_df</span><span>,</span><span> </span><span>pred</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<p></p>
<h3>Comparing other algorithms</h3>
<h4>Decision trees</h4>
<p>First, I am building a decision tree with the <strong>rpart</strong> package and <em>rpart()</em> function. This, we can plot with <em>rpart.plot()</em>.</p>
<p>Economy GDP per capita is the second highest node here, the best predictor here would be health and life expectancy.</p>
<div>
<pre><code><span>library</span><span>(</span><span>rpart</span><span>)</span><span>
</span><span>library</span><span>(</span><span>rpart.plot</span><span>)</span><span>
</span><span>set.seed</span><span>(</span><span>42</span><span>)</span><span>
</span><span>fit</span><span> </span><span>&lt;-</span><span> </span><span>rpart</span><span>(</span><span>Happiness.Score.l</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;class&quot;</span><span>,</span><span>
</span><span>control</span><span> </span><span>=</span><span> </span><span>rpart.control</span><span>(</span><span>xval</span><span> </span><span>=</span><span> </span><span>10</span><span>),</span><span> </span><span>parms</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>split</span><span> </span><span>=</span><span> </span><span>&quot;information&quot;</span><span>))</span><span>
</span><span>rpart.plot</span><span>(</span><span>fit</span><span>,</span><span> </span><span>extra</span><span> </span><span>=</span><span> </span><span>100</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/machine_learning/2017/04/23/one_r_files/figure-markdown_github/unnamed-chunk-23-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>In order to compare the models, I am producing the same output table for predictions from this model and combine it with the table from the OneR models.</p>
<div>
<pre><code><span>pred</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>model</span><span> </span><span>=</span><span> </span><span>&quot;rpart&quot;</span><span>,</span><span>
</span><span>sample_id</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>nrow</span><span>(</span><span>test_data</span><span>),</span><span>
</span><span>predict</span><span>(</span><span>fit</span><span>,</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>actual</span><span> </span><span>=</span><span> </span><span>test_data</span><span>$</span><span>Happiness.Score.l</span><span>)</span><span>
</span><span>pred</span><span>$</span><span>prediction</span><span> </span><span>&lt;-</span><span> </span><span>colnames</span><span>(</span><span>pred</span><span>)[</span><span>3</span><span>:</span><span>5</span><span>][</span><span>apply</span><span>(</span><span>pred</span><span>[,</span><span> </span><span>3</span><span>:</span><span>5</span><span>],</span><span> </span><span>1</span><span>,</span><span> </span><span>which.max</span><span>)]</span><span>
</span><span>pred</span><span>$</span><span>correct</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>pred</span><span>$</span><span>actual</span><span> </span><span>==</span><span> </span><span>pred</span><span>$</span><span>prediction</span><span>,</span><span> </span><span>&quot;correct&quot;</span><span>,</span><span> </span><span>&quot;wrong&quot;</span><span>)</span><span>
</span><span>pred</span><span>$</span><span>pred_prob</span><span> </span><span>&lt;-</span><span> </span><span>NA</span><span>
</span><span>for</span><span> </span><span>(</span><span>j</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>nrow</span><span>(</span><span>pred</span><span>))</span><span> </span><span>{</span><span>
</span><span>pred</span><span>[</span><span>j</span><span>,</span><span> </span><span>&quot;pred_prob&quot;</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>max</span><span>(</span><span>pred</span><span>[</span><span>j</span><span>,</span><span> </span><span>3</span><span>:</span><span>5</span><span>])</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>pred_df_final</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>pred_df</span><span>,</span><span>
</span><span>pred</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<h3>Random Forest</h3>
<p>Next, I am training a Random Forest model. For more details on Random Forest, check out my <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post">post “Can we predict flu deaths with Machine Learning and R?”</a>.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>42</span><span>)</span><span>
</span><span>model_rf</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>Happiness.Score.l</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;rf&quot;</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span></code></pre>
</div>
<p>The <em>varImp()</em> function from caret shows us which feature was of highest importance for the model and its predictions.</p>
<p>Here, we again find Economy GDP per captia on top.</p>
<div>
<pre><code><span>varImp</span><span>(</span><span>model_rf</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## rf variable importance
## 
##                               Overall
## Economy..GDP.per.Capita.       100.00
## Dystopia.Residual               97.89
## Health..Life.Expectancy.        77.10
## Family                          47.17
## Trust..Government.Corruption.   29.89
## Freedom                         19.29
## Generosity                       0.00
</code></pre>
</div>
<div>
<pre><code><span>pred</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>model</span><span> </span><span>=</span><span> </span><span>&quot;rf&quot;</span><span>,</span><span>
</span><span>sample_id</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>nrow</span><span>(</span><span>test_data</span><span>),</span><span>
</span><span>predict</span><span>(</span><span>model_rf</span><span>,</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>actual</span><span> </span><span>=</span><span> </span><span>test_data</span><span>$</span><span>Happiness.Score.l</span><span>)</span><span>
</span><span>pred</span><span>$</span><span>prediction</span><span> </span><span>&lt;-</span><span> </span><span>colnames</span><span>(</span><span>pred</span><span>)[</span><span>3</span><span>:</span><span>5</span><span>][</span><span>apply</span><span>(</span><span>pred</span><span>[,</span><span> </span><span>3</span><span>:</span><span>5</span><span>],</span><span> </span><span>1</span><span>,</span><span> </span><span>which.max</span><span>)]</span><span>
</span><span>pred</span><span>$</span><span>correct</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>pred</span><span>$</span><span>actual</span><span> </span><span>==</span><span> </span><span>pred</span><span>$</span><span>prediction</span><span>,</span><span> </span><span>&quot;correct&quot;</span><span>,</span><span> </span><span>&quot;wrong&quot;</span><span>)</span><span>
</span><span>pred</span><span>$</span><span>pred_prob</span><span> </span><span>&lt;-</span><span> </span><span>NA</span><span>
</span><span>for</span><span> </span><span>(</span><span>j</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>nrow</span><span>(</span><span>pred</span><span>))</span><span> </span><span>{</span><span>
</span><span>pred</span><span>[</span><span>j</span><span>,</span><span> </span><span>&quot;pred_prob&quot;</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>max</span><span>(</span><span>pred</span><span>[</span><span>j</span><span>,</span><span> </span><span>3</span><span>:</span><span>5</span><span>])</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>pred_df_final</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>pred_df_final</span><span>,</span><span>
</span><span>pred</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<h3>Extreme gradient boosting trees</h3>
<p>Gradient boosting is another decision tree-based algorithm, explained in more detail in <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2016/12/02/flu_outcome_ML_2_post">my post “Extreme Gradient Boosting and Preprocessing in Machine Learning”</a>.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>42</span><span>)</span><span>
</span><span>model_xgb</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>Happiness.Score.l</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;xgbTree&quot;</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span></code></pre>
</div>
<p>As before, we again find Economy GDP per capita as most important feature.</p>
<div>
<pre><code><span>varImp</span><span>(</span><span>model_xgb</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## xgbTree variable importance
## 
##                          Overall
## Economy..GDP.per.Capita.  100.00
## Health..Life.Expectancy.   67.43
## Family                     46.59
## Freedom                     0.00
</code></pre>
</div>
<div>
<pre><code><span>pred</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>model</span><span> </span><span>=</span><span> </span><span>&quot;xgb&quot;</span><span>,</span><span>
</span><span>sample_id</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>nrow</span><span>(</span><span>test_data</span><span>),</span><span>
</span><span>predict</span><span>(</span><span>model_xgb</span><span>,</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>actual</span><span> </span><span>=</span><span> </span><span>test_data</span><span>$</span><span>Happiness.Score.l</span><span>)</span><span>
</span><span>pred</span><span>$</span><span>prediction</span><span> </span><span>&lt;-</span><span> </span><span>colnames</span><span>(</span><span>pred</span><span>)[</span><span>3</span><span>:</span><span>5</span><span>][</span><span>apply</span><span>(</span><span>pred</span><span>[,</span><span> </span><span>3</span><span>:</span><span>5</span><span>],</span><span> </span><span>1</span><span>,</span><span> </span><span>which.max</span><span>)]</span><span>
</span><span>pred</span><span>$</span><span>correct</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>pred</span><span>$</span><span>actual</span><span> </span><span>==</span><span> </span><span>pred</span><span>$</span><span>prediction</span><span>,</span><span> </span><span>&quot;correct&quot;</span><span>,</span><span> </span><span>&quot;wrong&quot;</span><span>)</span><span>
</span><span>pred</span><span>$</span><span>pred_prob</span><span> </span><span>&lt;-</span><span> </span><span>NA</span><span>
</span><span>for</span><span> </span><span>(</span><span>j</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>nrow</span><span>(</span><span>pred</span><span>))</span><span> </span><span>{</span><span>
</span><span>pred</span><span>[</span><span>j</span><span>,</span><span> </span><span>&quot;pred_prob&quot;</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>max</span><span>(</span><span>pred</span><span>[</span><span>j</span><span>,</span><span> </span><span>3</span><span>:</span><span>5</span><span>])</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>pred_df_final</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>pred_df_final</span><span>,</span><span>
</span><span>pred</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<h3>Neural network</h3>
<p>Finally, I am comparing a neural network model. Here as well, I have a post where I talk about what they are in more detail: <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2017/02/27/h2o">“Building deep neural nets with h2o and rsparkling that predict arrhythmia of the heart”</a>.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>42</span><span>)</span><span>
</span><span>model_nn</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>Happiness.Score.l</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;mlp&quot;</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span></code></pre>
</div>
<p>And Economy GDP per capita is again the most important feature!</p>
<div>
<pre><code><span>varImp</span><span>(</span><span>model_nn</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## ROC curve variable importance
## 
##   variables are sorted by maximum importance across the classes
##                                  low medium    high
## Economy..GDP.per.Capita.      100.00 66.548 100.000
## Health..Life.Expectancy.       95.22 63.632  95.222
## Family                         84.48 45.211  84.483
## Dystopia.Residual              71.23 43.450  71.232
## Freedom                        64.58 41.964  64.581
## Trust..Government.Corruption.  26.13 40.573  40.573
## Generosity                      0.00  3.462   3.462
</code></pre>
</div>
<div>
<pre><code><span>pred</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>model</span><span> </span><span>=</span><span> </span><span>&quot;nn&quot;</span><span>,</span><span>
</span><span>sample_id</span><span> </span><span>=</span><span> </span><span>1</span><span>:</span><span>nrow</span><span>(</span><span>test_data</span><span>),</span><span>
</span><span>predict</span><span>(</span><span>model_nn</span><span>,</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>actual</span><span> </span><span>=</span><span> </span><span>test_data</span><span>$</span><span>Happiness.Score.l</span><span>)</span><span>
</span><span>pred</span><span>$</span><span>prediction</span><span> </span><span>&lt;-</span><span> </span><span>colnames</span><span>(</span><span>pred</span><span>)[</span><span>3</span><span>:</span><span>5</span><span>][</span><span>apply</span><span>(</span><span>pred</span><span>[,</span><span> </span><span>3</span><span>:</span><span>5</span><span>],</span><span> </span><span>1</span><span>,</span><span> </span><span>which.max</span><span>)]</span><span>
</span><span>pred</span><span>$</span><span>correct</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>pred</span><span>$</span><span>actual</span><span> </span><span>==</span><span> </span><span>pred</span><span>$</span><span>prediction</span><span>,</span><span> </span><span>&quot;correct&quot;</span><span>,</span><span> </span><span>&quot;wrong&quot;</span><span>)</span><span>
</span><span>pred</span><span>$</span><span>pred_prob</span><span> </span><span>&lt;-</span><span> </span><span>NA</span><span>
</span><span>for</span><span> </span><span>(</span><span>j</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>nrow</span><span>(</span><span>pred</span><span>))</span><span> </span><span>{</span><span>
</span><span>pred</span><span>[</span><span>j</span><span>,</span><span> </span><span>&quot;pred_prob&quot;</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>max</span><span>(</span><span>pred</span><span>[</span><span>j</span><span>,</span><span> </span><span>3</span><span>:</span><span>5</span><span>])</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>pred_df_final</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>pred_df_final</span><span>,</span><span>
</span><span>pred</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<h2>Model comparisons</h2>
<p>Now to the final verdict: How do the different models compare?</p>
<p>The first plot below shows the prediction probabilites for the three happiness levels low, medium and high for each test data instance. For each instance, only the prediction probability of the predicted class (i.e. with the highest value) is shown. The upper row shows correct predictions, the lower row shows wrong predictions.</p>
<p>Sometimes, it is obvious from such a plot if a more stringent prediction threshold could improve things (when wrong predictions tend to be close to the threshold). With three classes to predict, this is obviously not as trivial as if we only had two but the same principle holds true: the smaller the prediction probability, the more uncertain it tends to be.</p>
<div>
<pre><code><span>pred_df_final</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>actual</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>pred_prob</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>prediction</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>prediction</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_boxplot</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.7</span><span>)</span><span> </span><span>+</span><span>
</span><span>facet_grid</span><span>(</span><span>correct</span><span> </span><span>~</span><span> </span><span>model</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_color_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i1.wp.com/shiring.github.io/machine_learning/2017/04/23/one_r_files/figure-markdown_github/unnamed-chunk-44-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Probably the most straight-forwards performance measure is accuracy: i.e. the proportion of correct predictions vs the total number of instances to predict. The closer to 1, the better the accuracy.</p>
<p>Not surprisingly, the more complex models tend to be more accurate – albeit only slightly.</p>
<div>
<pre><code><span>pred_df_final</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>model</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>dplyr</span><span>::</span><span>summarise</span><span>(</span><span>correct</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>correct</span><span> </span><span>==</span><span> </span><span>&quot;correct&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>accuracy</span><span> </span><span>=</span><span> </span><span>correct</span><span> </span><span>/</span><span> </span><span>nrow</span><span>(</span><span>test_data</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>model</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>accuracy</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>model</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>stat</span><span> </span><span>=</span><span> </span><span>&quot;identity&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i1.wp.com/shiring.github.io/machine_learning/2017/04/23/one_r_files/figure-markdown_github/unnamed-chunk-45-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>When we look at the three classes individually, it looks a bit more complicated but most models achieved highest accuracy for class “high”.</p>
<div>
<pre><code><span>pred_df_final</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>model</span><span>,</span><span> </span><span>prediction</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>dplyr</span><span>::</span><span>summarise</span><span>(</span><span>correct</span><span> </span><span>=</span><span> </span><span>sum</span><span>(</span><span>correct</span><span> </span><span>==</span><span> </span><span>&quot;correct&quot;</span><span>),</span><span>
</span><span>n</span><span> </span><span>=</span><span> </span><span>n</span><span>())</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>accuracy</span><span> </span><span>=</span><span> </span><span>correct</span><span> </span><span>/</span><span> </span><span>n</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>model</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>accuracy</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>prediction</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>stat</span><span> </span><span>=</span><span> </span><span>&quot;identity&quot;</span><span>,</span><span> </span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;dodge&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/machine_learning/2017/04/23/one_r_files/figure-markdown_github/unnamed-chunk-47-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<hr>
<p>If you are interested in more machine learning posts, check out <a target="_blank" rel="nofollow" href="https://shiring.github.io/categories.html#machine_learning-ref">the category listing for <strong>machine_learning</strong> on my blog</a>.</p>
<hr>
<div>
<pre><code><span>sessionInfo</span><span>()</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## R version 3.3.3 (2017-03-06)
## Platform: x86_64-apple-darwin13.4.0 (64-bit)
## Running under: macOS Sierra 10.12.3
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] parallel  stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] RSNNS_0.4-9         Rcpp_0.12.10        plyr_1.8.4         
##  [4] xgboost_0.6-4       randomForest_4.6-12 rpart.plot_2.1.1   
##  [7] rpart_4.1-10        caret_6.0-73        lattice_0.20-35    
## [10] doParallel_1.0.10   iterators_1.0.8     foreach_1.4.3      
## [13] dplyr_0.5.0         purrr_0.2.2         readr_1.1.0        
## [16] tidyr_0.6.1         tibble_1.3.0        ggplot2_2.2.1      
## [19] tidyverse_1.1.1     OneR_2.1           
## 
## loaded via a namespace (and not attached):
##  [1] lubridate_1.6.0    assertthat_0.2.0   rprojroot_1.2     
##  [4] digest_0.6.12      psych_1.7.3.21     R6_2.2.0          
##  [7] backports_1.0.5    MatrixModels_0.4-1 stats4_3.3.3      
## [10] evaluate_0.10      httr_1.2.1         lazyeval_0.2.0    
## [13] readxl_0.1.1       data.table_1.10.4  minqa_1.2.4       
## [16] SparseM_1.76       car_2.1-4          nloptr_1.0.4      
## [19] Matrix_1.2-8       rmarkdown_1.4      labeling_0.3      
## [22] splines_3.3.3      lme4_1.1-12        stringr_1.2.0     
## [25] foreign_0.8-67     munsell_0.4.3      broom_0.4.2       
## [28] modelr_0.1.0       mnormt_1.5-5       mgcv_1.8-17       
## [31] htmltools_0.3.5    nnet_7.3-12        codetools_0.2-15  
## [34] MASS_7.3-45        ModelMetrics_1.1.0 grid_3.3.3        
## [37] nlme_3.1-131       jsonlite_1.4       gtable_0.2.0      
## [40] DBI_0.6-1          magrittr_1.5       scales_0.4.1      
## [43] stringi_1.1.5      reshape2_1.4.2     xml2_1.1.1        
## [46] RColorBrewer_1.1-2 tools_3.3.3        forcats_0.2.0     
## [49] hms_0.3            pbkrtest_0.4-7     yaml_2.1.14       
## [52] colorspace_1.3-2   rvest_0.3.2        knitr_1.15.1      
## [55] haven_1.0.0        quantreg_5.29
</code></pre>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://shiring.github.io/machine_learning/2017/04/23/one_r"> Shirin's playgRound</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/n3Jea27xPZ4"></a></div>