<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/building-meaningful-machine-learning-models-for-disease-prediction/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://shiring.github.io/machine_learning/2017/03/31/webinar_code"> Shirin's playgRound</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h2>Webinar for the <a target="_blank" rel="nofollow" href="http://www.syndromic.org/cop/r">ISDS R Group</a></h2>
<p>This document presents the code I used to produce the example analysis and figures shown in my webinar on building meaningful machine learning models for disease prediction.</p>
<p><a target="_blank" rel="nofollow" href="https://github.com/ShirinG/Webinar_ISDS/blob/master/Webinar_slides.pdf">My webinar slides are available on Github</a></p>
<p></p>
<blockquote>
<p><strong>Description:</strong> Dr Shirin Glander will go over her work on building machine-learning models to predict the course of different diseases. She will go over building a model, evaluating its performance, and answering or addressing different disease related questions using machine learning. Her talk will cover the theory of machine learning as it is applied using R.</p>
</blockquote>
<hr>
<h3>Setup</h3>
<p>All analyses are done in R using RStudio. For detailed session information including R version, operating system and package versions, see the <code>sessionInfo()</code> output at the end of this document.</p>
<p>All figures are produced with ggplot2.</p>
<p></p>
<h3>The dataset</h3>
<p>The dataset I am using in these example analyses, is the <strong>Breast Cancer Wisconsin (Diagnostic) Dataset</strong>. The data was downloaded from the <a target="_blank" rel="nofollow" href="http://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29">UC Irvine Machine Learning Repository</a>.</p>
<p>The first dataset looks at the predictor classes:</p>
<ul>
<li>malignant or</li>
<li>benign breast mass.</li>
</ul>
<p>The features characterize cell nucleus properties and were generated from image analysis of <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Fine-needle_aspiration">fine needle aspirates (FNA)</a> of breast masses:</p>
<ul>
<li>Sample ID (code number)</li>
<li>Clump thickness</li>
<li>Uniformity of cell size</li>
<li>Uniformity of cell shape</li>
<li>Marginal adhesion</li>
<li>Single epithelial cell size</li>
<li>Number of bare nuclei</li>
<li>Bland chromatin</li>
<li>Number of normal nuclei</li>
<li>Mitosis</li>
<li>Classes, i.e. diagnosis</li>
</ul>
<div>
<pre><code><span>bc_data</span><span> </span><span>&lt;-</span><span> </span><span>read.table</span><span>(</span><span>&quot;datasets/breast-cancer-wisconsin.data.txt&quot;</span><span>,</span><span> </span><span>header</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>sep</span><span> </span><span>=</span><span> </span><span>&quot;,&quot;</span><span>)</span><span>
</span><span>colnames</span><span>(</span><span>bc_data</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;sample_code_number&quot;</span><span>,</span><span> </span><span>&quot;clump_thickness&quot;</span><span>,</span><span> </span><span>&quot;uniformity_of_cell_size&quot;</span><span>,</span><span> </span><span>&quot;uniformity_of_cell_shape&quot;</span><span>,</span><span> </span><span>&quot;marginal_adhesion&quot;</span><span>,</span><span> </span><span>&quot;single_epithelial_cell_size&quot;</span><span>,</span><span> </span><span>&quot;bare_nuclei&quot;</span><span>,</span><span> </span><span>&quot;bland_chromatin&quot;</span><span>,</span><span> </span><span>&quot;normal_nucleoli&quot;</span><span>,</span><span> </span><span>&quot;mitosis&quot;</span><span>,</span><span> </span><span>&quot;classes&quot;</span><span>)</span><span>
</span><span>bc_data</span><span>$</span><span>classes</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>bc_data</span><span>$</span><span>classes</span><span> </span><span>==</span><span> </span><span>&quot;2&quot;</span><span>,</span><span> </span><span>&quot;benign&quot;</span><span>,</span><span>
</span><span>ifelse</span><span>(</span><span>bc_data</span><span>$</span><span>classes</span><span> </span><span>==</span><span> </span><span>&quot;4&quot;</span><span>,</span><span> </span><span>&quot;malignant&quot;</span><span>,</span><span> </span><span>NA</span><span>))</span><span>
</span></code></pre>
</div>
<p></p>
<h4>Missing data</h4>
<div>
<pre><code><span>bc_data</span><span>[</span><span>bc_data</span><span> </span><span>==</span><span> </span><span>&quot;?&quot;</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>NA</span><span>
</span><span># how many NAs are in the data
</span><span>length</span><span>(</span><span>which</span><span>(</span><span>is.na</span><span>(</span><span>bc_data</span><span>)))</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 16
</code></pre>
</div>
<div>
<pre><code><span># how many samples would we loose, if we removed them?
</span><span>nrow</span><span>(</span><span>bc_data</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 699
</code></pre>
</div>
<div>
<pre><code><span>nrow</span><span>(</span><span>bc_data</span><span>[</span><span>is.na</span><span>(</span><span>bc_data</span><span>),</span><span> </span><span>])</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 16
</code></pre>
</div>
<p></p>
<p>Missing values are imputed with the <em>mice</em> package.</p>
<div>
<pre><code><span># impute missing data
</span><span>library</span><span>(</span><span>mice</span><span>)</span><span>
</span><span>bc_data</span><span>[,</span><span>2</span><span>:</span><span>10</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>apply</span><span>(</span><span>bc_data</span><span>[,</span><span> </span><span>2</span><span>:</span><span>10</span><span>],</span><span> </span><span>2</span><span>,</span><span> </span><span>function</span><span>(</span><span>x</span><span>)</span><span> </span><span>as.numeric</span><span>(</span><span>as.character</span><span>(</span><span>x</span><span>)))</span><span>
</span><span>dataset_impute</span><span> </span><span>&lt;-</span><span> </span><span>mice</span><span>(</span><span>bc_data</span><span>[,</span><span> </span><span>2</span><span>:</span><span>10</span><span>],</span><span> </span><span>print</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>bc_data</span><span> </span><span>&lt;-</span><span> </span><span>cbind</span><span>(</span><span>bc_data</span><span>[,</span><span> </span><span>11</span><span>,</span><span> </span><span>drop</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>],</span><span> </span><span>mice</span><span>::</span><span>complete</span><span>(</span><span>dataset_impute</span><span>,</span><span> </span><span>1</span><span>))</span><span>
</span><span>bc_data</span><span>$</span><span>classes</span><span> </span><span>&lt;-</span><span> </span><span>as.factor</span><span>(</span><span>bc_data</span><span>$</span><span>classes</span><span>)</span><span>
</span><span># how many benign and malignant cases are there?
</span><span>summary</span><span>(</span><span>bc_data</span><span>$</span><span>classes</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<h4>Data exploration</h4>
<ul>
<li>Response variable for classification</li>
</ul>
<div>
<pre><code><span>library</span><span>(</span><span>ggplot2</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>bc_data</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>classes</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>classes</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>()</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/response_classification-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<ul>
<li>Response variable for regression</li>
</ul>
<div>
<pre><code><span>ggplot</span><span>(</span><span>bc_data</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>clump_thickness</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_histogram</span><span>(</span><span>bins</span><span> </span><span>=</span><span> </span><span>10</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/response_regression-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<ul>
<li>Principal Component Analysis</li>
</ul>
<div>
<pre><code><span>library</span><span>(</span><span>pcaGoPromoter</span><span>)</span><span>
</span><span>library</span><span>(</span><span>ellipse</span><span>)</span><span>
</span><span># perform pca and extract scores
</span><span>pcaOutput</span><span> </span><span>&lt;-</span><span> </span><span>pca</span><span>(</span><span>t</span><span>(</span><span>bc_data</span><span>[,</span><span> </span><span>-1</span><span>]),</span><span> </span><span>printDropped</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>scale</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>center</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>pcaOutput2</span><span> </span><span>&lt;-</span><span> </span><span>as.data.frame</span><span>(</span><span>pcaOutput</span><span>$</span><span>scores</span><span>)</span><span>
</span><span># define groups for plotting
</span><span>pcaOutput2</span><span>$</span><span>groups</span><span> </span><span>&lt;-</span><span> </span><span>bc_data</span><span>$</span><span>classes</span><span>
</span><span>centroids</span><span> </span><span>&lt;-</span><span> </span><span>aggregate</span><span>(</span><span>cbind</span><span>(</span><span>PC1</span><span>,</span><span> </span><span>PC2</span><span>)</span><span> </span><span>~</span><span> </span><span>groups</span><span>,</span><span> </span><span>pcaOutput2</span><span>,</span><span> </span><span>mean</span><span>)</span><span>
</span><span>conf.rgn</span><span> </span><span>&lt;-</span><span> </span><span>do.call</span><span>(</span><span>rbind</span><span>,</span><span> </span><span>lapply</span><span>(</span><span>unique</span><span>(</span><span>pcaOutput2</span><span>$</span><span>groups</span><span>),</span><span> </span><span>function</span><span>(</span><span>t</span><span>)</span><span>
</span><span>data.frame</span><span>(</span><span>groups</span><span> </span><span>=</span><span> </span><span>as.character</span><span>(</span><span>t</span><span>),</span><span>
</span><span>ellipse</span><span>(</span><span>cov</span><span>(</span><span>pcaOutput2</span><span>[</span><span>pcaOutput2</span><span>$</span><span>groups</span><span> </span><span>==</span><span> </span><span>t</span><span>,</span><span> </span><span>1</span><span>:</span><span>2</span><span>]),</span><span>
</span><span>centre</span><span> </span><span>=</span><span> </span><span>as.matrix</span><span>(</span><span>centroids</span><span>[</span><span>centroids</span><span>$</span><span>groups</span><span> </span><span>==</span><span> </span><span>t</span><span>,</span><span> </span><span>2</span><span>:</span><span>3</span><span>]),</span><span>
</span><span>level</span><span> </span><span>=</span><span> </span><span>0.95</span><span>),</span><span>
</span><span>stringsAsFactors</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)))</span><span>
</span><span>ggplot</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>pcaOutput2</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>PC1</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>PC2</span><span>,</span><span> </span><span>group</span><span> </span><span>=</span><span> </span><span>groups</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>groups</span><span>))</span><span> </span><span>+</span><span> </span><span>geom_polygon</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>conf.rgn</span><span>,</span><span> </span><span>aes</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>groups</span><span>),</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.2</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>2</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.6</span><span>)</span><span> </span><span>+</span><span> </span><span>scale_color_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>paste0</span><span>(</span><span>&quot;PC1: &quot;</span><span>,</span><span> </span><span>round</span><span>(</span><span>pcaOutput</span><span>$</span><span>pov</span><span>[</span><span>1</span><span>],</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span> </span><span>*</span><span> </span><span>100</span><span>,</span><span> </span><span>&quot;% variance&quot;</span><span>),</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>paste0</span><span>(</span><span>&quot;PC2: &quot;</span><span>,</span><span> </span><span>round</span><span>(</span><span>pcaOutput</span><span>$</span><span>pov</span><span>[</span><span>2</span><span>],</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span> </span><span>*</span><span> </span><span>100</span><span>,</span><span> </span><span>&quot;% variance&quot;</span><span>))</span><span> </span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/pca-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<ul>
<li>Features</li>
</ul>
<div>
<pre><code><span>library</span><span>(</span><span>tidyr</span><span>)</span><span>
</span><span>gather</span><span>(</span><span>bc_data</span><span>,</span><span> </span><span>x</span><span>,</span><span> </span><span>y</span><span>,</span><span> </span><span>clump_thickness</span><span>:</span><span>mitosis</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>y</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>classes</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>classes</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_density</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.3</span><span>)</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span> </span><span>~</span><span> </span><span>x</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>3</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/features-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h2>Machine Learning packages for R</h2>
<h3><a target="_blank" rel="nofollow" href="http://topepo.github.io/caret/index.html">caret</a></h3>
<div>
<pre><code><span># configure multicore
</span><span>library</span><span>(</span><span>doParallel</span><span>)</span><span>
</span><span>cl</span><span> </span><span>&lt;-</span><span> </span><span>makeCluster</span><span>(</span><span>detectCores</span><span>())</span><span>
</span><span>registerDoParallel</span><span>(</span><span>cl</span><span>)</span><span>
</span><span>library</span><span>(</span><span>caret</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<h4>Training, validation and test data</h4>
<div>
<pre><code><span>set.seed</span><span>(</span><span>42</span><span>)</span><span>
</span><span>index</span><span> </span><span>&lt;-</span><span> </span><span>createDataPartition</span><span>(</span><span>bc_data</span><span>$</span><span>classes</span><span>,</span><span> </span><span>p</span><span> </span><span>=</span><span> </span><span>0.7</span><span>,</span><span> </span><span>list</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>train_data</span><span> </span><span>&lt;-</span><span> </span><span>bc_data</span><span>[</span><span>index</span><span>,</span><span> </span><span>]</span><span>
</span><span>test_data</span><span> </span><span>&lt;-</span><span> </span><span>bc_data</span><span>[</span><span>-</span><span>index</span><span>,</span><span> </span><span>]</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>rbind</span><span>(</span><span>data.frame</span><span>(</span><span>group</span><span> </span><span>=</span><span> </span><span>&quot;train&quot;</span><span>,</span><span> </span><span>train_data</span><span>),</span><span>
</span><span>data.frame</span><span>(</span><span>group</span><span> </span><span>=</span><span> </span><span>&quot;test&quot;</span><span>,</span><span> </span><span>test_data</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>x</span><span>,</span><span> </span><span>y</span><span>,</span><span> </span><span>clump_thickness</span><span>:</span><span>mitosis</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>y</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>group</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>group</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_density</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.3</span><span>)</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span> </span><span>~</span><span> </span><span>x</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>3</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/distribution-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h4>Regression</h4>
<div>
<pre><code><span>set.seed</span><span>(</span><span>42</span><span>)</span><span>
</span><span>model_glm</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>clump_thickness</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;glm&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;scale&quot;</span><span>,</span><span> </span><span>&quot;center&quot;</span><span>),</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>savePredictions</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>model_glm</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Generalized Linear Model 
## 
## 490 samples
##   9 predictor
## 
## Pre-processing: scaled (9), centered (9) 
## Resampling: Cross-Validated (10 fold, repeated 10 times) 
## Summary of sample sizes: 441, 441, 440, 442, 441, 440, ... 
## Resampling results:
## 
##   RMSE      Rsquared 
##   1.974296  0.5016141
## 
## 
</code></pre>
</div>
<div>
<pre><code><span>predictions</span><span> </span><span>&lt;-</span><span> </span><span>predict</span><span>(</span><span>model_glm</span><span>,</span><span> </span><span>test_data</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span># model_glm$finalModel$linear.predictors == model_glm$finalModel$fitted.values
</span><span>data.frame</span><span>(</span><span>residuals</span><span> </span><span>=</span><span> </span><span>resid</span><span>(</span><span>model_glm</span><span>),</span><span>
</span><span>predictors</span><span> </span><span>=</span><span> </span><span>model_glm</span><span>$</span><span>finalModel</span><span>$</span><span>linear.predictors</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>predictors</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>residuals</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_jitter</span><span>()</span><span> </span><span>+</span><span>
</span><span>geom_smooth</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;lm&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/residuals-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span># y == train_data$clump_thickness
</span><span>data.frame</span><span>(</span><span>residuals</span><span> </span><span>=</span><span> </span><span>resid</span><span>(</span><span>model_glm</span><span>),</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>model_glm</span><span>$</span><span>finalModel</span><span>$</span><span>y</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>y</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>residuals</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_jitter</span><span>()</span><span> </span><span>+</span><span>
</span><span>geom_smooth</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;lm&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/residuals-2.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>data.frame</span><span>(</span><span>actual</span><span> </span><span>=</span><span> </span><span>test_data</span><span>$</span><span>clump_thickness</span><span>,</span><span>
</span><span>predicted</span><span> </span><span>=</span><span> </span><span>predictions</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>actual</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>predicted</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_jitter</span><span>()</span><span> </span><span>+</span><span>
</span><span>geom_smooth</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;lm&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/regression_result-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h4>Classification</h4>
<h5>Decision trees</h5>
<p><a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/rpart/rpart.pdf">rpart</a></p>
<div>
<pre><code><span>library</span><span>(</span><span>rpart</span><span>)</span><span>
</span><span>library</span><span>(</span><span>rpart.plot</span><span>)</span><span>
</span><span>set.seed</span><span>(</span><span>42</span><span>)</span><span>
</span><span>fit</span><span> </span><span>&lt;-</span><span> </span><span>rpart</span><span>(</span><span>classes</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;class&quot;</span><span>,</span><span>
</span><span>control</span><span> </span><span>=</span><span> </span><span>rpart.control</span><span>(</span><span>xval</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>minbucket</span><span> </span><span>=</span><span> </span><span>2</span><span>,</span><span> </span><span>cp</span><span> </span><span>=</span><span> </span><span>0</span><span>),</span><span> </span><span>parms</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>split</span><span> </span><span>=</span><span> </span><span>&quot;information&quot;</span><span>))</span><span>
</span><span>rpart.plot</span><span>(</span><span>fit</span><span>,</span><span> </span><span>extra</span><span> </span><span>=</span><span> </span><span>100</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/decision_tree-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h4>Random Forests</h4>
<p><a target="_blank" rel="nofollow" href="https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm">Random Forests</a> predictions are based on the generation of multiple classification trees. They can be used for both, classification and regression tasks. Here, I show a classification task.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>42</span><span>)</span><span>
</span><span>model_rf</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>classes</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;rf&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;scale&quot;</span><span>,</span><span> </span><span>&quot;center&quot;</span><span>),</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>savePredictions</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span></code></pre>
</div>
<p>When you specify <code>savePredictions = TRUE</code>, you can access the cross-validation resuls with <code>model_rf$pred</code>.</p>
<div>
<pre><code><span>model_rf</span><span>$</span><span>finalModel</span><span>$</span><span>confusion</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##           benign malignant class.error
## benign       313         8  0.02492212
## malignant      4       165  0.02366864
</code></pre>
</div>
<p></p>
<ul>
<li>Feature Importance</li>
</ul>
<div>
<pre><code><span>imp</span><span> </span><span>&lt;-</span><span> </span><span>model_rf</span><span>$</span><span>finalModel</span><span>$</span><span>importance</span><span>
</span><span>imp</span><span>[</span><span>order</span><span>(</span><span>imp</span><span>,</span><span> </span><span>decreasing</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>),</span><span> </span><span>]</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##     uniformity_of_cell_size    uniformity_of_cell_shape 
##                   54.416003                   41.553022 
##             bland_chromatin                 bare_nuclei 
##                   29.343027                   28.483842 
##             normal_nucleoli single_epithelial_cell_size 
##                   19.239635                   18.480155 
##             clump_thickness           marginal_adhesion 
##                   13.276702                   12.143355 
##                     mitosis 
##                    3.081635
</code></pre>
</div>
<div>
<pre><code><span># estimate variable importance
</span><span>importance</span><span> </span><span>&lt;-</span><span> </span><span>varImp</span><span>(</span><span>model_rf</span><span>,</span><span> </span><span>scale</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>plot</span><span>(</span><span>importance</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/importance_rf-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<ul>
<li>predicting test data</li>
</ul>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_rf</span><span>,</span><span> </span><span>test_data</span><span>),</span><span> </span><span>test_data</span><span>$</span><span>classes</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##            Reference
## Prediction  benign malignant
##   benign       133         2
##   malignant      4        70
##                                           
##                Accuracy : 0.9713          
##                  95% CI : (0.9386, 0.9894)
##     No Information Rate : 0.6555          
##     P-Value [Acc > NIR] : &lt;2e-16          
##                                           
##                   Kappa : 0.9369          
##  Mcnemar's Test P-Value : 0.6831          
##                                           
##             Sensitivity : 0.9708          
##             Specificity : 0.9722          
##          Pos Pred Value : 0.9852          
##          Neg Pred Value : 0.9459          
##              Prevalence : 0.6555          
##          Detection Rate : 0.6364          
##    Detection Prevalence : 0.6459          
##       Balanced Accuracy : 0.9715          
##                                           
##        'Positive' Class : benign          
## 
</code></pre>
</div>
<div>
<pre><code><span>results</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>actual</span><span> </span><span>=</span><span> </span><span>test_data</span><span>$</span><span>classes</span><span>,</span><span>
</span><span>predict</span><span>(</span><span>model_rf</span><span>,</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;prob&quot;</span><span>))</span><span>
</span><span>results</span><span>$</span><span>prediction</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>results</span><span>$</span><span>benign</span><span> </span><span>></span><span> </span><span>0.5</span><span>,</span><span> </span><span>&quot;benign&quot;</span><span>,</span><span>
</span><span>ifelse</span><span>(</span><span>results</span><span>$</span><span>malignant</span><span> </span><span>></span><span> </span><span>0.5</span><span>,</span><span> </span><span>&quot;malignant&quot;</span><span>,</span><span> </span><span>NA</span><span>))</span><span>
</span><span>results</span><span>$</span><span>correct</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>results</span><span>$</span><span>actual</span><span> </span><span>==</span><span> </span><span>results</span><span>$</span><span>prediction</span><span>,</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>results</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>prediction</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>correct</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;dodge&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i0.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/results_bar_rf-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>ggplot</span><span>(</span><span>results</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>prediction</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>benign</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>correct</span><span>,</span><span> </span><span>shape</span><span> </span><span>=</span><span> </span><span>correct</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_jitter</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.6</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/results_jitter_rf-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h4>Extreme gradient boosting trees</h4>
<p><a target="_blank" rel="nofollow" href="http://xgboost.readthedocs.io/en/latest/model.html">Extreme gradient boosting (XGBoost)</a> is a faster and improved implementation of <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Gradient_boosting">gradient boosting</a> for supervised learning.</p>
<blockquote>
<p>“XGBoost uses a more regularized model formalization to control over-fitting, which gives it better performance.” Tianqi Chen, developer of xgboost</p>
</blockquote>
<p>XGBoost is a tree ensemble model, which means the sum of predictions from a set of classification and regression trees (CART). In that, XGBoost is similar to Random Forests but it uses a different approach to model training. Can be used for classification and regression tasks. Here, I show a classification task.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>42</span><span>)</span><span>
</span><span>model_xgb</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>classes</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;xgbTree&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;scale&quot;</span><span>,</span><span> </span><span>&quot;center&quot;</span><span>),</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>savePredictions</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span></code></pre>
</div>
<p></p>
<ul>
<li>Feature Importance</li>
</ul>
<div>
<pre><code><span>importance</span><span> </span><span>&lt;-</span><span> </span><span>varImp</span><span>(</span><span>model_xgb</span><span>,</span><span> </span><span>scale</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>plot</span><span>(</span><span>importance</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/importance_xgb-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<ul>
<li>predicting test data</li>
</ul>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_xgb</span><span>,</span><span> </span><span>test_data</span><span>),</span><span> </span><span>test_data</span><span>$</span><span>classes</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##            Reference
## Prediction  benign malignant
##   benign       132         2
##   malignant      5        70
##                                           
##                Accuracy : 0.9665          
##                  95% CI : (0.9322, 0.9864)
##     No Information Rate : 0.6555          
##     P-Value [Acc > NIR] : &lt;2e-16          
##                                           
##                   Kappa : 0.9266          
##  Mcnemar's Test P-Value : 0.4497          
##                                           
##             Sensitivity : 0.9635          
##             Specificity : 0.9722          
##          Pos Pred Value : 0.9851          
##          Neg Pred Value : 0.9333          
##              Prevalence : 0.6555          
##          Detection Rate : 0.6316          
##    Detection Prevalence : 0.6411          
##       Balanced Accuracy : 0.9679          
##                                           
##        'Positive' Class : benign          
## 
</code></pre>
</div>
<div>
<pre><code><span>results</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>actual</span><span> </span><span>=</span><span> </span><span>test_data</span><span>$</span><span>classes</span><span>,</span><span>
</span><span>predict</span><span>(</span><span>model_xgb</span><span>,</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;prob&quot;</span><span>))</span><span>
</span><span>results</span><span>$</span><span>prediction</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>results</span><span>$</span><span>benign</span><span> </span><span>></span><span> </span><span>0.5</span><span>,</span><span> </span><span>&quot;benign&quot;</span><span>,</span><span>
</span><span>ifelse</span><span>(</span><span>results</span><span>$</span><span>malignant</span><span> </span><span>></span><span> </span><span>0.5</span><span>,</span><span> </span><span>&quot;malignant&quot;</span><span>,</span><span> </span><span>NA</span><span>))</span><span>
</span><span>results</span><span>$</span><span>correct</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>results</span><span>$</span><span>actual</span><span> </span><span>==</span><span> </span><span>results</span><span>$</span><span>prediction</span><span>,</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>results</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>prediction</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>correct</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;dodge&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/results_bar_xgb-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>ggplot</span><span>(</span><span>results</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>prediction</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>benign</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>correct</span><span>,</span><span> </span><span>shape</span><span> </span><span>=</span><span> </span><span>correct</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_jitter</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.6</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i0.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/results_jitter_xgb-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h3>Feature Selection</h3>
<p>Performing feature selection on the whole dataset would lead to prediction bias, we therefore need to run the whole modeling process on the training data alone!</p>
<ul>
<li>Correlation</li>
</ul>
<p>Correlations between all features are calculated and visualised with the <em>corrplot</em> package. I am then removing all features with a correlation higher than 0.7, keeping the feature with the lower mean.</p>
<div>
<pre><code><span>library</span><span>(</span><span>corrplot</span><span>)</span><span>
</span><span># calculate correlation matrix
</span><span>corMatMy</span><span> </span><span>&lt;-</span><span> </span><span>cor</span><span>(</span><span>train_data</span><span>[,</span><span> </span><span>-1</span><span>])</span><span>
</span><span>corrplot</span><span>(</span><span>corMatMy</span><span>,</span><span> </span><span>order</span><span> </span><span>=</span><span> </span><span>&quot;hclust&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/unnamed-chunk-25-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>#Apply correlation filter at 0.70,
</span><span>highlyCor</span><span> </span><span>&lt;-</span><span> </span><span>colnames</span><span>(</span><span>train_data</span><span>[,</span><span> </span><span>-1</span><span>])[</span><span>findCorrelation</span><span>(</span><span>corMatMy</span><span>,</span><span> </span><span>cutoff</span><span> </span><span>=</span><span> </span><span>0.7</span><span>,</span><span> </span><span>verbose</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)]</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Compare row 2  and column  3 with corr  0.899 
##   Means:  0.696 vs 0.575 so flagging column 2 
## Compare row 3  and column  7 with corr  0.736 
##   Means:  0.654 vs 0.55 so flagging column 3 
## All correlations &lt;= 0.7
</code></pre>
</div>
<div>
<pre><code><span># which variables are flagged for removal?
</span><span>highlyCor</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] &quot;uniformity_of_cell_size&quot;  &quot;uniformity_of_cell_shape&quot;
</code></pre>
</div>
<div>
<pre><code><span>#then we remove these variables
</span><span>train_data_cor</span><span> </span><span>&lt;-</span><span> </span><span>train_data</span><span>[,</span><span> </span><span>which</span><span>(</span><span>!</span><span>colnames</span><span>(</span><span>train_data</span><span>)</span><span> </span><span>%in%</span><span> </span><span>highlyCor</span><span>)]</span><span>
</span></code></pre>
</div>
<p></p>
<ul>
<li>Recursive Feature Elimination (RFE)</li>
</ul>
<p>Another way to choose features is with Recursive Feature Elimination. RFE uses a Random Forest algorithm to test combinations of features and rate each with an accuracy score. The combination with the highest score is usually preferential.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>7</span><span>)</span><span>
</span><span>results_rfe</span><span> </span><span>&lt;-</span><span> </span><span>rfe</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>train_data</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>train_data</span><span>$</span><span>classes</span><span>,</span><span> </span><span>sizes</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>1</span><span>:</span><span>9</span><span>),</span><span> </span><span>rfeControl</span><span> </span><span>=</span><span> </span><span>rfeControl</span><span>(</span><span>functions</span><span> </span><span>=</span><span> </span><span>rfFuncs</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;cv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span># chosen features
</span><span>predictors</span><span>(</span><span>results_rfe</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] &quot;bare_nuclei&quot;                 &quot;uniformity_of_cell_size&quot;    
## [3] &quot;clump_thickness&quot;             &quot;uniformity_of_cell_shape&quot;   
## [5] &quot;bland_chromatin&quot;             &quot;marginal_adhesion&quot;          
## [7] &quot;normal_nucleoli&quot;             &quot;single_epithelial_cell_size&quot;
## [9] &quot;mitosis&quot;
</code></pre>
</div>
<div>
<pre><code><span>train_data_rfe</span><span> </span><span>&lt;-</span><span> </span><span>train_data</span><span>[,</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>which</span><span>(</span><span>colnames</span><span>(</span><span>train_data</span><span>)</span><span> </span><span>%in%</span><span> </span><span>predictors</span><span>(</span><span>results_rfe</span><span>)))]</span><span>
</span></code></pre>
</div>
<p></p>
<ul>
<li>Genetic Algorithm (GA)</li>
</ul>
<p>The Genetic Algorithm (GA) has been developed based on evolutionary principles of natural selection: It aims to optimize a population of individuals with a given set of genotypes by modeling selection over time. In each generation (i.e. iteration), each individual’s fitness is calculated based on their genotypes. Then, the fittest individuals are chosen to produce the next generation. This subsequent generation of individuals will have genotypes resulting from (re-) combinations of the parental alleles. These new genotypes will again determine each individual’s fitness. This selection process is iterated for a specified number of generations and (ideally) leads to fixation of the fittest alleles in the gene pool.</p>
<p>This concept of optimization can be applied to non-evolutionary models as well, like feature selection processes in machine learning.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_ga</span><span> </span><span>&lt;-</span><span> </span><span>gafs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>train_data</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>train_data</span><span>$</span><span>classes</span><span>,</span><span>
</span><span>iters</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span># generations of algorithm
</span><span> </span><span>popSize</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span># population size for each generation
</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;malignant&quot;</span><span>,</span><span> </span><span>&quot;benign&quot;</span><span>),</span><span>
</span><span>gafsControl</span><span> </span><span>=</span><span> </span><span>gafsControl</span><span>(</span><span>functions</span><span> </span><span>=</span><span> </span><span>rfGA</span><span>,</span><span> </span><span># Assess fitness with RF
</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;cv&quot;</span><span>,</span><span> </span><span># 10 fold cross validation
</span><span> </span><span>genParallel</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span># Use parallel programming
</span><span> </span><span>allowParallel</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>plot</span><span>(</span><span>model_ga</span><span>)</span><span> </span><span># Plot mean fitness (AUC) by generation
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/unnamed-chunk-33-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>train_data_ga</span><span> </span><span>&lt;-</span><span> </span><span>train_data</span><span>[,</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>which</span><span>(</span><span>colnames</span><span>(</span><span>train_data</span><span>)</span><span> </span><span>%in%</span><span> </span><span>model_ga</span><span>$</span><span>ga</span><span>$</span><span>final</span><span>))]</span><span>
</span></code></pre>
</div>
<p></p>
<h3>Grid search with caret</h3>
<ul>
<li>Automatic Grid</li>
</ul>
<div>
<pre><code><span>set.seed</span><span>(</span><span>42</span><span>)</span><span>
</span><span>model_rf_tune_auto</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>classes</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;rf&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;scale&quot;</span><span>,</span><span> </span><span>&quot;center&quot;</span><span>),</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>savePredictions</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span>
</span><span>search</span><span> </span><span>=</span><span> </span><span>&quot;random&quot;</span><span>),</span><span>
</span><span>tuneLength</span><span> </span><span>=</span><span> </span><span>15</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>model_rf_tune_auto</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Random Forest 
## 
## 490 samples
##   9 predictor
##   2 classes: 'benign', 'malignant' 
## 
## Pre-processing: scaled (9), centered (9) 
## Resampling: Cross-Validated (10 fold, repeated 10 times) 
## Summary of sample sizes: 442, 441, 441, 441, 441, 441, ... 
## Resampling results across tuning parameters:
## 
##   mtry  Accuracy   Kappa    
##   1     0.9692153  0.9323624
##   2     0.9704277  0.9350498
##   5     0.9645085  0.9216721
##   6     0.9639087  0.9201998
##   7     0.9632842  0.9186919
##   8     0.9626719  0.9172257
##   9     0.9636801  0.9195036
## 
## Accuracy was used to select the optimal model using  the largest value.
## The final value used for the model was mtry = 2.
</code></pre>
</div>
<div>
<pre><code><span>plot</span><span>(</span><span>model_rf_tune_auto</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/unnamed-chunk-37-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<ul>
<li>
<p>Manual Grid</p>
</li>
<li>
<p>mtry: Number of variables randomly sampled as candidates at each split.</p>
</li>
</ul>
<div>
<pre><code><span>set.seed</span><span>(</span><span>42</span><span>)</span><span>
</span><span>grid</span><span> </span><span>&lt;-</span><span> </span><span>expand.grid</span><span>(</span><span>mtry</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>1</span><span>:</span><span>10</span><span>))</span><span>
</span><span>model_rf_tune_man</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>classes</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;rf&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;scale&quot;</span><span>,</span><span> </span><span>&quot;center&quot;</span><span>),</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>savePredictions</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span>
</span><span>search</span><span> </span><span>=</span><span> </span><span>&quot;random&quot;</span><span>),</span><span>
</span><span>tuneGrid</span><span> </span><span>=</span><span> </span><span>grid</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>model_rf_tune_man</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Random Forest 
## 
## 490 samples
##   9 predictor
##   2 classes: 'benign', 'malignant' 
## 
## Pre-processing: scaled (9), centered (9) 
## Resampling: Cross-Validated (10 fold, repeated 10 times) 
## Summary of sample sizes: 442, 441, 441, 441, 441, 441, ... 
## Resampling results across tuning parameters:
## 
##   mtry  Accuracy   Kappa    
##    1    0.9696153  0.9332392
##    2    0.9706440  0.9354737
##    3    0.9696194  0.9330647
##    4    0.9661495  0.9253163
##    5    0.9649252  0.9225586
##    6    0.9653209  0.9233806
##    7    0.9634881  0.9192265
##    8    0.9624718  0.9169227
##    9    0.9641005  0.9203072
##   10    0.9628760  0.9176675
## 
## Accuracy was used to select the optimal model using  the largest value.
## The final value used for the model was mtry = 2.
</code></pre>
</div>
<div>
<pre><code><span>plot</span><span>(</span><span>model_rf_tune_man</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/unnamed-chunk-41-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h3>Grid search with h2o</h3>
<p>The R package h2o provides a convenient interface to <a target="_blank" rel="nofollow" href="http://www.h2o.ai/h2o/">H2O</a>, which is an open-source machine learning and deep learning platform. H2O distributes a wide range of common machine learning algorithms for classification, regression and deep learning.</p>
<div>
<pre><code><span>library</span><span>(</span><span>h</span><span>2</span><span>o</span><span>)</span><span>
</span><span>h</span><span>2</span><span>o.init</span><span>(</span><span>nthreads</span><span> </span><span>=</span><span> </span><span>-1</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## 
## H2O is not running yet, starting it now...
## 
## Note:  In case of errors look at the following log files:
##     C:\Users\s_glan02\AppData\Local\Temp\RtmpwDqf33/h2o_s_glan02_started_from_r.out
##     C:\Users\s_glan02\AppData\Local\Temp\RtmpwDqf33/h2o_s_glan02_started_from_r.err
## 
## 
## Starting H2O JVM and connecting: . Connection successful!
## 
## R is connected to the H2O cluster: 
##     H2O cluster uptime:         1 seconds 815 milliseconds 
##     H2O cluster version:        3.10.3.6 
##     H2O cluster version age:    1 month and 10 days  
##     H2O cluster name:           H2O_started_from_R_s_glan02_tvy462 
##     H2O cluster total nodes:    1 
##     H2O cluster total memory:   3.54 GB 
##     H2O cluster total cores:    8 
##     H2O cluster allowed cores:  8 
##     H2O cluster healthy:        TRUE 
##     H2O Connection ip:          localhost 
##     H2O Connection port:        54321 
##     H2O Connection proxy:       NA 
##     R Version:                  R version 3.3.3 (2017-03-06)
</code></pre>
</div>
<div>
<pre><code><span>bc_data_hf</span><span> </span><span>&lt;-</span><span> </span><span>as.h2o</span><span>(</span><span>bc_data</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## 
  |                                                                       
  |                                                                 |   0%
  |                                                                       
  |=================================================================| 100%
</code></pre>
</div>
<div>
<pre><code><span>h</span><span>2</span><span>o.describe</span><span>(</span><span>bc_data_hf</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>x</span><span>,</span><span> </span><span>y</span><span>,</span><span> </span><span>Zeros</span><span>:</span><span>Sigma</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>group</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>x</span><span> </span><span>%in%</span><span> </span><span>c</span><span>(</span><span>&quot;Min&quot;</span><span>,</span><span> </span><span>&quot;Max&quot;</span><span>,</span><span> </span><span>&quot;Mean&quot;</span><span>),</span><span> </span><span>&quot;min, mean, max&quot;</span><span>,</span><span> </span><span>ifelse</span><span>(</span><span>x</span><span> </span><span>%in%</span><span> </span><span>c</span><span>(</span><span>&quot;NegInf&quot;</span><span>,</span><span> </span><span>&quot;PosInf&quot;</span><span>),</span><span> </span><span>&quot;Inf&quot;</span><span>,</span><span> </span><span>&quot;sigma, zeros&quot;</span><span>)))</span><span> </span><span>%>%</span><span> </span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>Label</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>y</span><span>),</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>x</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>4</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.6</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_color_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>axis.text.x</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>angle</span><span> </span><span>=</span><span> </span><span>45</span><span>,</span><span> </span><span>vjust</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>hjust</span><span> </span><span>=</span><span> </span><span>1</span><span>))</span><span> </span><span>+</span><span>
</span><span>facet_grid</span><span>(</span><span>group</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Feature&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Value&quot;</span><span>,</span><span>
</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/h2o_describe-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>library</span><span>(</span><span>reshape2</span><span>)</span><span> </span><span># for melting
</span><span>
</span><span>bc_data_hf</span><span>[,</span><span> </span><span>1</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.asfactor</span><span>(</span><span>bc_data_hf</span><span>[,</span><span> </span><span>1</span><span>])</span><span>
</span><span>cor</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.cor</span><span>(</span><span>bc_data_hf</span><span>)</span><span>
</span><span>rownames</span><span>(</span><span>cor</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>colnames</span><span>(</span><span>cor</span><span>)</span><span>
</span><span>melt</span><span>(</span><span>cor</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Var2</span><span> </span><span>=</span><span> </span><span>rep</span><span>(</span><span>rownames</span><span>(</span><span>cor</span><span>),</span><span> </span><span>nrow</span><span>(</span><span>cor</span><span>)))</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Var2</span><span> </span><span>=</span><span> </span><span>factor</span><span>(</span><span>Var2</span><span>,</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>colnames</span><span>(</span><span>cor</span><span>)))</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>variable</span><span> </span><span>=</span><span> </span><span>factor</span><span>(</span><span>variable</span><span>,</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>colnames</span><span>(</span><span>cor</span><span>)))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>variable</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>Var2</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>value</span><span>))</span><span> </span><span>+</span><span> </span><span>geom_tile</span><span>(</span><span>width</span><span> </span><span>=</span><span> </span><span>0.9</span><span>,</span><span> </span><span>height</span><span> </span><span>=</span><span> </span><span>0.9</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_gradient2</span><span>(</span><span>low</span><span> </span><span>=</span><span> </span><span>&quot;white&quot;</span><span>,</span><span> </span><span>high</span><span> </span><span>=</span><span> </span><span>&quot;red&quot;</span><span>,</span><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>&quot;Cor.&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>axis.text.x</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>angle</span><span> </span><span>=</span><span> </span><span>90</span><span>,</span><span> </span><span>vjust</span><span> </span><span>=</span><span> </span><span>0.5</span><span>,</span><span> </span><span>hjust</span><span> </span><span>=</span><span> </span><span>1</span><span>))</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/corr_plot-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h4>Training, validation and test data</h4>
<div>
<pre><code><span>splits</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.splitFrame</span><span>(</span><span>bc_data_hf</span><span>,</span><span> </span><span>ratios</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>0.7</span><span>,</span><span> </span><span>0.15</span><span>),</span><span> </span><span>seed</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span><span>train</span><span> </span><span>&lt;-</span><span> </span><span>splits</span><span>[[</span><span>1</span><span>]]</span><span>
</span><span>valid</span><span> </span><span>&lt;-</span><span> </span><span>splits</span><span>[[</span><span>2</span><span>]]</span><span>
</span><span>test</span><span> </span><span>&lt;-</span><span> </span><span>splits</span><span>[[</span><span>3</span><span>]]</span><span>
</span><span>response</span><span> </span><span>&lt;-</span><span> </span><span>&quot;classes&quot;</span><span>
</span><span>features</span><span> </span><span>&lt;-</span><span> </span><span>setdiff</span><span>(</span><span>colnames</span><span>(</span><span>train</span><span>),</span><span> </span><span>response</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>summary</span><span>(</span><span>train</span><span>$</span><span>classes</span><span>,</span><span> </span><span>exact_quantiles</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##  classes       
##  benign   :317 
##  malignant:174
</code></pre>
</div>
<div>
<pre><code><span>summary</span><span>(</span><span>valid</span><span>$</span><span>classes</span><span>,</span><span> </span><span>exact_quantiles</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##  classes      
##  benign   :71 
##  malignant:35
</code></pre>
</div>
<div>
<pre><code><span>summary</span><span>(</span><span>test</span><span>$</span><span>classes</span><span>,</span><span> </span><span>exact_quantiles</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##  classes      
##  benign   :70 
##  malignant:32
</code></pre>
</div>
<div>
<pre><code><span>pca</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.prcomp</span><span>(</span><span>training_frame</span><span> </span><span>=</span><span> </span><span>train</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>features</span><span>,</span><span>
</span><span>validation_frame</span><span> </span><span>=</span><span> </span><span>valid</span><span>,</span><span>
</span><span>transform</span><span> </span><span>=</span><span> </span><span>&quot;NORMALIZE&quot;</span><span>,</span><span>
</span><span>impute_missing</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>k</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span>
</span><span>seed</span><span> </span><span>=</span><span> </span><span>42</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## 
  |                                                                       
  |                                                                 |   0%
  |                                                                       
  |====================================================             |  80%
  |                                                                       
  |=================================================================| 100%
</code></pre>
</div>
<div>
<pre><code><span>eigenvec</span><span> </span><span>&lt;-</span><span> </span><span>as.data.frame</span><span>(</span><span>pca</span><span>@</span><span>model</span><span>$</span><span>eigenvectors</span><span>)</span><span>
</span><span>eigenvec</span><span>$</span><span>label</span><span> </span><span>&lt;-</span><span> </span><span>features</span><span>
</span><span>library</span><span>(</span><span>ggrepel</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>eigenvec</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>pc1</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>pc2</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>label</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;navy&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.7</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_text_repel</span><span>()</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/pca_features-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h4>Classification</h4>
<h5>Random Forest</h5>
<div>
<pre><code><span>hyper_params</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>
</span><span>ntrees</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>25</span><span>,</span><span> </span><span>50</span><span>,</span><span> </span><span>75</span><span>,</span><span> </span><span>100</span><span>),</span><span>
</span><span>max_depth</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>10</span><span>,</span><span> </span><span>20</span><span>,</span><span> </span><span>30</span><span>),</span><span>
</span><span>min_rows</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>3</span><span>,</span><span> </span><span>5</span><span>)</span><span>
</span><span>)</span><span>
</span><span>search_criteria</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>
</span><span>strategy</span><span> </span><span>=</span><span> </span><span>&quot;RandomDiscrete&quot;</span><span>,</span><span> </span><span>max_models</span><span> </span><span>=</span><span> </span><span>50</span><span>,</span><span>
</span><span>max_runtime_secs</span><span> </span><span>=</span><span> </span><span>360</span><span>,</span><span>
</span><span>stopping_rounds</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>stopping_metric</span><span> </span><span>=</span><span> </span><span>&quot;AUC&quot;</span><span>,</span><span> </span><span>stopping_tolerance</span><span> </span><span>=</span><span> </span><span>0.0005</span><span>,</span><span>
</span><span>seed</span><span> </span><span>=</span><span> </span><span>42</span><span>
</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>rf_grid</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.grid</span><span>(</span><span>algorithm</span><span> </span><span>=</span><span> </span><span>&quot;randomForest&quot;</span><span>,</span><span> </span><span># h2o.randomForest, </span><span> </span><span># alternatively h2o.gbm </span><span> </span><span># for Gradient boosting trees
</span><span> </span><span>x</span><span> </span><span>=</span><span> </span><span>features</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>response</span><span>,</span><span>
</span><span>grid_id</span><span> </span><span>=</span><span> </span><span>&quot;rf_grid&quot;</span><span>,</span><span>
</span><span>training_frame</span><span> </span><span>=</span><span> </span><span>train</span><span>,</span><span>
</span><span>validation_frame</span><span> </span><span>=</span><span> </span><span>valid</span><span>,</span><span>
</span><span>nfolds</span><span> </span><span>=</span><span> </span><span>25</span><span>,</span><span> </span><span>fold_assignment</span><span> </span><span>=</span><span> </span><span>&quot;Stratified&quot;</span><span>,</span><span>
</span><span>hyper_params</span><span> </span><span>=</span><span> </span><span>hyper_params</span><span>,</span><span>
</span><span>search_criteria</span><span> </span><span>=</span><span> </span><span>search_criteria</span><span>,</span><span>
</span><span>seed</span><span> </span><span>=</span><span> </span><span>42</span><span>
</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span># performance metrics where smaller is better -> order with decreasing = FALSE
</span><span>sort_options_1</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;mean_per_class_error&quot;</span><span>,</span><span> </span><span>&quot;mse&quot;</span><span>,</span><span> </span><span>&quot;err&quot;</span><span>,</span><span> </span><span>&quot;logloss&quot;</span><span>)</span><span>
</span><span>for</span><span> </span><span>(</span><span>sort_by_1</span><span> </span><span>in</span><span> </span><span>sort_options_1</span><span>)</span><span> </span><span>{</span><span>
</span><span>grid</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.getGrid</span><span>(</span><span>&quot;rf_grid&quot;</span><span>,</span><span> </span><span>sort_by</span><span> </span><span>=</span><span> </span><span>sort_by_1</span><span>,</span><span> </span><span>decreasing</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>model_ids</span><span> </span><span>&lt;-</span><span> </span><span>grid</span><span>@</span><span>model_ids</span><span>
</span><span>best_model</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.getModel</span><span>(</span><span>model_ids</span><span>[[</span><span>1</span><span>]])</span><span>
</span><span>h</span><span>2</span><span>o.saveModel</span><span>(</span><span>best_model</span><span>,</span><span> </span><span>path</span><span>=</span><span>&quot;models&quot;</span><span>,</span><span> </span><span>force</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>}</span><span>
</span><span># performance metrics where bigger is better -> order with decreasing = TRUE
</span><span>sort_options_2</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;auc&quot;</span><span>,</span><span> </span><span>&quot;precision&quot;</span><span>,</span><span> </span><span>&quot;accuracy&quot;</span><span>,</span><span> </span><span>&quot;recall&quot;</span><span>,</span><span> </span><span>&quot;specificity&quot;</span><span>)</span><span>
</span><span>for</span><span> </span><span>(</span><span>sort_by_2</span><span> </span><span>in</span><span> </span><span>sort_options_2</span><span>)</span><span> </span><span>{</span><span>
</span><span>grid</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.getGrid</span><span>(</span><span>&quot;rf_grid&quot;</span><span>,</span><span> </span><span>sort_by</span><span> </span><span>=</span><span> </span><span>sort_by_2</span><span>,</span><span> </span><span>decreasing</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>model_ids</span><span> </span><span>&lt;-</span><span> </span><span>grid</span><span>@</span><span>model_ids</span><span>
</span><span>best_model</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.getModel</span><span>(</span><span>model_ids</span><span>[[</span><span>1</span><span>]])</span><span>
</span><span>h</span><span>2</span><span>o.saveModel</span><span>(</span><span>best_model</span><span>,</span><span> </span><span>path</span><span> </span><span>=</span><span> </span><span>&quot;models&quot;</span><span>,</span><span> </span><span>force</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>files</span><span> </span><span>&lt;-</span><span> </span><span>list.files</span><span>(</span><span>path</span><span> </span><span>=</span><span> </span><span>&quot;models&quot;</span><span>)</span><span>
</span><span>rf_models</span><span> </span><span>&lt;-</span><span> </span><span>files</span><span>[</span><span>grep</span><span>(</span><span>&quot;rf_grid_model&quot;</span><span>,</span><span> </span><span>files</span><span>)]</span><span>
</span><span>for</span><span> </span><span>(</span><span>model_id</span><span> </span><span>in</span><span> </span><span>rf_models</span><span>)</span><span> </span><span>{</span><span>
</span><span>path</span><span> </span><span>&lt;-</span><span> </span><span>paste0</span><span>(</span><span>&quot;U:\\Github_blog\\Webinar\\Webinar_ML_for_disease\\models\\&quot;</span><span>,</span><span> </span><span>model_id</span><span>)</span><span>
</span><span>best_model</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.loadModel</span><span>(</span><span>path</span><span>)</span><span>
</span><span>mse_auc_test</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>model_id</span><span> </span><span>=</span><span> </span><span>model_id</span><span>,</span><span> </span><span>mse</span><span> </span><span>=</span><span> </span><span>h</span><span>2</span><span>o.mse</span><span>(</span><span>h</span><span>2</span><span>o.performance</span><span>(</span><span>best_model</span><span>,</span><span> </span><span>test</span><span>)),</span><span>
</span><span>auc</span><span> </span><span>=</span><span> </span><span>h</span><span>2</span><span>o.auc</span><span>(</span><span>h</span><span>2</span><span>o.performance</span><span>(</span><span>best_model</span><span>,</span><span> </span><span>test</span><span>)))</span><span>
</span><span>if</span><span> </span><span>(</span><span>model_id</span><span> </span><span>==</span><span> </span><span>rf_models</span><span>[[</span><span>1</span><span>]])</span><span> </span><span>{</span><span>
</span><span>mse_auc_test_comb</span><span> </span><span>&lt;-</span><span> </span><span>mse_auc_test</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>mse_auc_test_comb</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>mse_auc_test_comb</span><span>,</span><span> </span><span>mse_auc_test</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span><span>mse_auc_test_comb</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>x</span><span>,</span><span> </span><span>y</span><span>,</span><span> </span><span>mse</span><span>:</span><span>auc</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>model_id</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>y</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>model_id</span><span>))</span><span> </span><span>+</span><span>
</span><span>facet_grid</span><span>(</span><span>x</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>stat</span><span> </span><span>=</span><span> </span><span>&quot;identity&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>,</span><span> </span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;dodge&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>axis.text.x</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>angle</span><span> </span><span>=</span><span> </span><span>45</span><span>,</span><span> </span><span>vjust</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>hjust</span><span> </span><span>=</span><span> </span><span>1</span><span>),</span><span>
</span><span>plot.margin</span><span> </span><span>=</span><span> </span><span>unit</span><span>(</span><span>c</span><span>(</span><span>0.5</span><span>,</span><span> </span><span>0</span><span>,</span><span> </span><span>0</span><span>,</span><span> </span><span>1.5</span><span>),</span><span> </span><span>&quot;cm&quot;</span><span>))</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;value&quot;</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i0.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/auc_mse-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>for</span><span> </span><span>(</span><span>model_id</span><span> </span><span>in</span><span> </span><span>rf_models</span><span>)</span><span> </span><span>{</span><span>
</span><span>best_model</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.getModel</span><span>(</span><span>model_id</span><span>)</span><span>
</span><span>finalRf_predictions</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>model_id</span><span> </span><span>=</span><span> </span><span>rep</span><span>(</span><span>best_model</span><span>@</span><span>model_id</span><span>,</span><span> </span><span>nrow</span><span>(</span><span>test</span><span>)),</span><span>
</span><span>actual</span><span> </span><span>=</span><span> </span><span>as.vector</span><span>(</span><span>test</span><span>$</span><span>classes</span><span>),</span><span> </span><span>as.data.frame</span><span>(</span><span>h</span><span>2</span><span>o.predict</span><span>(</span><span>object</span><span> </span><span>=</span><span> </span><span>best_model</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test</span><span>)))</span><span>
</span><span>finalRf_predictions</span><span>$</span><span>accurate</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>actual</span><span> </span><span>==</span><span> </span><span>finalRf_predictions</span><span>$</span><span>predict</span><span>,</span><span> </span><span>&quot;yes&quot;</span><span>,</span><span> </span><span>&quot;no&quot;</span><span>)</span><span>
</span><span>finalRf_predictions</span><span>$</span><span>predict_stringent</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>benign</span><span> </span><span>></span><span> </span><span>0.8</span><span>,</span><span> </span><span>&quot;benign&quot;</span><span>,</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>malignant</span><span> </span><span>></span><span> </span><span>0.8</span><span>,</span><span> </span><span>&quot;malignant&quot;</span><span>,</span><span> </span><span>&quot;uncertain&quot;</span><span>))</span><span>
</span><span>finalRf_predictions</span><span>$</span><span>accurate_stringent</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>actual</span><span> </span><span>==</span><span> </span><span>finalRf_predictions</span><span>$</span><span>predict_stringent</span><span>,</span><span> </span><span>&quot;yes&quot;</span><span>,</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>predict_stringent</span><span> </span><span>==</span><span> </span><span>&quot;uncertain&quot;</span><span>,</span><span> </span><span>&quot;na&quot;</span><span>,</span><span> </span><span>&quot;no&quot;</span><span>))</span><span>
</span><span>if</span><span> </span><span>(</span><span>model_id</span><span> </span><span>==</span><span> </span><span>rf_models</span><span>[[</span><span>1</span><span>]])</span><span> </span><span>{</span><span>
</span><span>finalRf_predictions_comb</span><span> </span><span>&lt;-</span><span> </span><span>finalRf_predictions</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>finalRf_predictions_comb</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>finalRf_predictions_comb</span><span>,</span><span> </span><span>finalRf_predictions</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## 
  |                                                                       
  |                                                                 |   0%
  |                                                                       
  |=================================================================| 100%
## 
  |                                                                       
  |                                                                 |   0%
  |                                                                       
  |=================================================================| 100%
## 
  |                                                                       
  |                                                                 |   0%
  |                                                                       
  |=================================================================| 100%
</code></pre>
</div>
<div>
<pre><code><span>finalRf_predictions_comb</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>actual</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>accurate</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;dodge&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>~</span><span> </span><span>model_id</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>3</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;Were\npredictions\naccurate?&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Default predictions&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i0.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/final_predictions_rf-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>finalRf_predictions_comb</span><span> </span><span>%>%</span><span>
</span><span>subset</span><span>(</span><span>accurate_stringent</span><span> </span><span>!=</span><span> </span><span>&quot;na&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>actual</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>accurate_stringent</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;dodge&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>~</span><span> </span><span>model_id</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>3</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;Were\npredictions\naccurate?&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Stringent predictions&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i0.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/final_predictions_rf-2.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>rf_model</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.loadModel</span><span>(</span><span>&quot;models/rf_grid_model_6&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>h</span><span>2</span><span>o.varimp_plot</span><span>(</span><span>rf_model</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/unnamed-chunk-50-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>#h2o.varimp(rf_model)
</span></code></pre>
</div>
<div>
<pre><code><span>h</span><span>2</span><span>o.mean_per_class_error</span><span>(</span><span>rf_model</span><span>,</span><span> </span><span>train</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>valid</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>xval</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##       train       valid        xval 
## 0.024674571 0.007042254 0.023097284
</code></pre>
</div>
<div>
<pre><code><span>h</span><span>2</span><span>o.confusionMatrix</span><span>(</span><span>rf_model</span><span>,</span><span> </span><span>valid</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix (vertical: actual; across: predicted)  for max f1 @ threshold = 0.293125896751881:
##           benign malignant    Error    Rate
## benign        70         1 0.014085   =1/71
## malignant      0        35 0.000000   =0/35
## Totals        70        36 0.009434  =1/106
</code></pre>
</div>
<div>
<pre><code><span>plot</span><span>(</span><span>rf_model</span><span>,</span><span>
</span><span>timestep</span><span> </span><span>=</span><span> </span><span>&quot;number_of_trees&quot;</span><span>,</span><span>
</span><span>metric</span><span> </span><span>=</span><span> </span><span>&quot;classification_error&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i0.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/unnamed-chunk-53-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>plot</span><span>(</span><span>rf_model</span><span>,</span><span>
</span><span>timestep</span><span> </span><span>=</span><span> </span><span>&quot;number_of_trees&quot;</span><span>,</span><span>
</span><span>metric</span><span> </span><span>=</span><span> </span><span>&quot;logloss&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i0.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/unnamed-chunk-54-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>plot</span><span>(</span><span>rf_model</span><span>,</span><span>
</span><span>timestep</span><span> </span><span>=</span><span> </span><span>&quot;number_of_trees&quot;</span><span>,</span><span>
</span><span>metric</span><span> </span><span>=</span><span> </span><span>&quot;AUC&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/unnamed-chunk-55-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>plot</span><span>(</span><span>rf_model</span><span>,</span><span>
</span><span>timestep</span><span> </span><span>=</span><span> </span><span>&quot;number_of_trees&quot;</span><span>,</span><span>
</span><span>metric</span><span> </span><span>=</span><span> </span><span>&quot;rmse&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/unnamed-chunk-56-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>h</span><span>2</span><span>o.auc</span><span>(</span><span>rf_model</span><span>,</span><span> </span><span>train</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 0.989521
</code></pre>
</div>
<div>
<pre><code><span>h</span><span>2</span><span>o.auc</span><span>(</span><span>rf_model</span><span>,</span><span> </span><span>valid</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 0.9995976
</code></pre>
</div>
<div>
<pre><code><span>h</span><span>2</span><span>o.auc</span><span>(</span><span>rf_model</span><span>,</span><span> </span><span>xval</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 0.9890496
</code></pre>
</div>
<div>
<pre><code><span>perf</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.performance</span><span>(</span><span>rf_model</span><span>,</span><span> </span><span>test</span><span>)</span><span>
</span><span>perf</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## H2OBinomialMetrics: drf
## 
## MSE:  0.03673598
## RMSE:  0.1916663
## LogLoss:  0.1158835
## Mean Per-Class Error:  0.0625
## AUC:  0.990625
## Gini:  0.98125
## 
## Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
##           benign malignant    Error    Rate
## benign        70         0 0.000000   =0/70
## malignant      4        28 0.125000   =4/32
## Totals        74        28 0.039216  =4/102
## 
## Maximum Metrics: Maximum metrics at their respective thresholds
##                         metric threshold    value idx
## 1                       max f1  0.735027 0.933333  25
## 2                       max f2  0.294222 0.952381  37
## 3                 max f0point5  0.735027 0.972222  25
## 4                 max accuracy  0.735027 0.960784  25
## 5                max precision  1.000000 1.000000   0
## 6                   max recall  0.294222 1.000000  37
## 7              max specificity  1.000000 1.000000   0
## 8             max absolute_mcc  0.735027 0.909782  25
## 9   max min_per_class_accuracy  0.424524 0.937500  31
## 10 max mean_per_class_accuracy  0.294222 0.942857  37
## 
## Gains/Lift Table: Extract with `h2o.gainsLift(&lt;model>, &lt;data>)` or `h2o.gainsLift(&lt;model>, valid=&lt;T/F>, xval=&lt;T/F>)`
</code></pre>
</div>
<div>
<pre><code><span>plot</span><span>(</span><span>perf</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/auc_curve-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>h</span><span>2</span><span>o.logloss</span><span>(</span><span>perf</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 0.1158835
</code></pre>
</div>
<div>
<pre><code><span>h</span><span>2</span><span>o.mse</span><span>(</span><span>perf</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 0.03673598
</code></pre>
</div>
<div>
<pre><code><span>h</span><span>2</span><span>o.auc</span><span>(</span><span>perf</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 0.990625
</code></pre>
</div>
<div>
<pre><code><span>head</span><span>(</span><span>h</span><span>2</span><span>o.metric</span><span>(</span><span>perf</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Metrics for Thresholds: Binomial metrics as a function of classification thresholds
##   threshold       f1       f2 f0point5 accuracy precision   recall
## 1  1.000000 0.171429 0.114504 0.340909 0.715686  1.000000 0.093750
## 2  0.998333 0.222222 0.151515 0.416667 0.725490  1.000000 0.125000
## 3  0.998000 0.270270 0.187970 0.480769 0.735294  1.000000 0.156250
## 4  0.997222 0.315789 0.223881 0.535714 0.745098  1.000000 0.187500
## 5  0.996210 0.358974 0.259259 0.583333 0.754902  1.000000 0.218750
## 6  0.994048 0.400000 0.294118 0.625000 0.764706  1.000000 0.250000
##   specificity absolute_mcc min_per_class_accuracy mean_per_class_accuracy
## 1    1.000000     0.257464               0.093750                0.546875
## 2    1.000000     0.298807               0.125000                0.562500
## 3    1.000000     0.335794               0.156250                0.578125
## 4    1.000000     0.369755               0.187500                0.593750
## 5    1.000000     0.401478               0.218750                0.609375
## 6    1.000000     0.431474               0.250000                0.625000
##   tns fns fps tps      tnr      fnr      fpr      tpr idx
## 1  70  29   0   3 1.000000 0.906250 0.000000 0.093750   0
## 2  70  28   0   4 1.000000 0.875000 0.000000 0.125000   1
## 3  70  27   0   5 1.000000 0.843750 0.000000 0.156250   2
## 4  70  26   0   6 1.000000 0.812500 0.000000 0.187500   3
## 5  70  25   0   7 1.000000 0.781250 0.000000 0.218750   4
## 6  70  24   0   8 1.000000 0.750000 0.000000 0.250000   5
</code></pre>
</div>
<div>
<pre><code><span>finalRf_predictions</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>actual</span><span> </span><span>=</span><span> </span><span>as.vector</span><span>(</span><span>test</span><span>$</span><span>classes</span><span>),</span><span> </span><span>as.data.frame</span><span>(</span><span>h</span><span>2</span><span>o.predict</span><span>(</span><span>object</span><span> </span><span>=</span><span> </span><span>rf_model</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test</span><span>)))</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## 
  |                                                                       
  |                                                                 |   0%
  |                                                                       
  |=================================================================| 100%
</code></pre>
</div>
<div>
<pre><code><span>finalRf_predictions</span><span>$</span><span>accurate</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>actual</span><span> </span><span>==</span><span> </span><span>finalRf_predictions</span><span>$</span><span>predict</span><span>,</span><span> </span><span>&quot;yes&quot;</span><span>,</span><span> </span><span>&quot;no&quot;</span><span>)</span><span>
</span><span>finalRf_predictions</span><span>$</span><span>predict_stringent</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>benign</span><span> </span><span>></span><span> </span><span>0.8</span><span>,</span><span> </span><span>&quot;benign&quot;</span><span>,</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>malignant</span><span> </span><span>></span><span> </span><span>0.8</span><span>,</span><span> </span><span>&quot;malignant&quot;</span><span>,</span><span> </span><span>&quot;uncertain&quot;</span><span>))</span><span>
</span><span>finalRf_predictions</span><span>$</span><span>accurate_stringent</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>actual</span><span> </span><span>==</span><span> </span><span>finalRf_predictions</span><span>$</span><span>predict_stringent</span><span>,</span><span> </span><span>&quot;yes&quot;</span><span>,</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>predict_stringent</span><span> </span><span>==</span><span> </span><span>&quot;uncertain&quot;</span><span>,</span><span> </span><span>&quot;na&quot;</span><span>,</span><span> </span><span>&quot;no&quot;</span><span>))</span><span>
</span><span>finalRf_predictions</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>actual</span><span>,</span><span> </span><span>predict</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>dplyr</span><span>::</span><span>summarise</span><span>(</span><span>n</span><span> </span><span>=</span><span> </span><span>n</span><span>())</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Source: local data frame [3 x 3]
## Groups: actual [?]
## 
##      actual   predict     n
##      &lt;fctr>    &lt;fctr> &lt;int>
## 1    benign    benign    62
## 2    benign malignant     8
## 3 malignant malignant    32
</code></pre>
</div>
<div>
<pre><code><span>finalRf_predictions</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>actual</span><span>,</span><span> </span><span>predict_stringent</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>dplyr</span><span>::</span><span>summarise</span><span>(</span><span>n</span><span> </span><span>=</span><span> </span><span>n</span><span>())</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Source: local data frame [4 x 3]
## Groups: actual [?]
## 
##      actual predict_stringent     n
##      &lt;fctr>             &lt;chr> &lt;int>
## 1    benign            benign    61
## 2    benign         uncertain     9
## 3 malignant         malignant    26
## 4 malignant         uncertain     6
</code></pre>
</div>
<div>
<pre><code><span>finalRf_predictions</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>actual</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>accurate</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;dodge&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;Were\npredictions\naccurate?&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Default predictions&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/unnamed-chunk-62-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>finalRf_predictions</span><span> </span><span>%>%</span><span>
</span><span>subset</span><span>(</span><span>accurate_stringent</span><span> </span><span>!=</span><span> </span><span>&quot;na&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>actual</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>accurate_stringent</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;dodge&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;Were\npredictions\naccurate?&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Stringent predictions&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/unnamed-chunk-62-2.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>df</span><span> </span><span>&lt;-</span><span> </span><span>finalRf_predictions</span><span>[,</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>3</span><span>,</span><span> </span><span>4</span><span>)]</span><span>
</span><span>thresholds</span><span> </span><span>&lt;-</span><span> </span><span>seq</span><span>(</span><span>from</span><span> </span><span>=</span><span> </span><span>0</span><span>,</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>0.1</span><span>)</span><span>
</span><span>prop_table</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>threshold</span><span> </span><span>=</span><span> </span><span>thresholds</span><span>,</span><span> </span><span>prop_true_b</span><span> </span><span>=</span><span> </span><span>NA</span><span>,</span><span> </span><span>prop_true_m</span><span> </span><span>=</span><span> </span><span>NA</span><span>)</span><span>
</span><span>for</span><span> </span><span>(</span><span>threshold</span><span> </span><span>in</span><span> </span><span>thresholds</span><span>)</span><span> </span><span>{</span><span>
</span><span>pred</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>df</span><span>$</span><span>benign</span><span> </span><span>></span><span> </span><span>threshold</span><span>,</span><span> </span><span>&quot;benign&quot;</span><span>,</span><span> </span><span>&quot;malignant&quot;</span><span>)</span><span>
</span><span>pred_t</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>pred</span><span> </span><span>==</span><span> </span><span>df</span><span>$</span><span>actual</span><span>,</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>group</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>df</span><span>,</span><span> </span><span>&quot;pred&quot;</span><span> </span><span>=</span><span> </span><span>pred_t</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>group_by</span><span>(</span><span>actual</span><span>,</span><span> </span><span>pred</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>dplyr</span><span>::</span><span>summarise</span><span>(</span><span>n</span><span> </span><span>=</span><span> </span><span>n</span><span>())</span><span>
</span><span>group_b</span><span> </span><span>&lt;-</span><span> </span><span>filter</span><span>(</span><span>group</span><span>,</span><span> </span><span>actual</span><span> </span><span>==</span><span> </span><span>&quot;benign&quot;</span><span>)</span><span>
</span><span>prop_b</span><span> </span><span>&lt;-</span><span> </span><span>sum</span><span>(</span><span>filter</span><span>(</span><span>group_b</span><span>,</span><span> </span><span>pred</span><span> </span><span>==</span><span> </span><span>TRUE</span><span>)</span><span>$</span><span>n</span><span>)</span><span> </span><span>/</span><span> </span><span>sum</span><span>(</span><span>group_b</span><span>$</span><span>n</span><span>)</span><span>
</span><span>prop_table</span><span>[</span><span>prop_table</span><span>$</span><span>threshold</span><span> </span><span>==</span><span> </span><span>threshold</span><span>,</span><span> </span><span>&quot;prop_true_b&quot;</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>prop_b</span><span>
</span><span>group_m</span><span> </span><span>&lt;-</span><span> </span><span>filter</span><span>(</span><span>group</span><span>,</span><span> </span><span>actual</span><span> </span><span>==</span><span> </span><span>&quot;malignant&quot;</span><span>)</span><span>
</span><span>prop_m</span><span> </span><span>&lt;-</span><span> </span><span>sum</span><span>(</span><span>filter</span><span>(</span><span>group_m</span><span>,</span><span> </span><span>pred</span><span> </span><span>==</span><span> </span><span>TRUE</span><span>)</span><span>$</span><span>n</span><span>)</span><span> </span><span>/</span><span> </span><span>sum</span><span>(</span><span>group_m</span><span>$</span><span>n</span><span>)</span><span>
</span><span>prop_table</span><span>[</span><span>prop_table</span><span>$</span><span>threshold</span><span> </span><span>==</span><span> </span><span>threshold</span><span>,</span><span> </span><span>&quot;prop_true_m&quot;</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>prop_m</span><span>
</span><span>}</span><span>
</span><span>prop_table</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>x</span><span>,</span><span> </span><span>y</span><span>,</span><span> </span><span>prop_true_b</span><span>:</span><span>prop_true_m</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>threshold</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>y</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>x</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>()</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_color_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;proportion of true predictions&quot;</span><span>,</span><span>
</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;b: benign cases\nm: malignant cases&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i0.wp.com/shiring.github.io/machine_learning/2017/03/31/webinar_code_files/figure-markdown_github/prop_table-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>h</span><span>2</span><span>o.shutdown</span><span>()</span><span>
</span></code></pre>
</div>
<hr>
<p>If you are interested in more machine learning posts, check out <a target="_blank" rel="nofollow" href="https://shiring.github.io/categories.html#machine_learning-ref">the category listing for <strong>machine_learning</strong> on my blog</a>.</p>
<hr>
<p></p>
<div>
<pre><code><span>sessionInfo</span><span>()</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## R version 3.3.3 (2017-03-06)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 7 x64 (build 7601) Service Pack 1
## 
## locale:
## [1] LC_COLLATE=English_United States.1252 
## [2] LC_CTYPE=English_United States.1252   
## [3] LC_MONETARY=English_United States.1252
## [4] LC_NUMERIC=C                          
## [5] LC_TIME=English_United States.1252    
## 
## attached base packages:
## [1] stats4    parallel  stats     graphics  grDevices utils     datasets 
## [8] methods   base     
## 
## other attached packages:
##  [1] ggrepel_0.6.5        reshape2_1.4.2       h2o_3.10.3.6        
##  [4] corrplot_0.77        plyr_1.8.4           xgboost_0.6-4       
##  [7] randomForest_4.6-12  dplyr_0.5.0          caret_6.0-73        
## [10] lattice_0.20-35      doParallel_1.0.10    iterators_1.0.8     
## [13] foreach_1.4.3        tidyr_0.6.1          pcaGoPromoter_1.18.0
## [16] Biostrings_2.42.1    XVector_0.14.0       IRanges_2.8.1       
## [19] S4Vectors_0.12.1     BiocGenerics_0.20.0  ellipse_0.3-8       
## [22] ggplot2_2.2.1.9000  
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_0.12.10         class_7.3-14         assertthat_0.1      
##  [4] rprojroot_1.2        digest_0.6.12        R6_2.2.0            
##  [7] backports_1.0.5      MatrixModels_0.4-1   RSQLite_1.1-2       
## [10] evaluate_0.10        e1071_1.6-8          zlibbioc_1.20.0     
## [13] lazyeval_0.2.0       minqa_1.2.4          data.table_1.10.4   
## [16] SparseM_1.76         car_2.1-4            nloptr_1.0.4        
## [19] Matrix_1.2-8         rmarkdown_1.4        labeling_0.3        
## [22] splines_3.3.3        lme4_1.1-12          stringr_1.2.0       
## [25] RCurl_1.95-4.8       munsell_0.4.3        mgcv_1.8-17         
## [28] htmltools_0.3.5      nnet_7.3-12          tibble_1.2          
## [31] codetools_0.2-15     MASS_7.3-45          bitops_1.0-6        
## [34] ModelMetrics_1.1.0   grid_3.3.3           nlme_3.1-131        
## [37] jsonlite_1.3         gtable_0.2.0         DBI_0.6             
## [40] magrittr_1.5         scales_0.4.1         stringi_1.1.3       
## [43] RColorBrewer_1.1-2   tools_3.3.3          Biobase_2.34.0      
## [46] pbkrtest_0.4-7       yaml_2.1.14          AnnotationDbi_1.36.2
## [49] colorspace_1.3-2     memoise_1.0.0        knitr_1.15.1        
## [52] quantreg_5.29
</code></pre>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://shiring.github.io/machine_learning/2017/03/31/webinar_code"> Shirin's playgRound</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/pP3y0dtPUeQ"></a></div>