<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/can-we-predict-flu-deaths-with-machine-learning-and-r/"></div></div><div><a data-url="https://www.r-bloggers.com/can-we-predict-flu-deaths-with-machine-learning-and-r/" data-counturl="https://www.r-bloggers.com/can-we-predict-flu-deaths-with-machine-learning-and-r/" data-count="horizontal" data-text="Can we predict flu deaths with Machine Learning and R?" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post"> Shirin's playgRound</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Among the many R packages, there is the <a target="_blank" rel="nofollow" href="https://mran.microsoft.com/web/packages/outbreaks/outbreaks.pdf">outbreaks</a> package. It contains datasets on epidemics, on of which is from the 2013 outbreak of <a target="_blank" rel="nofollow" href="http://www.who.int/influenza/human_animal_interface/faq_H7N9/en/">influenza A H7N9</a> in <a target="_blank" rel="nofollow" href="http://www.who.int/influenza/human_animal_interface/influenza_h7n9/ChinaH7N9JointMissionReport2013u.pdf?ua=1">China</a>, as analysed by Kucharski et al. (2014):</p>
<blockquote>
<p>A. Kucharski, H. Mills, A. Pinsent, C. Fraser, M. Van Kerkhove, C. A. Donnelly, and S. Riley. 2014. Distinguishing between reservoir exposure and human-to-human transmission for emerging pathogens using case onset data. PLOS Currents Outbreaks. Mar 7, edition 1. doi: 10.1371/currents.outbreaks.e1473d9bfc99d080ca242139a06c455f.</p>
</blockquote>
<blockquote>
<p>A. Kucharski, H. Mills, A. Pinsent, C. Fraser, M. Van Kerkhove, C. A. Donnelly, and S. Riley. 2014. Data from: Distinguishing between reservoir exposure and human-to-human transmission for emerging pathogens using case onset data. Dryad Digital Repository. <a target="_blank" rel="nofollow" href="http://dx.doi.org/10.5061/dryad.2g43n">http://dx.doi.org/10.5061/dryad.2g43n</a>.</p>
</blockquote>
<p>I will be using their data as an example to test whether we can use Machine Learning algorithms for predicting disease outcome.</p>
<hr>
<p><strong>Disclaimer:</strong> I am not an expert in Machine Learning. Everything I know, I taught myself. So, if you identify any mistakes or have tips and tricks for improvement, please don’t hesitate to let me know! Thanks. <img alt="🙂" class="wp-smiley" src="https://s.w.org/images/core/emoji/2/72x72/1f642.png"></p>
<hr>
<p></p>
<h1>The data</h1>
<p>The dataset contains case ID, date of onset, date of hospitalisation, date of outcome, gender, age, province and of course the outcome: Death or Recovery.<br>
I can already see that there are a couple of missing values in the data, which I will deal with later.</p>
<div>
<pre><code><span># install and load package
</span><span>if</span><span> </span><span>(</span><span>!</span><span>require</span><span>(</span><span>&quot;outbreaks&quot;</span><span>))</span><span> </span><span>install.packages</span><span>(</span><span>&quot;outbreaks&quot;</span><span>)</span><span>
</span><span>library</span><span>(</span><span>outbreaks</span><span>)</span><span>
</span><span>fluH7N9.china.2013_backup</span><span> </span><span>&lt;-</span><span> </span><span>fluH7N9.china.2013</span><span> </span><span># back up original dataset in case something goes awry along the way
</span><span>
</span><span># convert ? to NAs
</span><span>fluH7N9.china.2013</span><span>$</span><span>age</span><span>[</span><span>which</span><span>(</span><span>fluH7N9.china.2013</span><span>$</span><span>age</span><span> </span><span>==</span><span> </span><span>&quot;?&quot;</span><span>)]</span><span> </span><span>&lt;-</span><span> </span><span>NA</span><span>
</span><span># create a new column with case ID
</span><span>fluH7N9.china.2013</span><span>$</span><span>case.ID</span><span> </span><span>&lt;-</span><span> </span><span>paste</span><span>(</span><span>&quot;case&quot;</span><span>,</span><span> </span><span>fluH7N9.china.2013</span><span>$</span><span>case.ID</span><span>,</span><span> </span><span>sep</span><span> </span><span>=</span><span> </span><span>&quot;_&quot;</span><span>)</span><span>
</span><span>head</span><span>(</span><span>fluH7N9.china.2013</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##   case.ID date.of.onset date.of.hospitalisation date.of.outcome outcome gender age province
## 1  case_1    2013-02-19                    &lt;NA>      2013-03-04   Death      m  87 Shanghai
## 2  case_2    2013-02-27              2013-03-03      2013-03-10   Death      m  27 Shanghai
## 3  case_3    2013-03-09              2013-03-19      2013-04-09   Death      f  35    Anhui
## 4  case_4    2013-03-19              2013-03-27            &lt;NA>    &lt;NA>      f  45  Jiangsu
## 5  case_5    2013-03-19              2013-03-30      2013-05-15 Recover      f  48  Jiangsu
## 6  case_6    2013-03-21              2013-03-28      2013-04-26   Death      f  32  Jiangsu
</code></pre>
</div>
<p></p>
<p>Before I start preparing the data for Machine Learning, I want to get an idea of the distribution of the data points and their different variables by plotting.</p>
<p>Most provinces have only a handful of cases, so I am combining them into the category “other” and keep only Jiangsu, Shanghai and Zhejian and separate provinces.</p>
<div>
<pre><code><span># gather for plotting with ggplot2
</span><span>library</span><span>(</span><span>tidyr</span><span>)</span><span>
</span><span>fluH7N9.china.2013_gather</span><span> </span><span>&lt;-</span><span> </span><span>fluH7N9.china.2013</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>Group</span><span>,</span><span> </span><span>Date</span><span>,</span><span> </span><span>date.of.onset</span><span>:</span><span>date.of.outcome</span><span>)</span><span>
</span><span># rearrange group order
</span><span>fluH7N9.china.2013_gather</span><span>$</span><span>Group</span><span> </span><span>&lt;-</span><span> </span><span>factor</span><span>(</span><span>fluH7N9.china.2013_gather</span><span>$</span><span>Group</span><span>,</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;date.of.onset&quot;</span><span>,</span><span> </span><span>&quot;date.of.hospitalisation&quot;</span><span>,</span><span> </span><span>&quot;date.of.outcome&quot;</span><span>))</span><span>
</span><span># rename groups
</span><span>library</span><span>(</span><span>plyr</span><span>)</span><span>
</span><span>fluH7N9.china.2013_gather</span><span>$</span><span>Group</span><span> </span><span>&lt;-</span><span> </span><span>mapvalues</span><span>(</span><span>fluH7N9.china.2013_gather</span><span>$</span><span>Group</span><span>,</span><span> </span><span>from</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;date.of.onset&quot;</span><span>,</span><span> </span><span>&quot;date.of.hospitalisation&quot;</span><span>,</span><span> </span><span>&quot;date.of.outcome&quot;</span><span>),</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Date of onset&quot;</span><span>,</span><span> </span><span>&quot;Date of hospitalisation&quot;</span><span>,</span><span> </span><span>&quot;Date of outcome&quot;</span><span>))</span><span>
</span><span># renaming provinces
</span><span>fluH7N9.china.2013_gather</span><span>$</span><span>province</span><span> </span><span>&lt;-</span><span> </span><span>mapvalues</span><span>(</span><span>fluH7N9.china.2013_gather</span><span>$</span><span>province</span><span>,</span><span> </span><span>from</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Anhui&quot;</span><span>,</span><span> </span><span>&quot;Beijing&quot;</span><span>,</span><span> </span><span>&quot;Fujian&quot;</span><span>,</span><span> </span><span>&quot;Guangdong&quot;</span><span>,</span><span> </span><span>&quot;Hebei&quot;</span><span>,</span><span> </span><span>&quot;Henan&quot;</span><span>,</span><span> </span><span>&quot;Hunan&quot;</span><span>,</span><span> </span><span>&quot;Jiangxi&quot;</span><span>,</span><span> </span><span>&quot;Shandong&quot;</span><span>,</span><span> </span><span>&quot;Taiwan&quot;</span><span>),</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>rep</span><span>(</span><span>&quot;Other&quot;</span><span>,</span><span> </span><span>10</span><span>))</span><span>
</span><span># add a level for unknown gender
</span><span>levels</span><span>(</span><span>fluH7N9.china.2013_gather</span><span>$</span><span>gender</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>levels</span><span>(</span><span>fluH7N9.china.2013_gather</span><span>$</span><span>gender</span><span>),</span><span> </span><span>&quot;unknown&quot;</span><span>)</span><span>
</span><span>fluH7N9.china.2013_gather</span><span>$</span><span>gender</span><span>[</span><span>is.na</span><span>(</span><span>fluH7N9.china.2013_gather</span><span>$</span><span>gender</span><span>)]</span><span> </span><span>&lt;-</span><span> </span><span>&quot;unknown&quot;</span><span>
</span><span># rearrange province order so that Other is the last
</span><span>fluH7N9.china.2013_gather</span><span>$</span><span>province</span><span> </span><span>&lt;-</span><span> </span><span>factor</span><span>(</span><span>fluH7N9.china.2013_gather</span><span>$</span><span>province</span><span>,</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Jiangsu&quot;</span><span>,</span><span> </span><span>&quot;Shanghai&quot;</span><span>,</span><span> </span><span>&quot;Zhejiang&quot;</span><span>,</span><span> </span><span>&quot;Other&quot;</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span># preparing my ggplot2 theme of choice
</span><span>
</span><span>library</span><span>(</span><span>ggplot2</span><span>)</span><span>
</span><span>my_theme</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>base_size</span><span> </span><span>=</span><span> </span><span>12</span><span>,</span><span> </span><span>base_family</span><span> </span><span>=</span><span> </span><span>&quot;sans&quot;</span><span>){</span><span>
</span><span>theme_minimal</span><span>(</span><span>base_size</span><span> </span><span>=</span><span> </span><span>base_size</span><span>,</span><span> </span><span>base_family</span><span> </span><span>=</span><span> </span><span>base_family</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>
</span><span>axis.text</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>12</span><span>),</span><span>
</span><span>axis.text.x</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>angle</span><span> </span><span>=</span><span> </span><span>45</span><span>,</span><span> </span><span>vjust</span><span> </span><span>=</span><span> </span><span>0.5</span><span>,</span><span> </span><span>hjust</span><span> </span><span>=</span><span> </span><span>0.5</span><span>),</span><span>
</span><span>axis.title</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>14</span><span>),</span><span>
</span><span>panel.grid.major</span><span> </span><span>=</span><span> </span><span>element_line</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>),</span><span>
</span><span>panel.grid.minor</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>panel.background</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;aliceblue&quot;</span><span>),</span><span>
</span><span>strip.background</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;lightgrey&quot;</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1</span><span>),</span><span>
</span><span>strip.text</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>face</span><span> </span><span>=</span><span> </span><span>&quot;bold&quot;</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>12</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;black&quot;</span><span>),</span><span>
</span><span>legend.position</span><span> </span><span>=</span><span> </span><span>&quot;bottom&quot;</span><span>,</span><span>
</span><span>legend.justification</span><span> </span><span>=</span><span> </span><span>&quot;top&quot;</span><span>,</span><span> </span><span>legend.box</span><span> </span><span>=</span><span> </span><span>&quot;horizontal&quot;</span><span>,</span><span>
</span><span>legend.box.background</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>colour</span><span> </span><span>=</span><span> </span><span>&quot;grey50&quot;</span><span>),</span><span>
</span><span>legend.background</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>panel.border</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>NA</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>0.5</span><span>)</span><span>
</span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span># plotting raw data
</span><span>
</span><span>ggplot</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>fluH7N9.china.2013_gather</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>Date</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>age</span><span>),</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>outcome</span><span>))</span><span> </span><span>+</span><span>
</span><span>stat_density2d</span><span>(</span><span>aes</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>..level..</span><span>),</span><span> </span><span>geom</span><span> </span><span>=</span><span> </span><span>&quot;polygon&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_jitter</span><span>(</span><span>aes</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>outcome</span><span>,</span><span> </span><span>shape</span><span> </span><span>=</span><span> </span><span>gender</span><span>),</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1.5</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_rug</span><span>(</span><span>aes</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>outcome</span><span>))</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>
</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;Outcome&quot;</span><span>,</span><span>
</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;Outcome&quot;</span><span>,</span><span>
</span><span>alpha</span><span> </span><span>=</span><span> </span><span>&quot;Level&quot;</span><span>,</span><span>
</span><span>shape</span><span> </span><span>=</span><span> </span><span>&quot;Gender&quot;</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Date in 2013&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Age&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;2013 Influenza A H7N9 cases in China&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Dataset from 'outbreaks' package (Kucharski et al. 2014)&quot;</span><span>,</span><span>
</span><span>caption</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>
</span><span>)</span><span> </span><span>+</span><span>
</span><span>facet_grid</span><span>(</span><span>Group</span><span> </span><span>~</span><span> </span><span>province</span><span>)</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_shape_manual</span><span>(</span><span>values</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>15</span><span>,</span><span> </span><span>16</span><span>,</span><span> </span><span>17</span><span>))</span><span> </span><span>+</span><span>
</span><span>scale_color_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>,</span><span> </span><span>na.value</span><span> </span><span>=</span><span> </span><span>&quot;grey50&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post_files/figure-markdown_github/unnamed-chunk-4-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>This plot shows the dates of onset, hospitalisation and outcome (if known) of each data point. Outcome is marked by color and age shown on the y-axis. Gender is marked by point shape.</p>
<p>The density distribution of date by age for the cases seems to indicate that older people died more frequently in the Jiangsu and Zhejiang province than in Shanghai and in other provinces.</p>
<p>When we look at the distribution of points along the time axis, it suggests that their might be positive correlation between the likelihood of death and an early onset or early outcome.</p>
<p></p>
<p>I also want to know how many cases there are for each gender and province and compare the genders’ age distribution.</p>
<div>
<pre><code><span>fluH7N9.china.2013_gather_2</span><span> </span><span>&lt;-</span><span> </span><span>fluH7N9.china.2013_gather</span><span>[,</span><span> </span><span>-4</span><span>]</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>group_2</span><span>,</span><span> </span><span>value</span><span>,</span><span> </span><span>gender</span><span>:</span><span>province</span><span>)</span><span>
</span><span>fluH7N9.china.2013_gather_2</span><span>$</span><span>value</span><span> </span><span>&lt;-</span><span> </span><span>mapvalues</span><span>(</span><span>fluH7N9.china.2013_gather_2</span><span>$</span><span>value</span><span>,</span><span> </span><span>from</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;m&quot;</span><span>,</span><span> </span><span>&quot;f&quot;</span><span>,</span><span> </span><span>&quot;unknown&quot;</span><span>,</span><span> </span><span>&quot;Other&quot;</span><span>),</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Male&quot;</span><span>,</span><span> </span><span>&quot;Female&quot;</span><span>,</span><span> </span><span>&quot;Unknown gender&quot;</span><span>,</span><span> </span><span>&quot;Other province&quot;</span><span>))</span><span>
</span><span>fluH7N9.china.2013_gather_2</span><span>$</span><span>value</span><span> </span><span>&lt;-</span><span> </span><span>factor</span><span>(</span><span>fluH7N9.china.2013_gather_2</span><span>$</span><span>value</span><span>,</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Female&quot;</span><span>,</span><span> </span><span>&quot;Male&quot;</span><span>,</span><span> </span><span>&quot;Unknown gender&quot;</span><span>,</span><span> </span><span>&quot;Jiangsu&quot;</span><span>,</span><span> </span><span>&quot;Shanghai&quot;</span><span>,</span><span> </span><span>&quot;Zhejiang&quot;</span><span>,</span><span> </span><span>&quot;Other province&quot;</span><span>))</span><span>
</span><span>p</span><span>1</span><span> </span><span>&lt;-</span><span> </span><span>ggplot</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>fluH7N9.china.2013_gather_2</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>value</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>outcome</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>outcome</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;dodge&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.7</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>,</span><span> </span><span>na.value</span><span> </span><span>=</span><span> </span><span>&quot;grey50&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_color_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>,</span><span> </span><span>na.value</span><span> </span><span>=</span><span> </span><span>&quot;grey50&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>
</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Count&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;2013 Influenza A H7N9 cases in China&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Gender and Province numbers of flu cases&quot;</span><span>,</span><span>
</span><span>caption</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>
</span><span>)</span><span>
</span><span>p</span><span>2</span><span> </span><span>&lt;-</span><span> </span><span>ggplot</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>fluH7N9.china.2013_gather</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>age</span><span>),</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>outcome</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>outcome</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_density</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.3</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_rug</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_color_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>,</span><span> </span><span>na.value</span><span> </span><span>=</span><span> </span><span>&quot;grey50&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>,</span><span> </span><span>na.value</span><span> </span><span>=</span><span> </span><span>&quot;grey50&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>
</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Age&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Density&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Age distribution of flu cases&quot;</span><span>,</span><span>
</span><span>caption</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>
</span><span>)</span><span>
</span><span>library</span><span>(</span><span>gridExtra</span><span>)</span><span>
</span><span>library</span><span>(</span><span>grid</span><span>)</span><span>
</span><span>grid.arrange</span><span>(</span><span>p</span><span>1</span><span>,</span><span> </span><span>p</span><span>2</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post_files/figure-markdown_github/unnamed-chunk-5-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>In the dataset, there are more male than female cases and correspondingly, we see more deaths, recoveries and unknown outcomes in men than in women. This is potentially a problem later on for modeling because the inherent likelihoods for outcome are not directly comparable between the sexes.</p>
<p>Most unknown outcomes were recorded in Zhejiang. Similarly to gender, we don’t have an equal distribution of data points across provinces either.</p>
<p>When we look at the age distribution it is obvious that people who died tended to be slightly older than those who recovered. The density curve of unknown outcomes is more similar to that of death than of recovery, suggesting that among these people there might have been more deaths than recoveries.</p>
<p></p>
<p>And lastly, I want to plot how many days passed between onset, hospitalisation and outcome for each case.</p>
<div>
<pre><code><span>ggplot</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>fluH7N9.china.2013_gather</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>Date</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>age</span><span>),</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>outcome</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>(</span><span>aes</span><span>(</span><span>shape</span><span> </span><span>=</span><span> </span><span>gender</span><span>),</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1.5</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.6</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_path</span><span>(</span><span>aes</span><span>(</span><span>group</span><span> </span><span>=</span><span> </span><span>case.ID</span><span>))</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span> </span><span>~</span><span> </span><span>province</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_shape_manual</span><span>(</span><span>values</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>15</span><span>,</span><span> </span><span>16</span><span>,</span><span> </span><span>17</span><span>))</span><span> </span><span>+</span><span>
</span><span>scale_color_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>,</span><span> </span><span>na.value</span><span> </span><span>=</span><span> </span><span>&quot;grey50&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>
</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;Outcome&quot;</span><span>,</span><span>
</span><span>shape</span><span> </span><span>=</span><span> </span><span>&quot;Gender&quot;</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Date in 2013&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Age&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;2013 Influenza A H7N9 cases in China&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Dataset from 'outbreaks' package (Kucharski et al. 2014)&quot;</span><span>,</span><span>
</span><span>caption</span><span> </span><span>=</span><span> </span><span>&quot;\nTime from onset of flu to outcome.&quot;</span><span>
</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post_files/figure-markdown_github/unnamed-chunk-6-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>This plot shows that there are many missing values in the dates, so it is hard to draw a general conclusion.</p>
<p></p>
<h1>Features</h1>
<p>In Machine Learning-speak features are the variables used for model training. Using the right features dramatically influences the accuracy of the model.</p>
<p>Because we don’t have many features, I am keeping age as it is, but I am also generating new features:</p>
<ul>
<li>from the date information I am calculating the days between onset and outcome and between onset and hospitalisation</li>
<li>I am converting gender into numeric values with 1 for female and 0 for male</li>
<li>similarly, I am converting provinces to binary classifiers (yes == 1, no == 0) for Shanghai, Zhejiang, Jiangsu and other provinces</li>
<li>the same binary classification is given for whether a case was hospitalised, and whether they had an early onset or early outcome (earlier than the median date)</li>
</ul>
<div>
<pre><code><span># preparing the data frame for modeling
# </span><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>dataset</span><span> </span><span>&lt;-</span><span> </span><span>fluH7N9.china.2013</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>hospital</span><span> </span><span>=</span><span> </span><span>as.factor</span><span>(</span><span>ifelse</span><span>(</span><span>is.na</span><span>(</span><span>date.of.hospitalisation</span><span>),</span><span> </span><span>0</span><span>,</span><span> </span><span>1</span><span>)),</span><span>
</span><span>gender_f</span><span> </span><span>=</span><span> </span><span>as.factor</span><span>(</span><span>ifelse</span><span>(</span><span>gender</span><span> </span><span>==</span><span> </span><span>&quot;f&quot;</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>0</span><span>)),</span><span>
</span><span>province_Jiangsu</span><span> </span><span>=</span><span> </span><span>as.factor</span><span>(</span><span>ifelse</span><span>(</span><span>province</span><span> </span><span>==</span><span> </span><span>&quot;Jiangsu&quot;</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>0</span><span>)),</span><span>
</span><span>province_Shanghai</span><span> </span><span>=</span><span> </span><span>as.factor</span><span>(</span><span>ifelse</span><span>(</span><span>province</span><span> </span><span>==</span><span> </span><span>&quot;Shanghai&quot;</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>0</span><span>)),</span><span>
</span><span>province_Zhejiang</span><span> </span><span>=</span><span> </span><span>as.factor</span><span>(</span><span>ifelse</span><span>(</span><span>province</span><span> </span><span>==</span><span> </span><span>&quot;Zhejiang&quot;</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>0</span><span>)),</span><span>
</span><span>province_other</span><span> </span><span>=</span><span> </span><span>as.factor</span><span>(</span><span>ifelse</span><span>(</span><span>province</span><span> </span><span>==</span><span> </span><span>&quot;Zhejiang&quot;</span><span> </span><span>|</span><span> </span><span>province</span><span> </span><span>==</span><span> </span><span>&quot;Jiangsu&quot;</span><span> </span><span>|</span><span> </span><span>province</span><span> </span><span>==</span><span> </span><span>&quot;Shanghai&quot;</span><span>,</span><span> </span><span>0</span><span>,</span><span> </span><span>1</span><span>)),</span><span>
</span><span>days_onset_to_outcome</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>as.character</span><span>(</span><span>gsub</span><span>(</span><span>&quot; days&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>as.Date</span><span>(</span><span>as.character</span><span>(</span><span>date.of.outcome</span><span>),</span><span> </span><span>format</span><span> </span><span>=</span><span> </span><span>&quot;%Y-%m-%d&quot;</span><span>)</span><span> </span><span>-</span><span> </span><span>as.Date</span><span>(</span><span>as.character</span><span>(</span><span>date.of.onset</span><span>),</span><span> </span><span>format</span><span> </span><span>=</span><span> </span><span>&quot;%Y-%m-%d&quot;</span><span>)))),</span><span>
</span><span>days_onset_to_hospital</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>as.character</span><span>(</span><span>gsub</span><span>(</span><span>&quot; days&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>as.Date</span><span>(</span><span>as.character</span><span>(</span><span>date.of.hospitalisation</span><span>),</span><span> </span><span>format</span><span> </span><span>=</span><span> </span><span>&quot;%Y-%m-%d&quot;</span><span>)</span><span> </span><span>-</span><span> </span><span>as.Date</span><span>(</span><span>as.character</span><span>(</span><span>date.of.onset</span><span>),</span><span> </span><span>format</span><span> </span><span>=</span><span> </span><span>&quot;%Y-%m-%d&quot;</span><span>)))),</span><span>
</span><span>age</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>as.character</span><span>(</span><span>age</span><span>)),</span><span>
</span><span>early_onset</span><span> </span><span>=</span><span> </span><span>as.factor</span><span>(</span><span>ifelse</span><span>(</span><span>date.of.onset</span><span> </span><span>&lt;</span><span> </span><span>summary</span><span>(</span><span>fluH7N9.china.2013</span><span>$</span><span>date.of.onset</span><span>)[[</span><span>3</span><span>]],</span><span> </span><span>1</span><span>,</span><span> </span><span>0</span><span>)),</span><span>
</span><span>early_outcome</span><span> </span><span>=</span><span> </span><span>as.factor</span><span>(</span><span>ifelse</span><span>(</span><span>date.of.outcome</span><span> </span><span>&lt;</span><span> </span><span>summary</span><span>(</span><span>fluH7N9.china.2013</span><span>$</span><span>date.of.outcome</span><span>)[[</span><span>3</span><span>]],</span><span> </span><span>1</span><span>,</span><span> </span><span>0</span><span>)))</span><span> </span><span>%>%</span><span>
</span><span>subset</span><span>(</span><span>select</span><span> </span><span>=</span><span> </span><span>-</span><span>c</span><span>(</span><span>2</span><span>:</span><span>4</span><span>,</span><span> </span><span>6</span><span>,</span><span> </span><span>8</span><span>))</span><span>
</span><span>rownames</span><span>(</span><span>dataset</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>dataset</span><span>$</span><span>case.ID</span><span>
</span><span>dataset</span><span> </span><span>&lt;-</span><span> </span><span>dataset</span><span>[,</span><span> </span><span>-1</span><span>]</span><span>
</span><span>head</span><span>(</span><span>dataset</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##        outcome age hospital gender_f province_Jiangsu province_Shanghai province_Zhejiang province_other days_onset_to_outcome days_onset_to_hospital early_onset early_outcome
## case_1   Death  87        0        0                0                 1                 0              0                    13                     NA           1             1
## case_2   Death  27        1        0                0                 1                 0              0                    11                      4           1             1
## case_3   Death  35        1        1                0                 0                 0              1                    31                     10           1             1
## case_4    &lt;NA>  45        1        1                1                 0                 0              0                    NA                      8           1          &lt;NA>
## case_5 Recover  48        1        1                1                 0                 0              0                    57                     11           1             0
## case_6   Death  32        1        1                1                 0                 0              0                    36                      7           1             1
</code></pre>
</div>
<p></p>
<h2>Imputing missing values</h2>
<p>When looking at the dataset I created for modeling, it is obvious that we have quite a few missing values.</p>
<p>The missing values from the outcome column are what I want to predict but for the rest I would either have to remove the entire row from the data or impute the missing information. I decided to try the latter with the <a target="_blank" rel="nofollow" href="https://www.r-bloggers.com/imputing-missing-data-with-r-mice-package">mice</a> package.</p>
<div>
<pre><code><span># impute missing data
</span><span>
</span><span>library</span><span>(</span><span>mice</span><span>)</span><span>
</span><span>dataset_impute</span><span> </span><span>&lt;-</span><span> </span><span>mice</span><span>(</span><span>dataset</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>print</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>dataset_impute</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Multiply imputed dataset
## Call:
## mice(data = dataset[, -1], printFlag = FALSE)
## Number of multiple imputations:  5
## Missing cells per column:
##                    age               hospital               gender_f       province_Jiangsu      province_Shanghai      province_Zhejiang         province_other  days_onset_to_outcome days_onset_to_hospital            early_onset          early_outcome 
##                      2                      0                      2                      0                      0                      0                      0                     67                     74                     10                     65 
## Imputation methods:
##                    age               hospital               gender_f       province_Jiangsu      province_Shanghai      province_Zhejiang         province_other  days_onset_to_outcome days_onset_to_hospital            early_onset          early_outcome 
##                  &quot;pmm&quot;                     &quot;&quot;               &quot;logreg&quot;                     &quot;&quot;                     &quot;&quot;                     &quot;&quot;                     &quot;&quot;                  &quot;pmm&quot;                  &quot;pmm&quot;               &quot;logreg&quot;               &quot;logreg&quot; 
## VisitSequence:
##                    age               gender_f  days_onset_to_outcome days_onset_to_hospital            early_onset          early_outcome 
##                      1                      3                      8                      9                     10                     11 
## PredictorMatrix:
##                        age hospital gender_f province_Jiangsu province_Shanghai province_Zhejiang province_other days_onset_to_outcome days_onset_to_hospital early_onset early_outcome
## age                      0        1        1                1                 1                 1              1                     1                      1           1             1
## hospital                 0        0        0                0                 0                 0              0                     0                      0           0             0
## gender_f                 1        1        0                1                 1                 1              1                     1                      1           1             1
## province_Jiangsu         0        0        0                0                 0                 0              0                     0                      0           0             0
## province_Shanghai        0        0        0                0                 0                 0              0                     0                      0           0             0
## province_Zhejiang        0        0        0                0                 0                 0              0                     0                      0           0             0
## province_other           0        0        0                0                 0                 0              0                     0                      0           0             0
## days_onset_to_outcome    1        1        1                1                 1                 1              1                     0                      1           1             1
## days_onset_to_hospital   1        1        1                1                 1                 1              1                     1                      0           1             1
## early_onset              1        1        1                1                 1                 1              1                     1                      1           0             1
## early_outcome            1        1        1                1                 1                 1              1                     1                      1           1             0
## Random generator seed value:  NA
</code></pre>
</div>
<div>
<pre><code><span># recombine imputed data frame with the outcome column
</span><span>
</span><span>dataset_complete</span><span> </span><span>&lt;-</span><span> </span><span>merge</span><span>(</span><span>dataset</span><span>[,</span><span> </span><span>1</span><span>,</span><span> </span><span>drop</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>],</span><span> </span><span>mice</span><span>::</span><span>complete</span><span>(</span><span>dataset_impute</span><span>,</span><span> </span><span>1</span><span>),</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;row.names&quot;</span><span>,</span><span> </span><span>all</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>rownames</span><span>(</span><span>dataset_complete</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>dataset_complete</span><span>$</span><span>Row.names</span><span>
</span><span>dataset_complete</span><span> </span><span>&lt;-</span><span> </span><span>dataset_complete</span><span>[,</span><span> </span><span>-1</span><span>]</span><span>
</span></code></pre>
</div>
<p></p>
<h1>Test, train and validation datasets</h1>
<p>For building the model, I am separating the imputed data frame into training and test data. Test data are the 57 cases with unknown outcome.</p>
<div>
<pre><code><span>summary</span><span>(</span><span>dataset</span><span>$</span><span>outcome</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##   Death Recover    NA's 
##      32      47      57
</code></pre>
</div>
<p></p>
<p>The training data will be further devided for validation of the models: 70% of the training data will be kept for model building and the remaining 30% will be used for model testing.</p>
<p>I am using the <a target="_blank" rel="nofollow" href="http://topepo.github.io/caret/index.html">caret</a> package for modeling.</p>
<div>
<pre><code><span>train_index</span><span> </span><span>&lt;-</span><span> </span><span>which</span><span>(</span><span>is.na</span><span>(</span><span>dataset_complete</span><span>$</span><span>outcome</span><span>))</span><span>
</span><span>train_data</span><span> </span><span>&lt;-</span><span> </span><span>dataset_complete</span><span>[</span><span>-</span><span>train_index</span><span>,</span><span> </span><span>]</span><span>
</span><span>test_data</span><span> </span><span>&lt;-</span><span> </span><span>dataset_complete</span><span>[</span><span>train_index</span><span>,</span><span> </span><span>-1</span><span>]</span><span>
</span><span>library</span><span>(</span><span>caret</span><span>)</span><span>
</span><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>val_index</span><span> </span><span>&lt;-</span><span> </span><span>createDataPartition</span><span>(</span><span>train_data</span><span>$</span><span>outcome</span><span>,</span><span> </span><span>p</span><span> </span><span>=</span><span> </span><span>0.7</span><span>,</span><span> </span><span>list</span><span>=</span><span>FALSE</span><span>)</span><span>
</span><span>val_train_data</span><span> </span><span>&lt;-</span><span> </span><span>train_data</span><span>[</span><span>val_index</span><span>,</span><span> </span><span>]</span><span>
</span><span>val_test_data</span><span> </span><span>&lt;-</span><span> </span><span>train_data</span><span>[</span><span>-</span><span>val_index</span><span>,</span><span> </span><span>]</span><span>
</span><span>val_train_X</span><span> </span><span>&lt;-</span><span> </span><span>val_train_data</span><span>[,</span><span>-1</span><span>]</span><span>
</span><span>val_test_X</span><span> </span><span>&lt;-</span><span> </span><span>val_test_data</span><span>[,</span><span>-1</span><span>]</span><span>
</span></code></pre>
</div>
<p></p>
<h2>Decision trees</h2>
<p>To get an idea about how each feature contributes to the prediction of the outcome, I first built a decision tree based on the training data using <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/rpart/rpart.pdf">rpart</a> and <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/rattle/vignettes/rattle.pdf">rattle</a>.</p>
<div>
<pre><code><span>library</span><span>(</span><span>rpart</span><span>)</span><span>
</span><span>library</span><span>(</span><span>rattle</span><span>)</span><span>
</span><span>library</span><span>(</span><span>rpart.plot</span><span>)</span><span>
</span><span>library</span><span>(</span><span>RColorBrewer</span><span>)</span><span>
</span><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>fit</span><span> </span><span>&lt;-</span><span> </span><span>rpart</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;class&quot;</span><span>,</span><span>
</span><span>control</span><span> </span><span>=</span><span> </span><span>rpart.control</span><span>(</span><span>xval</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>minbucket</span><span> </span><span>=</span><span> </span><span>2</span><span>,</span><span> </span><span>cp</span><span> </span><span>=</span><span> </span><span>0</span><span>),</span><span> </span><span>parms</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>split</span><span> </span><span>=</span><span> </span><span>&quot;information&quot;</span><span>))</span><span>
</span><span>fancyRpartPlot</span><span>(</span><span>fit</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post_files/figure-markdown_github/unnamed-chunk-12-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>This randomly generated decision tree shows that cases with an early outcome were most likely to die when they were 68 or older, when they also had an early onset and when they were sick for fewer than 13 days. If a person was not among the first cases and was younger than 52, they had a good chance of recovering, but if they were 82 or older, they were more likely to die from the flu.</p>
<p></p>
<h2>Feature Importance</h2>
<p>Not all of the features I created will be equally important to the model. The decision tree already gave me an idea of which features might be most important but I also want to estimate feature importance using a Random Forest approach with repeated cross validation.</p>
<div>
<pre><code><span># prepare training scheme
</span><span>control</span><span> </span><span>&lt;-</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>)</span><span>
</span><span># train the model
</span><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;rf&quot;</span><span>,</span><span> </span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span> </span><span>trControl</span><span> </span><span>=</span><span> </span><span>control</span><span>)</span><span>
</span><span># estimate variable importance
</span><span>importance</span><span> </span><span>&lt;-</span><span> </span><span>varImp</span><span>(</span><span>model</span><span>,</span><span> </span><span>scale</span><span>=</span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span># prepare for plotting
</span><span>importance_df_1</span><span> </span><span>&lt;-</span><span> </span><span>importance</span><span>$</span><span>importance</span><span>
</span><span>importance_df_1</span><span>$</span><span>group</span><span> </span><span>&lt;-</span><span> </span><span>rownames</span><span>(</span><span>importance_df_1</span><span>)</span><span>
</span><span>importance_df_1</span><span>$</span><span>group</span><span> </span><span>&lt;-</span><span> </span><span>mapvalues</span><span>(</span><span>importance_df_1</span><span>$</span><span>group</span><span>,</span><span> </span><span>from</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;age&quot;</span><span>,</span><span> </span><span>&quot;hospital2&quot;</span><span>,</span><span> </span><span>&quot;gender_f2&quot;</span><span>,</span><span> </span><span>&quot;province_Jiangsu2&quot;</span><span>,</span><span> </span><span>&quot;province_Shanghai2&quot;</span><span>,</span><span> </span><span>&quot;province_Zhejiang2&quot;</span><span>,</span><span>
</span><span>&quot;province_other2&quot;</span><span>,</span><span> </span><span>&quot;days_onset_to_outcome&quot;</span><span>,</span><span> </span><span>&quot;days_onset_to_hospital&quot;</span><span>,</span><span> </span><span>&quot;early_onset2&quot;</span><span>,</span><span> </span><span>&quot;early_outcome2&quot;</span><span> </span><span>),</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Age&quot;</span><span>,</span><span> </span><span>&quot;Hospital&quot;</span><span>,</span><span> </span><span>&quot;Female&quot;</span><span>,</span><span> </span><span>&quot;Jiangsu&quot;</span><span>,</span><span> </span><span>&quot;Shanghai&quot;</span><span>,</span><span> </span><span>&quot;Zhejiang&quot;</span><span>,</span><span>
</span><span>&quot;Other province&quot;</span><span>,</span><span> </span><span>&quot;Days onset to outcome&quot;</span><span>,</span><span> </span><span>&quot;Days onset to hospital&quot;</span><span>,</span><span> </span><span>&quot;Early onset&quot;</span><span>,</span><span> </span><span>&quot;Early outcome&quot;</span><span> </span><span>))</span><span>
</span><span>f</span><span> </span><span>=</span><span> </span><span>importance_df_1</span><span>[</span><span>order</span><span>(</span><span>importance_df_1</span><span>$</span><span>Overall</span><span>,</span><span> </span><span>decreasing</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>),</span><span> </span><span>&quot;group&quot;</span><span>]</span><span>
</span><span>importance_df_2</span><span> </span><span>&lt;-</span><span> </span><span>importance_df_1</span><span>
</span><span>importance_df_2</span><span>$</span><span>Overall</span><span> </span><span>&lt;-</span><span> </span><span>0</span><span>
</span><span>importance_df</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>importance_df_1</span><span>,</span><span> </span><span>importance_df_2</span><span>)</span><span>
</span><span># setting factor levels
</span><span>importance_df</span><span> </span><span>&lt;-</span><span> </span><span>within</span><span>(</span><span>importance_df</span><span>,</span><span> </span><span>group</span><span> </span><span>&lt;-</span><span> </span><span>factor</span><span>(</span><span>group</span><span>,</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>f</span><span>))</span><span>
</span><span>importance_df_1</span><span> </span><span>&lt;-</span><span> </span><span>within</span><span>(</span><span>importance_df_1</span><span>,</span><span> </span><span>group</span><span> </span><span>&lt;-</span><span> </span><span>factor</span><span>(</span><span>group</span><span>,</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>f</span><span>))</span><span>
</span><span>ggplot</span><span>()</span><span> </span><span>+</span><span>
</span><span>geom_point</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>importance_df_1</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>Overall</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>group</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>group</span><span>),</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_path</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>importance_df</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>Overall</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>group</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>group</span><span>,</span><span> </span><span>group</span><span> </span><span>=</span><span> </span><span>group</span><span>),</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_color_manual</span><span>(</span><span>values</span><span> </span><span>=</span><span> </span><span>rep</span><span>(</span><span>brewer.pal</span><span>(</span><span>1</span><span>,</span><span> </span><span>&quot;Set1&quot;</span><span>)[</span><span>1</span><span>],</span><span> </span><span>11</span><span>))</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>legend.position</span><span> </span><span>=</span><span> </span><span>&quot;none&quot;</span><span>,</span><span>
</span><span>axis.text.x</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>angle</span><span> </span><span>=</span><span> </span><span>0</span><span>,</span><span> </span><span>vjust</span><span> </span><span>=</span><span> </span><span>0.5</span><span>,</span><span> </span><span>hjust</span><span> </span><span>=</span><span> </span><span>0.5</span><span>))</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Importance&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;2013 Influenza A H7N9 cases in China&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Scaled feature importance&quot;</span><span>,</span><span>
</span><span>caption</span><span> </span><span>=</span><span> </span><span>&quot;\nDetermined with Random Forest and
repeated cross validation (10 repeats, 10 times)&quot;</span><span>
</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post_files/figure-markdown_github/unnamed-chunk-14-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>This tells me that age is the most important determining factor for predicting disease outcome, followed by days between onset an outcome, early outcome and days between onset and hospitalisation.</p>
<p></p>
<h2>Feature Plot</h2>
<p>Before I start actually building models, I want to check whether the distribution of feature values is comparable between training, validation and test datasets.</p>
<div>
<pre><code><span># prepare for plotting
</span><span>dataset_complete_gather</span><span> </span><span>&lt;-</span><span> </span><span>dataset_complete</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>set</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>rownames</span><span>(</span><span>dataset_complete</span><span>)</span><span> </span><span>%in%</span><span> </span><span>rownames</span><span>(</span><span>test_data</span><span>),</span><span> </span><span>&quot;Test Data&quot;</span><span>,</span><span> </span><span>ifelse</span><span>(</span><span>rownames</span><span>(</span><span>dataset_complete</span><span>)</span><span> </span><span>%in%</span><span> </span><span>rownames</span><span>(</span><span>val_train_data</span><span>),</span><span> </span><span>&quot;Validation Train Data&quot;</span><span>,</span><span>
</span><span>ifelse</span><span>(</span><span>rownames</span><span>(</span><span>dataset_complete</span><span>)</span><span> </span><span>%in%</span><span> </span><span>rownames</span><span>(</span><span>val_test_data</span><span>),</span><span> </span><span>&quot;Validation Test Data&quot;</span><span>,</span><span> </span><span>&quot;NA&quot;</span><span>))),</span><span>
</span><span>case_ID</span><span> </span><span>=</span><span> </span><span>rownames</span><span>(</span><span>.</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>group</span><span>,</span><span> </span><span>value</span><span>,</span><span> </span><span>age</span><span>:</span><span>early_outcome</span><span>)</span><span>
</span><span>dataset_complete_gather</span><span>$</span><span>group</span><span> </span><span>&lt;-</span><span> </span><span>mapvalues</span><span>(</span><span>dataset_complete_gather</span><span>$</span><span>group</span><span>,</span><span> </span><span>from</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;age&quot;</span><span>,</span><span> </span><span>&quot;hospital&quot;</span><span>,</span><span> </span><span>&quot;gender_f&quot;</span><span>,</span><span> </span><span>&quot;province_Jiangsu&quot;</span><span>,</span><span> </span><span>&quot;province_Shanghai&quot;</span><span>,</span><span> </span><span>&quot;province_Zhejiang&quot;</span><span>,</span><span>
</span><span>&quot;province_other&quot;</span><span>,</span><span> </span><span>&quot;days_onset_to_outcome&quot;</span><span>,</span><span> </span><span>&quot;days_onset_to_hospital&quot;</span><span>,</span><span> </span><span>&quot;early_onset&quot;</span><span>,</span><span> </span><span>&quot;early_outcome&quot;</span><span> </span><span>),</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Age&quot;</span><span>,</span><span> </span><span>&quot;Hospital&quot;</span><span>,</span><span> </span><span>&quot;Female&quot;</span><span>,</span><span> </span><span>&quot;Jiangsu&quot;</span><span>,</span><span> </span><span>&quot;Shanghai&quot;</span><span>,</span><span> </span><span>&quot;Zhejiang&quot;</span><span>,</span><span>
</span><span>&quot;Other province&quot;</span><span>,</span><span> </span><span>&quot;Days onset to outcome&quot;</span><span>,</span><span> </span><span>&quot;Days onset to hospital&quot;</span><span>,</span><span> </span><span>&quot;Early onset&quot;</span><span>,</span><span> </span><span>&quot;Early outcome&quot;</span><span> </span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>ggplot</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>dataset_complete_gather</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>value</span><span>),</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>outcome</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>outcome</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_density</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.2</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_rug</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_color_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>,</span><span> </span><span>na.value</span><span> </span><span>=</span><span> </span><span>&quot;grey50&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>,</span><span> </span><span>na.value</span><span> </span><span>=</span><span> </span><span>&quot;grey50&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>set</span><span> </span><span>~</span><span> </span><span>group</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>11</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Value&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Density&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;2013 Influenza A H7N9 cases in China&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Features for classifying outcome&quot;</span><span>,</span><span>
</span><span>caption</span><span> </span><span>=</span><span> </span><span>&quot;\nDensity distribution of all features used for classification of flu outcome.&quot;</span><span>
</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>ggplot</span><span>(</span><span>subset</span><span>(</span><span>dataset_complete_gather</span><span>,</span><span> </span><span>group</span><span> </span><span>==</span><span> </span><span>&quot;Age&quot;</span><span> </span><span>|</span><span> </span><span>group</span><span> </span><span>==</span><span> </span><span>&quot;Days onset to hospital&quot;</span><span> </span><span>|</span><span> </span><span>group</span><span> </span><span>==</span><span> </span><span>&quot;Days onset to outcome&quot;</span><span>),</span><span> </span><span>aes</span><span>(</span><span>x</span><span>=</span><span>outcome</span><span>,</span><span> </span><span>y</span><span>=</span><span>as.numeric</span><span>(</span><span>value</span><span>),</span><span> </span><span>fill</span><span>=</span><span>set</span><span>))</span><span> </span><span>+</span><span> </span><span>geom_boxplot</span><span>()</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;div &quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span> </span><span>~</span><span> </span><span>group</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>
</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Outcome&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Value&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;2013 Influenza A H7N9 cases in China&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Features for classifying outcome&quot;</span><span>,</span><span>
</span><span>caption</span><span> </span><span>=</span><span> </span><span>&quot;\nBoxplot of the features age, days from onset to hospitalisation and days from onset to outcome.&quot;</span><span>
</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post_files/figure-markdown_github/unnamed-chunk-16-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post_files/figure-markdown_github/unnamed-chunk-17-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Luckily, the distributions looks reasonably similar between the validation and test data, except for a few outliers.</p>
<p></p>
<hr>
<h1>Comparing Machine Learning algorithms</h1>
<p>Before I try to predict the outcome of the unknown cases, I am testing the models’ accuracy with the validation datasets on a couple of algorithms. I have chosen only a few more well known algorithms, but <a target="_blank" rel="nofollow" href="http://topepo.github.io/caret/index.html">caret</a> implements many more.</p>
<p></p>
<h2>Random Forest</h2>
<p><a target="_blank" rel="nofollow" href="https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm">Random Forests</a> predictions are based on the generation of multiple classification trees.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_rf</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>val_train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;rf&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_rf</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Random Forest 
## 
## 56 samples
## 11 predictors
##  2 classes: 'Death', 'Recover' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 10 times) 
## Summary of sample sizes: 50, 50, 51, 51, 51, 50, ... 
## Resampling results across tuning parameters:
## 
##   mtry  Accuracy   Kappa    
##    2    0.7801905  0.5338408
##    6    0.7650952  0.4926366
##   11    0.7527619  0.4638073
## 
## Accuracy was used to select the optimal model using  the largest value.
## The final value used for the model was mtry = 2.
</code></pre>
</div>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_rf</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>]),</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Death Recover
##    Death       4       4
##    Recover     5      10
##                                           
##                Accuracy : 0.6087          
##                  95% CI : (0.3854, 0.8029)
##     No Information Rate : 0.6087          
##     P-Value [Acc > NIR] : 0.5901          
##                                           
##                   Kappa : 0.1619          
##  Mcnemar's Test P-Value : 1.0000          
##                                           
##             Sensitivity : 0.4444          
##             Specificity : 0.7143          
##          Pos Pred Value : 0.5000          
##          Neg Pred Value : 0.6667          
##              Prevalence : 0.3913          
##          Detection Rate : 0.1739          
##    Detection Prevalence : 0.3478          
##       Balanced Accuracy : 0.5794          
##                                           
##        'Positive' Class : Death           
## 
</code></pre>
</div>
<p></p>
<h2>Elastic-Net Regularized Generalized Linear Models</h2>
<p><a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Elastic_net_regularization">Lasso or elastic net regularization for generalized linear model regression</a> are based on linear regression models and is useful when we have feature correlation in our model.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_glmnet</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>val_train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;glmnet&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_glmnet</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## glmnet 
## 
## 56 samples
## 11 predictors
##  2 classes: 'Death', 'Recover' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 10 times) 
## Summary of sample sizes: 50, 50, 51, 51, 51, 50, ... 
## Resampling results across tuning parameters:
## 
##   alpha  lambda        Accuracy   Kappa    
##   0.10   0.0005154913  0.7211905  0.4218952
##   0.10   0.0051549131  0.7189524  0.4189835
##   0.10   0.0515491312  0.7469524  0.4704687
##   0.55   0.0005154913  0.7211905  0.4218952
##   0.55   0.0051549131  0.7236190  0.4280528
##   0.55   0.0515491312  0.7531905  0.4801031
##   1.00   0.0005154913  0.7228571  0.4245618
##   1.00   0.0051549131  0.7278571  0.4361809
##   1.00   0.0515491312  0.7678571  0.5094194
## 
## Accuracy was used to select the optimal model using  the largest value.
## The final values used for the model were alpha = 1 and lambda = 0.05154913.
</code></pre>
</div>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_glmnet</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>]),</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Death Recover
##    Death       3       4
##    Recover     6      10
##                                           
##                Accuracy : 0.5652          
##                  95% CI : (0.3449, 0.7681)
##     No Information Rate : 0.6087          
##     P-Value [Acc > NIR] : 0.7418          
##                                           
##                   Kappa : 0.0496          
##  Mcnemar's Test P-Value : 0.7518          
##                                           
##             Sensitivity : 0.3333          
##             Specificity : 0.7143          
##          Pos Pred Value : 0.4286          
##          Neg Pred Value : 0.6250          
##              Prevalence : 0.3913          
##          Detection Rate : 0.1304          
##    Detection Prevalence : 0.3043          
##       Balanced Accuracy : 0.5238          
##                                           
##        'Positive' Class : Death           
## 
</code></pre>
</div>
<p></p>
<h2>k-Nearest Neighbors</h2>
<p><a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm">k-nearest neighbors</a> predicts based on point distances with predefined constants.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_kknn</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>val_train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;kknn&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_kknn</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## k-Nearest Neighbors 
## 
## 56 samples
## 11 predictors
##  2 classes: 'Death', 'Recover' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 10 times) 
## Summary of sample sizes: 50, 50, 51, 51, 51, 50, ... 
## Resampling results across tuning parameters:
## 
##   kmax  Accuracy   Kappa    
##   5     0.6531905  0.2670442
##   7     0.7120476  0.4031836
##   9     0.7106190  0.4004564
## 
## Tuning parameter 'distance' was held constant at a value of 2
## Tuning parameter 'kernel' was held constant at a value of optimal
## Accuracy was used to select the optimal model using  the largest value.
## The final values used for the model were kmax = 7, distance = 2 and kernel = optimal.
</code></pre>
</div>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_kknn</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>]),</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Death Recover
##    Death       3       3
##    Recover     6      11
##                                           
##                Accuracy : 0.6087          
##                  95% CI : (0.3854, 0.8029)
##     No Information Rate : 0.6087          
##     P-Value [Acc > NIR] : 0.5901          
##                                           
##                   Kappa : 0.1266          
##  Mcnemar's Test P-Value : 0.5050          
##                                           
##             Sensitivity : 0.3333          
##             Specificity : 0.7857          
##          Pos Pred Value : 0.5000          
##          Neg Pred Value : 0.6471          
##              Prevalence : 0.3913          
##          Detection Rate : 0.1304          
##    Detection Prevalence : 0.2609          
##       Balanced Accuracy : 0.5595          
##                                           
##        'Positive' Class : Death           
## 
</code></pre>
</div>
<p></p>
<h2>Penalized Discriminant Analysis</h2>
<p><a target="_blank" rel="nofollow" href="https://web.stanford.edu/~hastie/Papers/pda.pdf">Penalized Discriminant Analysis</a> is the penalized linear discriminant analysis and is also useful for when we have highly correlated features.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_pda</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>val_train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;pda&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_pda</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Penalized Discriminant Analysis 
## 
## 56 samples
## 11 predictors
##  2 classes: 'Death', 'Recover' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 10 times) 
## Summary of sample sizes: 50, 50, 51, 51, 51, 50, ... 
## Resampling results across tuning parameters:
## 
##   lambda  Accuracy   Kappa    
##   0e+00         NaN        NaN
##   1e-04   0.7255238  0.4373766
##   1e-01   0.7235238  0.4342554
## 
## Accuracy was used to select the optimal model using  the largest value.
## The final value used for the model was lambda = 1e-04.
</code></pre>
</div>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_pda</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>]),</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Death Recover
##    Death       4       4
##    Recover     5      10
##                                           
##                Accuracy : 0.6087          
##                  95% CI : (0.3854, 0.8029)
##     No Information Rate : 0.6087          
##     P-Value [Acc > NIR] : 0.5901          
##                                           
##                   Kappa : 0.1619          
##  Mcnemar's Test P-Value : 1.0000          
##                                           
##             Sensitivity : 0.4444          
##             Specificity : 0.7143          
##          Pos Pred Value : 0.5000          
##          Neg Pred Value : 0.6667          
##              Prevalence : 0.3913          
##          Detection Rate : 0.1739          
##    Detection Prevalence : 0.3478          
##       Balanced Accuracy : 0.5794          
##                                           
##        'Positive' Class : Death           
## 
</code></pre>
</div>
<p></p>
<h2>Stabilized Linear Discriminant Analysis</h2>
<p><a target="_blank" rel="nofollow" href="https://books.google.de/books?id=RYaMCwAAQBAJ&pg=PA89&lpg=PA89&dq=%22Stabilized+Linear+Discriminant+Analysis%22&source=bl&ots=YwY0mLEeXx&sig=74Uf3plf0Ma8CT1vh64Wc9MzFqI&hl=de&sa=X&ved=0ahUKEwjXgsOh7sPQAhUMBsAKHc_mAw4Q6AEINjAD#v=onepage&q=%22Stabilized%20Linear%20Discriminant%20Analysis%22&f=false">Stabilized Linear Discriminant Analysis</a> is designed for high-dimensional data and correlated co-variables.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_slda</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>val_train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;slda&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_slda</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Stabilized Linear Discriminant Analysis 
## 
## 56 samples
## 11 predictors
##  2 classes: 'Death', 'Recover' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 10 times) 
## Summary of sample sizes: 50, 50, 51, 51, 51, 50, ... 
## Resampling results:
## 
##   Accuracy   Kappa    
##   0.6886667  0.3512234
## 
## 
</code></pre>
</div>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_slda</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>]),</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Death Recover
##    Death       3       2
##    Recover     6      12
##                                           
##                Accuracy : 0.6522          
##                  95% CI : (0.4273, 0.8362)
##     No Information Rate : 0.6087          
##     P-Value [Acc > NIR] : 0.4216          
##                                           
##                   Kappa : 0.2069          
##  Mcnemar's Test P-Value : 0.2888          
##                                           
##             Sensitivity : 0.3333          
##             Specificity : 0.8571          
##          Pos Pred Value : 0.6000          
##          Neg Pred Value : 0.6667          
##              Prevalence : 0.3913          
##          Detection Rate : 0.1304          
##    Detection Prevalence : 0.2174          
##       Balanced Accuracy : 0.5952          
##                                           
##        'Positive' Class : Death           
## 
</code></pre>
</div>
<p></p>
<h2>Nearest Shrunken Centroids</h2>
<p><a target="_blank" rel="nofollow" href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC124443/">Nearest Shrunken Centroids</a> computes a standardized centroid for each class and shrinks each centroid toward the overall centroid for all classes.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_pam</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>val_train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;pam&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_pam</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Nearest Shrunken Centroids 
## 
## 56 samples
## 11 predictors
##  2 classes: 'Death', 'Recover' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 10 times) 
## Summary of sample sizes: 50, 50, 51, 51, 51, 50, ... 
## Resampling results across tuning parameters:
## 
##   threshold  Accuracy   Kappa    
##   0.1200215  0.7283333  0.4418904
##   1.7403111  0.6455714  0.2319674
##   3.3606007  0.5904762  0.0000000
## 
## Accuracy was used to select the optimal model using  the largest value.
## The final value used for the model was threshold = 0.1200215.
</code></pre>
</div>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_pam</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>]),</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Death Recover
##    Death       4       3
##    Recover     5      11
##                                           
##                Accuracy : 0.6522          
##                  95% CI : (0.4273, 0.8362)
##     No Information Rate : 0.6087          
##     P-Value [Acc > NIR] : 0.4216          
##                                           
##                   Kappa : 0.2397          
##  Mcnemar's Test P-Value : 0.7237          
##                                           
##             Sensitivity : 0.4444          
##             Specificity : 0.7857          
##          Pos Pred Value : 0.5714          
##          Neg Pred Value : 0.6875          
##              Prevalence : 0.3913          
##          Detection Rate : 0.1739          
##    Detection Prevalence : 0.3043          
##       Balanced Accuracy : 0.6151          
##                                           
##        'Positive' Class : Death           
## 
</code></pre>
</div>
<p></p>
<h2>Single C5.0 Tree</h2>
<p><a target="_blank" rel="nofollow" href="http://www.bowdoin.edu/~echown/courses/370/tutorial.html">C5.0</a> is another tree-based modeling algorithm.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_C5.0Tree</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>val_train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;C5.0Tree&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_C5.0Tree</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Single C5.0 Tree 
## 
## 56 samples
## 11 predictors
##  2 classes: 'Death', 'Recover' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 10 times) 
## Summary of sample sizes: 50, 50, 51, 51, 51, 50, ... 
## Resampling results:
## 
##   Accuracy   Kappa    
##   0.7103333  0.4047334
## 
## 
</code></pre>
</div>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_C5.0Tree</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>]),</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Death Recover
##    Death       5       4
##    Recover     4      10
##                                           
##                Accuracy : 0.6522          
##                  95% CI : (0.4273, 0.8362)
##     No Information Rate : 0.6087          
##     P-Value [Acc > NIR] : 0.4216          
##                                           
##                   Kappa : 0.2698          
##  Mcnemar's Test P-Value : 1.0000          
##                                           
##             Sensitivity : 0.5556          
##             Specificity : 0.7143          
##          Pos Pred Value : 0.5556          
##          Neg Pred Value : 0.7143          
##              Prevalence : 0.3913          
##          Detection Rate : 0.2174          
##    Detection Prevalence : 0.3913          
##       Balanced Accuracy : 0.6349          
##                                           
##        'Positive' Class : Death           
## 
</code></pre>
</div>
<p></p>
<h2>Partial Least Squares</h2>
<p><a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Partial_least_squares_regression">Partial least squares regression</a> combined principal component analysis and multiple regression and is useful for modeling with correlated features.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_pls</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>val_train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;pls&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_pls</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Partial Least Squares 
## 
## 56 samples
## 11 predictors
##  2 classes: 'Death', 'Recover' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold, repeated 10 times) 
## Summary of sample sizes: 50, 50, 51, 51, 51, 50, ... 
## Resampling results across tuning parameters:
## 
##   ncomp  Accuracy   Kappa    
##   1      0.6198571  0.2112982
##   2      0.6376190  0.2436222
##   3      0.6773810  0.3305780
## 
## Accuracy was used to select the optimal model using  the largest value.
## The final value used for the model was ncomp = 3.
</code></pre>
</div>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_pls</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>]),</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Death Recover
##    Death       3       2
##    Recover     6      12
##                                           
##                Accuracy : 0.6522          
##                  95% CI : (0.4273, 0.8362)
##     No Information Rate : 0.6087          
##     P-Value [Acc > NIR] : 0.4216          
##                                           
##                   Kappa : 0.2069          
##  Mcnemar's Test P-Value : 0.2888          
##                                           
##             Sensitivity : 0.3333          
##             Specificity : 0.8571          
##          Pos Pred Value : 0.6000          
##          Neg Pred Value : 0.6667          
##              Prevalence : 0.3913          
##          Detection Rate : 0.1304          
##    Detection Prevalence : 0.2174          
##       Balanced Accuracy : 0.5952          
##                                           
##        'Positive' Class : Death           
## 
</code></pre>
</div>
<p></p>
<h2>Comparing accuracy of models</h2>
<div>
<pre><code><span># Create a list of models
</span><span>models</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>rf</span><span> </span><span>=</span><span> </span><span>model_rf</span><span>,</span><span> </span><span>glmnet</span><span> </span><span>=</span><span> </span><span>model_glmnet</span><span>,</span><span> </span><span>kknn</span><span> </span><span>=</span><span> </span><span>model_kknn</span><span>,</span><span> </span><span>pda</span><span> </span><span>=</span><span> </span><span>model_pda</span><span>,</span><span> </span><span>slda</span><span> </span><span>=</span><span> </span><span>model_slda</span><span>,</span><span>
</span><span>pam</span><span> </span><span>=</span><span> </span><span>model_pam</span><span>,</span><span> </span><span>C</span><span>5.0</span><span>Tree</span><span> </span><span>=</span><span> </span><span>model_C5.0Tree</span><span>,</span><span> </span><span>pls</span><span> </span><span>=</span><span> </span><span>model_pls</span><span>)</span><span>
</span><span># Resample the models
</span><span>resample_results</span><span> </span><span>&lt;-</span><span> </span><span>resamples</span><span>(</span><span>models</span><span>)</span><span>
</span><span># Generate a summary
</span><span>summary</span><span>(</span><span>resample_results</span><span>,</span><span> </span><span>metric</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Kappa&quot;</span><span>,</span><span> </span><span>&quot;Accuracy&quot;</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## 
## Call:
## summary.resamples(object = resample_results, metric = c(&quot;Kappa&quot;, &quot;Accuracy&quot;))
## 
## Models: rf, glmnet, kknn, pda, slda, pam, C5.0Tree, pls 
## Number of resamples: 100 
## 
## Kappa 
##             Min. 1st Qu. Median   Mean 3rd Qu. Max. NA's
## rf       -0.3636  0.3214 0.5714 0.5338  0.6800    1    0
## glmnet   -0.3636  0.2768 0.5455 0.5094  0.6667    1    0
## kknn     -0.3636  0.1667 0.5035 0.4032  0.6282    1    0
## pda      -0.6667  0.2431 0.5455 0.4374  0.6667    1    0
## slda     -0.5217  0.0000 0.4308 0.3512  0.6667    1    0
## pam      -0.5000  0.2292 0.5455 0.4419  0.6667    1    0
## C5.0Tree -0.4286  0.1667 0.4167 0.4047  0.6154    1    0
## pls      -0.6667  0.0000 0.3333 0.3306  0.6282    1    0
## 
## Accuracy 
##            Min. 1st Qu. Median   Mean 3rd Qu. Max. NA's
## rf       0.4000  0.6667 0.8000 0.7802  0.8393    1    0
## glmnet   0.4000  0.6500 0.8000 0.7679  0.8333    1    0
## kknn     0.3333  0.6000 0.7571 0.7120  0.8333    1    0
## pda      0.2000  0.6000 0.8000 0.7255  0.8333    1    0
## slda     0.2000  0.6000 0.6905 0.6887  0.8333    1    0
## pam      0.2000  0.6000 0.8000 0.7283  0.8333    1    0
## C5.0Tree 0.2000  0.6000 0.7143 0.7103  0.8333    1    0
## pls      0.1429  0.5929 0.6667 0.6774  0.8333    1    0
</code></pre>
</div>
<div>
<pre><code><span>bwplot</span><span>(</span><span>resample_results</span><span> </span><span>,</span><span> </span><span>metric</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Kappa&quot;</span><span>,</span><span>&quot;Accuracy&quot;</span><span>))</span><span>
</span></code></pre>
</div>
<p><img src="https://i1.wp.com/shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post_files/figure-markdown_github/unnamed-chunk-35-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>All models are similarly accurate.</p>
<p></p>
<h3>Combined results of predicting validation test samples</h3>
<p>To compare the predictions from all models, I summed up the prediction probabilities for Death and Recovery from all models and calculated the log2 of the ratio between the summed probabilities for Recovery by the summed probabilities for Death. All cases with a log2 ratio bigger than 1.5 were defined as Recover, cases with a log2 ratio below -1.5 were defined as Death, and the remaining cases were defined as uncertain.</p>
<div>
<pre><code><span>results</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>randomForest</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_rf</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>glmnet</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_glmnet</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>kknn</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_kknn</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>pda</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_pda</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>slda</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_slda</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>pam</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_pam</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>C</span><span>5.0</span><span>Tree</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_C5.0Tree</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>pls</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_pls</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>))</span><span>
</span><span>results</span><span>$</span><span>sum_Death</span><span> </span><span>&lt;-</span><span> </span><span>rowSums</span><span>(</span><span>results</span><span>[,</span><span> </span><span>grep</span><span>(</span><span>&quot;Death&quot;</span><span>,</span><span> </span><span>colnames</span><span>(</span><span>results</span><span>))])</span><span>
</span><span>results</span><span>$</span><span>sum_Recover</span><span> </span><span>&lt;-</span><span> </span><span>rowSums</span><span>(</span><span>results</span><span>[,</span><span> </span><span>grep</span><span>(</span><span>&quot;Recover&quot;</span><span>,</span><span> </span><span>colnames</span><span>(</span><span>results</span><span>))])</span><span>
</span><span>results</span><span>$</span><span>log2_ratio</span><span> </span><span>&lt;-</span><span> </span><span>log2</span><span>(</span><span>results</span><span>$</span><span>sum_Recover</span><span>/</span><span>results</span><span>$</span><span>sum_Death</span><span>)</span><span>
</span><span>results</span><span>$</span><span>true_outcome</span><span> </span><span>&lt;-</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>
</span><span>results</span><span>$</span><span>pred_outcome</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>results</span><span>$</span><span>log2_ratio</span><span> </span><span>></span><span> </span><span>1.5</span><span>,</span><span> </span><span>&quot;Recover&quot;</span><span>,</span><span> </span><span>ifelse</span><span>(</span><span>results</span><span>$</span><span>log2_ratio</span><span> </span><span>&lt;</span><span> </span><span>-1.5</span><span>,</span><span> </span><span>&quot;Death&quot;</span><span>,</span><span> </span><span>&quot;uncertain&quot;</span><span>))</span><span>
</span><span>results</span><span>$</span><span>prediction</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>results</span><span>$</span><span>pred_outcome</span><span> </span><span>==</span><span> </span><span>results</span><span>$</span><span>true_outcome</span><span>,</span><span> </span><span>&quot;CORRECT&quot;</span><span>,</span><span> </span><span>ifelse</span><span>(</span><span>results</span><span>$</span><span>pred_outcome</span><span> </span><span>==</span><span> </span><span>&quot;uncertain&quot;</span><span>,</span><span> </span><span>&quot;uncertain&quot;</span><span>,</span><span> </span><span>&quot;wrong&quot;</span><span>))</span><span>
</span><span>results</span><span>[,</span><span> </span><span>-</span><span>c</span><span>(</span><span>1</span><span>:</span><span>16</span><span>)]</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##          sum_Death sum_Recover  log2_ratio true_outcome pred_outcome prediction
## case_123 2.2236374    5.776363  1.37723972        Death    uncertain  uncertain
## case_127 0.7522674    7.247733  3.26821230      Recover      Recover    CORRECT
## case_128 2.4448101    5.555190  1.18411381      Recover    uncertain  uncertain
## case_14  2.9135620    5.086438  0.80387172      Recover    uncertain  uncertain
## case_19  2.2113377    5.788662  1.38831062        Death    uncertain  uncertain
## case_2   3.6899508    4.310049  0.22410277        Death    uncertain  uncertain
## case_20  4.5621189    3.437881 -0.40818442      Recover    uncertain  uncertain
## case_21  4.9960238    3.003976 -0.73390698      Recover    uncertain  uncertain
## case_34  6.1430233    1.856977 -1.72599312        Death        Death    CORRECT
## case_37  0.8717595    7.128241  3.03154401      Recover      Recover    CORRECT
## case_5   1.7574945    6.242505  1.82860496      Recover      Recover    CORRECT
## case_51  3.8917736    4.108226  0.07808791      Recover    uncertain  uncertain
## case_55  3.1548368    4.845163  0.61897986      Recover    uncertain  uncertain
## case_6   3.5163388    4.483661  0.35060317        Death    uncertain  uncertain
## case_61  5.0803241    2.919676 -0.79911232        Death    uncertain  uncertain
## case_65  1.1282313    6.871769  2.60661863      Recover      Recover    CORRECT
## case_74  5.3438427    2.656157 -1.00853699      Recover    uncertain  uncertain
## case_78  3.2183947    4.781605  0.57115378        Death    uncertain  uncertain
## case_79  1.9521617    6.047838  1.63134700      Recover      Recover    CORRECT
## case_8   3.6106045    4.389395  0.28178184        Death    uncertain  uncertain
## case_87  4.9712879    3.028712 -0.71491522        Death    uncertain  uncertain
## case_91  1.8648837    6.135116  1.71800508      Recover      Recover    CORRECT
## case_94  2.1198087    5.880191  1.47192901      Recover    uncertain  uncertain
</code></pre>
</div>
<p>All predictions based on all models were either correct or uncertain.</p>
<p></p>
<h1>Predicting unknown outcomes</h1>
<p>The above models will now be used to predict the outcome of cases with unknown fate.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_rf</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;rf&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_glmnet</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;glmnet&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_kknn</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;kknn&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_pda</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;pda&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_slda</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;slda&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_pam</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;pam&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_C5.0Tree</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;C5.0Tree&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>model_pls</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;pls&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span><span>models</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>rf</span><span> </span><span>=</span><span> </span><span>model_rf</span><span>,</span><span> </span><span>glmnet</span><span> </span><span>=</span><span> </span><span>model_glmnet</span><span>,</span><span> </span><span>kknn</span><span> </span><span>=</span><span> </span><span>model_kknn</span><span>,</span><span> </span><span>pda</span><span> </span><span>=</span><span> </span><span>model_pda</span><span>,</span><span> </span><span>slda</span><span> </span><span>=</span><span> </span><span>model_slda</span><span>,</span><span>
</span><span>pam</span><span> </span><span>=</span><span> </span><span>model_pam</span><span>,</span><span> </span><span>C</span><span>5.0</span><span>Tree</span><span> </span><span>=</span><span> </span><span>model_C5.0Tree</span><span>,</span><span> </span><span>pls</span><span> </span><span>=</span><span> </span><span>model_pls</span><span>)</span><span>
</span><span># Resample the models
</span><span>resample_results</span><span> </span><span>&lt;-</span><span> </span><span>resamples</span><span>(</span><span>models</span><span>)</span><span>
</span><span>bwplot</span><span>(</span><span>resample_results</span><span> </span><span>,</span><span> </span><span>metric</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Kappa&quot;</span><span>,</span><span>&quot;Accuracy&quot;</span><span>))</span><span>
</span></code></pre>
</div>
<p><img src="https://i0.wp.com/shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post_files/figure-markdown_github/unnamed-chunk-37-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Here again, the accuracy is similar in all models.</p>
<p></p>
<p>The final results are calculate as described above.</p>
<div>
<pre><code><span>results</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>randomForest</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_rf</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>glmnet</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_glmnet</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>kknn</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_kknn</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>pda</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_pda</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>slda</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_slda</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>pam</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_pam</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>C</span><span>5.0</span><span>Tree</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_C5.0Tree</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>pls</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_pls</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>))</span><span>
</span><span>results</span><span>$</span><span>sum_Death</span><span> </span><span>&lt;-</span><span> </span><span>rowSums</span><span>(</span><span>results</span><span>[,</span><span> </span><span>grep</span><span>(</span><span>&quot;Death&quot;</span><span>,</span><span> </span><span>colnames</span><span>(</span><span>results</span><span>))])</span><span>
</span><span>results</span><span>$</span><span>sum_Recover</span><span> </span><span>&lt;-</span><span> </span><span>rowSums</span><span>(</span><span>results</span><span>[,</span><span> </span><span>grep</span><span>(</span><span>&quot;Recover&quot;</span><span>,</span><span> </span><span>colnames</span><span>(</span><span>results</span><span>))])</span><span>
</span><span>results</span><span>$</span><span>log2_ratio</span><span> </span><span>&lt;-</span><span> </span><span>log2</span><span>(</span><span>results</span><span>$</span><span>sum_Recover</span><span>/</span><span>results</span><span>$</span><span>sum_Death</span><span>)</span><span>
</span><span>results</span><span>$</span><span>predicted_outcome</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>results</span><span>$</span><span>log2_ratio</span><span> </span><span>></span><span> </span><span>1.5</span><span>,</span><span> </span><span>&quot;Recover&quot;</span><span>,</span><span> </span><span>ifelse</span><span>(</span><span>results</span><span>$</span><span>log2_ratio</span><span> </span><span>&lt;</span><span> </span><span>-1.5</span><span>,</span><span> </span><span>&quot;Death&quot;</span><span>,</span><span> </span><span>&quot;uncertain&quot;</span><span>))</span><span>
</span><span>results</span><span>[,</span><span> </span><span>-</span><span>c</span><span>(</span><span>1</span><span>:</span><span>16</span><span>)]</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##          sum_Death sum_Recover  log2_ratio predicted_outcome
## case_100 3.6707808    4.329219  0.23801986         uncertain
## case_101 6.3194220    1.680578 -1.91083509             Death
## case_102 6.4960471    1.503953 -2.11080274             Death
## case_103 2.3877223    5.612278  1.23295134         uncertain
## case_104 4.1254604    3.874540 -0.09053024         uncertain
## case_105 2.1082161    5.891784  1.48268174         uncertain
## case_108 6.5941436    1.405856 -2.22973606             Death
## case_109 2.2780779    5.721922  1.32868278         uncertain
## case_110 1.7853361    6.214664  1.79948072           Recover
## case_112 4.5102439    3.489756 -0.37007925         uncertain
## case_113 4.7047554    3.295245 -0.51373420         uncertain
## case_114 1.6397105    6.360290  1.95565132           Recover
## case_115 1.2351519    6.764848  2.45336902           Recover
## case_118 1.4675571    6.532443  2.15420601           Recover
## case_120 1.9322149    6.067785  1.65091447           Recover
## case_122 1.1165786    6.883421  2.62404109           Recover
## case_126 5.4933927    2.506607 -1.13196145         uncertain
## case_130 4.3035732    3.696427 -0.21940367         uncertain
## case_132 5.0166184    2.983382 -0.74976671         uncertain
## case_136 2.6824566    5.317543  0.98720505         uncertain
## case_15  5.8545370    2.145463 -1.44826607         uncertain
## case_16  6.5063794    1.493621 -2.12304122             Death
## case_22  4.2929861    3.707014 -0.21172398         uncertain
## case_28  6.6129447    1.387055 -2.25326754             Death
## case_31  6.2625800    1.737420 -1.84981057             Death
## case_32  6.0986316    1.901368 -1.68144746             Death
## case_38  1.7581540    6.241846  1.82791134           Recover
## case_39  2.4765188    5.523481  1.15726428         uncertain
## case_4   5.1309910    2.869009 -0.83868503         uncertain
## case_40  6.6069297    1.393070 -2.24571191             Death
## case_41  5.5499144    2.450086 -1.17963336         uncertain
## case_42  2.0709160    5.929084  1.51754019           Recover
## case_47  6.1988812    1.801119 -1.78311450             Death
## case_48  5.6500772    2.349923 -1.26565724         uncertain
## case_52  5.9096543    2.090346 -1.49933214         uncertain
## case_54  2.6763066    5.323693  0.99218409         uncertain
## case_56  2.6826414    5.317359  0.98705557         uncertain
## case_62  6.0722552    1.927745 -1.65531833             Death
## case_63  3.9541381    4.045862  0.03308379         uncertain
## case_66  5.8320010    2.167999 -1.42762690         uncertain
## case_67  2.1732059    5.826794  1.42287745         uncertain
## case_68  4.3464174    3.653583 -0.25051491         uncertain
## case_69  1.8902845    6.109715  1.69250181           Recover
## case_70  4.5986242    3.401376 -0.43508393         uncertain
## case_71  2.7966938    5.203306  0.89570626         uncertain
## case_80  2.7270986    5.272901  0.95123017         uncertain
## case_84  1.9485253    6.051475  1.63490411           Recover
## case_85  3.0076953    4.992305  0.73104755         uncertain
## case_86  2.0417802    5.958220  1.54505376           Recover
## case_88  0.7285419    7.271458  3.31916083           Recover
## case_9   2.2188620    5.781138  1.38153353         uncertain
## case_90  1.3973262    6.602674  2.24038155           Recover
## case_92  5.0994993    2.900501 -0.81405366         uncertain
## case_93  4.3979048    3.602095 -0.28798006         uncertain
## case_95  3.5561792    4.443821  0.32147260         uncertain
## case_96  1.3359082    6.664092  2.31858744           Recover
## case_99  5.0776686    2.922331 -0.79704644         uncertain
</code></pre>
</div>
<p>From 57 cases, 14 were defined as Recover, 10 as Death and 33 as uncertain.</p>
<div>
<pre><code><span>results_combined</span><span> </span><span>&lt;-</span><span> </span><span>merge</span><span>(</span><span>results</span><span>[,</span><span> </span><span>-</span><span>c</span><span>(</span><span>1</span><span>:</span><span>16</span><span>)],</span><span> </span><span>fluH7N9.china.2013</span><span>[</span><span>which</span><span>(</span><span>fluH7N9.china.2013</span><span>$</span><span>case.ID</span><span> </span><span>%in%</span><span> </span><span>rownames</span><span>(</span><span>results</span><span>)),</span><span> </span><span>],</span><span> </span><span>by.x</span><span> </span><span>=</span><span> </span><span>&quot;row.names&quot;</span><span>,</span><span> </span><span>by.y</span><span> </span><span>=</span><span> </span><span>&quot;case.ID&quot;</span><span>)</span><span>
</span><span>results_combined</span><span> </span><span>&lt;-</span><span> </span><span>results_combined</span><span>[,</span><span> </span><span>-</span><span>c</span><span>(</span><span>2</span><span>,</span><span> </span><span>3</span><span>,</span><span> </span><span>8</span><span>,</span><span> </span><span>9</span><span>)]</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>results_combined_gather</span><span> </span><span>&lt;-</span><span> </span><span>results_combined</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>group_dates</span><span>,</span><span> </span><span>date</span><span>,</span><span> </span><span>date.of.onset</span><span>:</span><span>date.of.hospitalisation</span><span>)</span><span>
</span><span>results_combined_gather</span><span>$</span><span>group_dates</span><span> </span><span>&lt;-</span><span> </span><span>factor</span><span>(</span><span>results_combined_gather</span><span>$</span><span>group_dates</span><span>,</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;date.of.onset&quot;</span><span>,</span><span> </span><span>&quot;date.of.hospitalisation&quot;</span><span>))</span><span>
</span><span>results_combined_gather</span><span>$</span><span>group_dates</span><span> </span><span>&lt;-</span><span> </span><span>mapvalues</span><span>(</span><span>results_combined_gather</span><span>$</span><span>group_dates</span><span>,</span><span> </span><span>from</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;date.of.onset&quot;</span><span>,</span><span> </span><span>&quot;date.of.hospitalisation&quot;</span><span>),</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Date of onset&quot;</span><span>,</span><span> </span><span>&quot;Date of hospitalisation&quot;</span><span>))</span><span>
</span><span>results_combined_gather</span><span>$</span><span>gender</span><span> </span><span>&lt;-</span><span> </span><span>mapvalues</span><span>(</span><span>results_combined_gather</span><span>$</span><span>gender</span><span>,</span><span> </span><span>from</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;f&quot;</span><span>,</span><span> </span><span>&quot;m&quot;</span><span>),</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Female&quot;</span><span>,</span><span> </span><span>&quot;Male&quot;</span><span>))</span><span>
</span><span>levels</span><span>(</span><span>results_combined_gather</span><span>$</span><span>gender</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>levels</span><span>(</span><span>results_combined_gather</span><span>$</span><span>gender</span><span>),</span><span> </span><span>&quot;unknown&quot;</span><span>)</span><span>
</span><span>results_combined_gather</span><span>$</span><span>gender</span><span>[</span><span>is.na</span><span>(</span><span>results_combined_gather</span><span>$</span><span>gender</span><span>)]</span><span> </span><span>&lt;-</span><span> </span><span>&quot;unknown&quot;</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>ggplot</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>results_combined_gather</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>date</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>log2_ratio</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>predicted_outcome</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_jitter</span><span>(</span><span>aes</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>age</span><span>)),</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.3</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_rug</span><span>()</span><span> </span><span>+</span><span>
</span><span>facet_grid</span><span>(</span><span>gender</span><span> </span><span>~</span><span> </span><span>group_dates</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>
</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;Predicted outcome&quot;</span><span>,</span><span>
</span><span>size</span><span> </span><span>=</span><span> </span><span>&quot;Age&quot;</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Date in 2013&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;log2 ratio of prediction Recover vs Death&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;2013 Influenza A H7N9 cases in China&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Predicted outcome&quot;</span><span>,</span><span>
</span><span>caption</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>
</span><span>)</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_shape_manual</span><span>(</span><span>values</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>15</span><span>,</span><span> </span><span>16</span><span>,</span><span> </span><span>17</span><span>))</span><span> </span><span>+</span><span>
</span><span>scale_color_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post_files/figure-markdown_github/unnamed-chunk-41-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>The comparison of date of onset, data of hospitalisation, gender and age with predicted outcome shows that predicted deaths were associated with older age than predicted Recoveries. Date of onset does not show a bias in either direction.</p>
<p></p>
<h1>Conclusion</h1>
<p>The dataset posed a couple of difficulties, like an unequal distribution of data points across variables and missing values. This makes the modeling inherently prone to flaws. However, real life data is seldom perfect, so I went ahead and tested the modeling success anyway.</p>
<p>By accounting for uncertain classification, the validation data could be classified accurately. However, for a more accurate model, these few cases don’t give enough information to reliably predict the outcome. More cases, more information (i.e. more features) and fewer missing data would improve the modeling outcome.</p>
<p>Unfortunately, it can’t be verified whether the unknown cases were classified correctly.<br>
Also, this data is specific for this case of flu. In order to be able to draw more general conclusions about flu outcome, other cases and other information, for example on medical parameters would be necessary.<br>
However, this dataset served as a nice example for the possibilities (and pitfalls) of machine learning applications and showcased a basic workflow for building prediction models with R.</p>
<hr>
<div>
<pre><code><span>sessionInfo</span><span>()</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## R version 3.3.2 (2016-10-31)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 7 x64 (build 7601) Service Pack 1
## 
## locale:
## [1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252 LC_NUMERIC=C                           LC_TIME=English_United States.1252    
## 
## attached base packages:
## [1] grid      stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] pls_2.5-0           C50_0.1.0-24        pamr_1.55           survival_2.40-1     cluster_2.0.5       ipred_0.9-5         mda_0.4-9           class_7.3-14        kknn_1.3.1          glmnet_2.0-5        foreach_1.4.3       Matrix_1.2-7.1      randomForest_4.6-12 RColorBrewer_1.1-2  rpart.plot_2.1.0    rattle_4.1.0        rpart_4.1-10        caret_6.0-73        lattice_0.20-34     mice_2.25           Rcpp_0.12.7         dplyr_0.5.0         gridExtra_2.2.1     ggplot2_2.2.0       plyr_1.8.4          tidyr_0.6.0         outbreaks_1.0.0    
## 
## loaded via a namespace (and not attached):
##  [1] RGtk2_2.20.31      assertthat_0.1     digest_0.6.10      R6_2.2.0           MatrixModels_0.4-1 stats4_3.3.2       evaluate_0.10      e1071_1.6-7        lazyeval_0.2.0     minqa_1.2.4        SparseM_1.74       car_2.1-3          nloptr_1.0.4       partykit_1.1-1     rmarkdown_1.1      labeling_0.3       splines_3.3.2      lme4_1.1-12        stringr_1.1.0      igraph_1.0.1       munsell_0.4.3      compiler_3.3.2     mgcv_1.8-16        htmltools_0.3.5    nnet_7.3-12        tibble_1.2         prodlim_1.5.7      codetools_0.2-15   MASS_7.3-45        ModelMetrics_1.1.0 nlme_3.1-128       gtable_0.2.0       DBI_0.5-1          magrittr_1.5       scales_0.4.1       stringi_1.1.2      reshape2_1.4.2     Formula_1.2-1      lava_1.4.5         iterators_1.0.8    tools_3.3.2        parallel_3.3.2     pbkrtest_0.4-6     yaml_2.1.14        colorspace_1.3-0   knitr_1.15         quantreg_5.29
</code></pre>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/can-we-predict-flu-deaths-with-machine-learning-and-r/"></div></div><div><a data-url="https://www.r-bloggers.com/can-we-predict-flu-deaths-with-machine-learning-and-r/" data-counturl="https://www.r-bloggers.com/can-we-predict-flu-deaths-with-machine-learning-and-r/" data-count="vertical" data-text="Can we predict flu deaths with Machine Learning and R?" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post"> Shirin's playgRound</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/7pULfZqUxWg"></a></div>