<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/extreme-gradient-boosting-and-preprocessing-in-machine-learning-addendum-to-predicting-flu-outcome-with-r/"></div></div><div><a data-url="https://www.r-bloggers.com/extreme-gradient-boosting-and-preprocessing-in-machine-learning-addendum-to-predicting-flu-outcome-with-r/" data-counturl="https://www.r-bloggers.com/extreme-gradient-boosting-and-preprocessing-in-machine-learning-addendum-to-predicting-flu-outcome-with-r/" data-count="horizontal" data-text="Extreme Gradient Boosting and Preprocessing in Machine Learning – Addendum to predicting flu outcome with R" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://shiring.github.io/machine_learning/2016/12/02/flu_outcome_ML_2_post"> Shirin's playgRound</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post">last week’s post</a> I explored whether machine learning models can be applied to predict flu deaths from the 2013 outbreak of influenza A H7N9 in China. There, I compared random forests, elastic-net regularized generalized linear models, k-nearest neighbors, penalized discriminant analysis, stabilized linear discriminant analysis, nearest shrunken centroids, single C5.0 tree and partial least squares.</p>
<p></p>
<h1>Extreme gradient boosting</h1>
<p><a target="_blank" rel="nofollow" href="http://xgboost.readthedocs.io/en/latest/model.html">Extreme gradient boosting (XGBoost)</a> is a faster and improved implementation of <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Gradient_boosting">gradient boosting</a> for supervised learning and has recently been very successfully applied in Kaggle competitions. Because I’ve heard XGBoost’s praise being sung everywhere lately, I wanted to get my feet wet with it too. So this week I want to compare the prediction success of gradient boosting with the same dataset. Additionally, I want to test the influence of different preprocessing methods on the outcome.</p>
<p></p>
<blockquote>
<p>“XGBoost uses a more regularized model formalization to control over-fitting, which gives it better performance.” Tianqi Chen, developer of xgboost</p>
</blockquote>
<p>XGBoost is a tree ensemble model, which means the sum of predictions from a set of classification and regression trees (CART). In that, XGBoost is similar to Random Forests but it uses a different approach to model training.</p>
<p></p>
<p>Starting with the same test and training data (partitioned into validation test and validation train subsets) from <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post">last week’s post</a>, I am training extreme gradient boosting models as implemented in the <a target="_blank" rel="nofollow">xgboost</a> and <a target="_blank" rel="nofollow" href="http://topepo.github.io/caret/index.html">caret</a> packages with different preprocessing settings.</p>
<p>Out of the different implementations and variations of gradient boosting algorithms, caret performed best on PCA-preprocessed data in the validation set. These paramteres were then used to predict the outcome in the test set and compare it to last week’s predictions.</p>
<p>Compared to last week, there is much less uncertainty in the predictions from XGBoost. Overall, I would say that this algorithm is superior to the others I have used before.</p>
<p></p>
<h2>xgboost</h2>
<p>Extreme gradient boosting is implemented in the xgboost package.</p>
<div>
<pre><code><span># install the stable/pre-compiled version from CRAN
</span><span>
</span><span>install.packages</span><span>(</span><span>'xgboost'</span><span>)</span><span>
</span><span># or install from weekly updated drat repo
</span><span>
</span><span>install.packages</span><span>(</span><span>&quot;drat&quot;</span><span>,</span><span> </span><span>repos</span><span>=</span><span>&quot;https://cran.rstudio.com&quot;</span><span>)</span><span>
</span><span>drat</span><span>:::</span><span>addRepo</span><span>(</span><span>&quot;dmlc&quot;</span><span>)</span><span>
</span><span>install.packages</span><span>(</span><span>&quot;xgboost&quot;</span><span>,</span><span> </span><span>repos</span><span>=</span><span>&quot;http://dmlc.ml/drat/&quot;</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;source&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p>XGBoost supports only numbers, so the outcome classes have to be converted into integers and both training and test data have to be in numeric matrix format.</p>
<div>
<pre><code><span>library</span><span>(</span><span>xgboost</span><span>)</span><span>
</span><span>matrix_train</span><span> </span><span>&lt;-</span><span> </span><span>apply</span><span>(</span><span>val_train_X</span><span>,</span><span> </span><span>2</span><span>,</span><span> </span><span>function</span><span>(</span><span>x</span><span>)</span><span> </span><span>as.numeric</span><span>(</span><span>as.character</span><span>(</span><span>x</span><span>)))</span><span>
</span><span>outcome_death_train</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>val_train_data</span><span>$</span><span>outcome</span><span> </span><span>==</span><span> </span><span>&quot;Death&quot;</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>0</span><span>)</span><span>
</span><span>matrix_test</span><span> </span><span>&lt;-</span><span> </span><span>apply</span><span>(</span><span>val_test_X</span><span>,</span><span> </span><span>2</span><span>,</span><span> </span><span>function</span><span>(</span><span>x</span><span>)</span><span> </span><span>as.numeric</span><span>(</span><span>as.character</span><span>(</span><span>x</span><span>)))</span><span>
</span><span>outcome_death_test</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>val_test_data</span><span>$</span><span>outcome</span><span> </span><span>==</span><span> </span><span>&quot;Death&quot;</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>0</span><span>)</span><span>
</span><span>xgb_train_matrix</span><span> </span><span>&lt;-</span><span> </span><span>xgb.DMatrix</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>as.matrix</span><span>(</span><span>matrix_train</span><span>),</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>outcome_death_train</span><span>)</span><span>
</span><span>xgb_test_matrix</span><span> </span><span>&lt;-</span><span> </span><span>xgb.DMatrix</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>as.matrix</span><span>(</span><span>matrix_test</span><span>),</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>outcome_death_test</span><span>)</span><span>
</span><span>watchlist</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>train</span><span> </span><span>=</span><span> </span><span>xgb_train_matrix</span><span>,</span><span> </span><span>test</span><span> </span><span>=</span><span> </span><span>xgb_test_matrix</span><span>)</span><span>
</span><span>label</span><span> </span><span>&lt;-</span><span> </span><span>getinfo</span><span>(</span><span>xgb_test_matrix</span><span>,</span><span> </span><span>&quot;label&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p>I am using cross validation to evaluate the error rate.</p>
<div>
<pre><code><span>param</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>&quot;objective&quot;</span><span> </span><span>=</span><span> </span><span>&quot;binary:logistic&quot;</span><span>)</span><span>
</span><span>xgb.cv</span><span>(</span><span>param</span><span> </span><span>=</span><span> </span><span>param</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>xgb_train_matrix</span><span>,</span><span> </span><span>nfold</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span>
</span><span>label</span><span> </span><span>=</span><span> </span><span>getinfo</span><span>(</span><span>xgb_train_matrix</span><span>,</span><span> </span><span>&quot;label&quot;</span><span>),</span><span>
</span><span>nrounds</span><span> </span><span>=</span><span> </span><span>5</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1]  train-error:0.133950+0.022131   test-error:0.341131+0.129919 
## [2]  train-error:0.124941+0.033432   test-error:0.358674+0.105191 
## [3]  train-error:0.115932+0.045501   test-error:0.340156+0.092598 
## [4]  train-error:0.124941+0.033432   test-error:0.340156+0.092598 
## [5]  train-error:0.089616+0.051301   test-error:0.394737+0.098465
</code></pre>
</div>
<p></p>
<h3>Training with gbtree</h3>
<p>gbtree is the default booster for xgb.train.</p>
<div>
<pre><code><span>bst_1</span><span> </span><span>&lt;-</span><span> </span><span>xgb.train</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>xgb_train_matrix</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>getinfo</span><span>(</span><span>xgb_train_matrix</span><span>,</span><span> </span><span>&quot;label&quot;</span><span>),</span><span>
</span><span>max.depth</span><span> </span><span>=</span><span> </span><span>2</span><span>,</span><span> </span><span>eta</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>nthread</span><span> </span><span>=</span><span> </span><span>4</span><span>,</span><span> </span><span>nround</span><span> </span><span>=</span><span> </span><span>50</span><span>,</span><span> </span><span># number of trees used for model building
</span><span> </span><span>watchlist</span><span> </span><span>=</span><span> </span><span>watchlist</span><span>,</span><span> </span><span>objective</span><span> </span><span>=</span><span> </span><span>&quot;binary:logistic&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1]  train-error:0.250000    test-error:0.347826 
## [2]  train-error:0.178571    test-error:0.304348 
## [3]  train-error:0.142857    test-error:0.347826 
## [4]  train-error:0.089286    test-error:0.260870 
## [5]  train-error:0.053571    test-error:0.347826 
## [6]  train-error:0.053571    test-error:0.391304 
## [7]  train-error:0.035714    test-error:0.347826 
## [8]  train-error:0.000000    test-error:0.391304 
## [9]  train-error:0.017857    test-error:0.391304 
## [10] train-error:0.000000    test-error:0.391304 
## [11] train-error:0.000000    test-error:0.391304 
## [12] train-error:0.000000    test-error:0.347826 
## [13] train-error:0.000000    test-error:0.347826 
## [14] train-error:0.000000    test-error:0.347826 
## [15] train-error:0.000000    test-error:0.347826 
## [16] train-error:0.000000    test-error:0.347826 
## [17] train-error:0.000000    test-error:0.347826 
## [18] train-error:0.000000    test-error:0.347826 
## [19] train-error:0.000000    test-error:0.347826 
## [20] train-error:0.000000    test-error:0.347826 
## [21] train-error:0.000000    test-error:0.347826 
## [22] train-error:0.000000    test-error:0.347826 
## [23] train-error:0.000000    test-error:0.347826 
## [24] train-error:0.000000    test-error:0.347826 
## [25] train-error:0.000000    test-error:0.347826 
## [26] train-error:0.000000    test-error:0.347826 
## [27] train-error:0.000000    test-error:0.347826 
## [28] train-error:0.000000    test-error:0.347826 
## [29] train-error:0.000000    test-error:0.347826 
## [30] train-error:0.000000    test-error:0.347826 
## [31] train-error:0.000000    test-error:0.347826 
## [32] train-error:0.000000    test-error:0.347826 
## [33] train-error:0.000000    test-error:0.347826 
## [34] train-error:0.000000    test-error:0.347826 
## [35] train-error:0.000000    test-error:0.347826 
## [36] train-error:0.000000    test-error:0.347826 
## [37] train-error:0.000000    test-error:0.347826 
## [38] train-error:0.000000    test-error:0.347826 
## [39] train-error:0.000000    test-error:0.347826 
## [40] train-error:0.000000    test-error:0.347826 
## [41] train-error:0.000000    test-error:0.347826 
## [42] train-error:0.000000    test-error:0.347826 
## [43] train-error:0.000000    test-error:0.347826 
## [44] train-error:0.000000    test-error:0.347826 
## [45] train-error:0.000000    test-error:0.347826 
## [46] train-error:0.000000    test-error:0.347826 
## [47] train-error:0.000000    test-error:0.347826 
## [48] train-error:0.000000    test-error:0.347826 
## [49] train-error:0.000000    test-error:0.347826 
## [50] train-error:0.000000    test-error:0.347826
</code></pre>
</div>
<p>Each feature is grouped by importance with k-means clustering. Gain is the improvement in accuracy that the addition of a feature brings to the branches it is on.</p>
<div>
<pre><code><span>features</span><span> </span><span>=</span><span> </span><span>colnames</span><span>(</span><span>matrix_train</span><span>)</span><span>
</span><span>importance_matrix_1</span><span> </span><span>&lt;-</span><span> </span><span>xgb.importance</span><span>(</span><span>features</span><span>,</span><span> </span><span>model</span><span> </span><span>=</span><span> </span><span>bst_1</span><span>)</span><span>
</span><span>print</span><span>(</span><span>importance_matrix_1</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##                   Feature       Gain      Cover  Frequency
## 1:                    age 0.54677689 0.45726557 0.43333333
## 2:  days_onset_to_outcome 0.14753698 0.11686537 0.11111111
## 3:            early_onset 0.06724531 0.07365416 0.07777778
## 4:          early_outcome 0.06679880 0.08869426 0.07777778
## 5:      province_Shanghai 0.05325295 0.04317030 0.04444444
## 6: days_onset_to_hospital 0.03883631 0.11300882 0.13333333
## 7:         province_other 0.03502971 0.03977657 0.04444444
## 8:               gender_f 0.02610959 0.02484170 0.01111111
## 9:               hospital 0.01841345 0.04272324 0.06666667
</code></pre>
</div>
<div>
<pre><code><span>xgb.ggplot.importance</span><span>(</span><span>importance_matrix_1</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_minimal</span><span>()</span><span>
</span></code></pre>
</div>
<p><img src="https://i0.wp.com/shiring.github.io/machine_learning/2016/12/02/flu_outcome_ML_2_post_files/figure-markdown_github/unnamed-chunk-6-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>pred_1</span><span> </span><span>&lt;-</span><span> </span><span>predict</span><span>(</span><span>bst_1</span><span>,</span><span> </span><span>xgb_test_matrix</span><span>)</span><span>
</span><span>result_1</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>case_ID</span><span> </span><span>=</span><span> </span><span>rownames</span><span>(</span><span>val_test_data</span><span>),</span><span>
</span><span>outcome</span><span> </span><span>=</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>label</span><span>,</span><span> </span><span>prediction_p_death</span><span> </span><span>=</span><span> </span><span>round</span><span>(</span><span>pred_1</span><span>,</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>),</span><span>
</span><span>prediction</span><span> </span><span>=</span><span> </span><span>as.integer</span><span>(</span><span>pred_1</span><span> </span><span>></span><span> </span><span>0.5</span><span>),</span><span>
</span><span>prediction_eval</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>as.integer</span><span>(</span><span>pred_1</span><span> </span><span>></span><span> </span><span>0.5</span><span>)</span><span> </span><span>!=</span><span> </span><span>label</span><span>,</span><span> </span><span>&quot;wrong&quot;</span><span>,</span><span> </span><span>&quot;correct&quot;</span><span>))</span><span>
</span><span>result_1</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##     case_ID outcome label prediction_p_death prediction prediction_eval
## 1  case_123   Death     1               0.01          0           wrong
## 2  case_127 Recover     0               0.00          0         correct
## 3  case_128 Recover     0               0.96          1           wrong
## 4   case_14 Recover     0               0.76          1           wrong
## 5   case_19   Death     1               0.95          1         correct
## 6    case_2   Death     1               0.85          1         correct
## 7   case_20 Recover     0               0.98          1           wrong
## 8   case_21 Recover     0               0.06          0         correct
## 9   case_34   Death     1               1.00          1         correct
## 10  case_37 Recover     0               0.00          0         correct
## 11   case_5 Recover     0               0.03          0         correct
## 12  case_51 Recover     0               0.27          0         correct
## 13  case_55 Recover     0               0.11          0         correct
## 14   case_6   Death     1               0.26          0           wrong
## 15  case_61   Death     1               1.00          1         correct
## 16  case_65 Recover     0               0.00          0         correct
## 17  case_74 Recover     0               0.92          1           wrong
## 18  case_78   Death     1               0.03          0           wrong
## 19  case_79 Recover     0               0.00          0         correct
## 20   case_8   Death     1               0.06          0           wrong
## 21  case_87   Death     1               0.64          1         correct
## 22  case_91 Recover     0               0.00          0         correct
## 23  case_94 Recover     0               0.01          0         correct
</code></pre>
</div>
<div>
<pre><code><span>err</span><span> </span><span>&lt;-</span><span> </span><span>as.numeric</span><span>(</span><span>sum</span><span>(</span><span>as.integer</span><span>(</span><span>pred_1</span><span> </span><span>></span><span> </span><span>0.5</span><span>)</span><span> </span><span>!=</span><span> </span><span>label</span><span>))</span><span>/</span><span>length</span><span>(</span><span>label</span><span>)</span><span>
</span><span>print</span><span>(</span><span>paste</span><span>(</span><span>&quot;test-error =&quot;</span><span>,</span><span> </span><span>round</span><span>(</span><span>err</span><span>,</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)))</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] &quot;test-error = 0.35&quot;
</code></pre>
</div>
<p></p>
<h3>Training with gblinear</h3>
<div>
<pre><code><span>bst_2</span><span> </span><span>&lt;-</span><span> </span><span>xgb.train</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>xgb_train_matrix</span><span>,</span><span> </span><span>booster</span><span> </span><span>=</span><span> </span><span>&quot;gblinear&quot;</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>getinfo</span><span>(</span><span>xgb_train_matrix</span><span>,</span><span> </span><span>&quot;label&quot;</span><span>),</span><span>
</span><span>max.depth</span><span> </span><span>=</span><span> </span><span>2</span><span>,</span><span> </span><span>eta</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>nthread</span><span> </span><span>=</span><span> </span><span>4</span><span>,</span><span> </span><span>nround</span><span> </span><span>=</span><span> </span><span>50</span><span>,</span><span> </span><span># number of trees used for model building
</span><span> </span><span>watchlist</span><span> </span><span>=</span><span> </span><span>watchlist</span><span>,</span><span> </span><span>objective</span><span> </span><span>=</span><span> </span><span>&quot;binary:logistic&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1]  train-error:0.339286    test-error:0.434783 
## [2]  train-error:0.303571    test-error:0.391304 
## [3]  train-error:0.285714    test-error:0.347826 
## [4]  train-error:0.303571    test-error:0.347826 
## [5]  train-error:0.285714    test-error:0.347826 
## [6]  train-error:0.285714    test-error:0.391304 
## [7]  train-error:0.285714    test-error:0.391304 
## [8]  train-error:0.267857    test-error:0.391304 
## [9]  train-error:0.250000    test-error:0.391304 
## [10] train-error:0.250000    test-error:0.478261 
## [11] train-error:0.250000    test-error:0.434783 
## [12] train-error:0.232143    test-error:0.434783 
## [13] train-error:0.232143    test-error:0.434783 
## [14] train-error:0.232143    test-error:0.434783 
## [15] train-error:0.232143    test-error:0.434783 
## [16] train-error:0.232143    test-error:0.434783 
## [17] train-error:0.232143    test-error:0.434783 
## [18] train-error:0.232143    test-error:0.434783 
## [19] train-error:0.232143    test-error:0.434783 
## [20] train-error:0.232143    test-error:0.434783 
## [21] train-error:0.214286    test-error:0.434783 
## [22] train-error:0.214286    test-error:0.434783 
## [23] train-error:0.214286    test-error:0.434783 
## [24] train-error:0.214286    test-error:0.434783 
## [25] train-error:0.214286    test-error:0.434783 
## [26] train-error:0.214286    test-error:0.434783 
## [27] train-error:0.214286    test-error:0.434783 
## [28] train-error:0.214286    test-error:0.434783 
## [29] train-error:0.214286    test-error:0.434783 
## [30] train-error:0.232143    test-error:0.434783 
## [31] train-error:0.214286    test-error:0.434783 
## [32] train-error:0.214286    test-error:0.434783 
## [33] train-error:0.214286    test-error:0.434783 
## [34] train-error:0.214286    test-error:0.434783 
## [35] train-error:0.214286    test-error:0.434783 
## [36] train-error:0.214286    test-error:0.434783 
## [37] train-error:0.214286    test-error:0.434783 
## [38] train-error:0.214286    test-error:0.434783 
## [39] train-error:0.214286    test-error:0.434783 
## [40] train-error:0.214286    test-error:0.434783 
## [41] train-error:0.214286    test-error:0.434783 
## [42] train-error:0.214286    test-error:0.434783 
## [43] train-error:0.214286    test-error:0.434783 
## [44] train-error:0.214286    test-error:0.434783 
## [45] train-error:0.214286    test-error:0.434783 
## [46] train-error:0.214286    test-error:0.434783 
## [47] train-error:0.214286    test-error:0.434783 
## [48] train-error:0.214286    test-error:0.434783 
## [49] train-error:0.214286    test-error:0.434783 
## [50] train-error:0.214286    test-error:0.434783
</code></pre>
</div>
<p>Each feature is grouped by importance with k-means clustering. Gain is the improvement in accuracy that the addition of a feature brings to the branches it is on.</p>
<div>
<pre><code><span>features</span><span> </span><span>=</span><span> </span><span>colnames</span><span>(</span><span>matrix_train</span><span>)</span><span>
</span><span>importance_matrix_2</span><span> </span><span>&lt;-</span><span> </span><span>xgb.importance</span><span>(</span><span>features</span><span>,</span><span> </span><span>model</span><span> </span><span>=</span><span> </span><span>bst_2</span><span>)</span><span>
</span><span>print</span><span>(</span><span>importance_matrix_2</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##                    Feature     Weight
##  1:                    age  0.0439012
##  2:               hospital -0.3131160
##  3:               gender_f  0.8067830
##  4:       province_Jiangsu -0.8807380
##  5:      province_Shanghai -0.4065330
##  6:      province_Zhejiang -0.1284590
##  7:         province_other  0.4510330
##  8:  days_onset_to_outcome  0.0209952
##  9: days_onset_to_hospital -0.0856504
## 10:            early_onset  0.6452190
## 11:          early_outcome  2.5134300
</code></pre>
</div>
<div>
<pre><code><span>xgb.ggplot.importance</span><span>(</span><span>importance_matrix_2</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_minimal</span><span>()</span><span>
</span></code></pre>
</div>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2016/12/02/flu_outcome_ML_2_post_files/figure-markdown_github/unnamed-chunk-10-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>pred_2</span><span> </span><span>&lt;-</span><span> </span><span>predict</span><span>(</span><span>bst_2</span><span>,</span><span> </span><span>xgb_test_matrix</span><span>)</span><span>
</span><span>result_2</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>case_ID</span><span> </span><span>=</span><span> </span><span>rownames</span><span>(</span><span>val_test_data</span><span>),</span><span>
</span><span>outcome</span><span> </span><span>=</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>label</span><span>,</span><span> </span><span>prediction_p_death</span><span> </span><span>=</span><span> </span><span>round</span><span>(</span><span>pred_2</span><span>,</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>),</span><span>
</span><span>prediction</span><span> </span><span>=</span><span> </span><span>as.integer</span><span>(</span><span>pred_2</span><span> </span><span>></span><span> </span><span>0.5</span><span>),</span><span>
</span><span>prediction_eval</span><span> </span><span>=</span><span> </span><span>ifelse</span><span>(</span><span>as.integer</span><span>(</span><span>pred_2</span><span> </span><span>></span><span> </span><span>0.5</span><span>)</span><span> </span><span>!=</span><span> </span><span>label</span><span>,</span><span> </span><span>&quot;wrong&quot;</span><span>,</span><span> </span><span>&quot;correct&quot;</span><span>))</span><span>
</span><span>result_2</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##     case_ID outcome label prediction_p_death prediction prediction_eval
## 1  case_123   Death     1               0.10          0           wrong
## 2  case_127 Recover     0               0.03          0         correct
## 3  case_128 Recover     0               0.13          0         correct
## 4   case_14 Recover     0               0.15          0         correct
## 5   case_19   Death     1               0.28          0           wrong
## 6    case_2   Death     1               0.28          0           wrong
## 7   case_20 Recover     0               0.80          1           wrong
## 8   case_21 Recover     0               0.79          1           wrong
## 9   case_34   Death     1               0.74          1         correct
## 10  case_37 Recover     0               0.06          0         correct
## 11   case_5 Recover     0               0.14          0         correct
## 12  case_51 Recover     0               0.77          1           wrong
## 13  case_55 Recover     0               0.23          0         correct
## 14   case_6   Death     1               0.47          0           wrong
## 15  case_61   Death     1               0.52          1         correct
## 16  case_65 Recover     0               0.03          0         correct
## 17  case_74 Recover     0               0.70          1           wrong
## 18  case_78   Death     1               0.22          0           wrong
## 19  case_79 Recover     0               0.06          0         correct
## 20   case_8   Death     1               0.36          0           wrong
## 21  case_87   Death     1               0.65          1         correct
## 22  case_91 Recover     0               0.05          0         correct
## 23  case_94 Recover     0               0.07          0         correct
</code></pre>
</div>
<div>
<pre><code><span>err</span><span> </span><span>&lt;-</span><span> </span><span>as.numeric</span><span>(</span><span>sum</span><span>(</span><span>as.integer</span><span>(</span><span>pred_2</span><span> </span><span>></span><span> </span><span>0.5</span><span>)</span><span> </span><span>!=</span><span> </span><span>label</span><span>))</span><span>/</span><span>length</span><span>(</span><span>label</span><span>)</span><span>
</span><span>print</span><span>(</span><span>paste</span><span>(</span><span>&quot;test-error =&quot;</span><span>,</span><span> </span><span>round</span><span>(</span><span>err</span><span>,</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)))</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] &quot;test-error = 0.43&quot;
</code></pre>
</div>
<p></p>
<h2>caret</h2>
<p>Extreme gradient boosting is also implemented in the caret package. Caret also provides options for preprocessing, of which I will compare a few.</p>
<p></p>
<h3>No preprocessing</h3>
<div>
<pre><code><span>library</span><span>(</span><span>caret</span><span>)</span><span>
</span><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_xgb_null</span><span> </span><span>&lt;-</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span>=</span><span>val_train_data</span><span>,</span><span>
</span><span>method</span><span>=</span><span>&quot;xgbTree&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_xgb_null</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>]),</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Death Recover
##    Death       5       4
##    Recover     4      10
##                                           
##                Accuracy : 0.6522          
##                  95% CI : (0.4273, 0.8362)
##     No Information Rate : 0.6087          
##     P-Value [Acc > NIR] : 0.4216          
##                                           
##                   Kappa : 0.2698          
##  Mcnemar's Test P-Value : 1.0000          
##                                           
##             Sensitivity : 0.5556          
##             Specificity : 0.7143          
##          Pos Pred Value : 0.5556          
##          Neg Pred Value : 0.7143          
##              Prevalence : 0.3913          
##          Detection Rate : 0.2174          
##    Detection Prevalence : 0.3913          
##       Balanced Accuracy : 0.6349          
##                                           
##        'Positive' Class : Death           
## 
</code></pre>
</div>
<p></p>
<h3>Scaling and centering</h3>
<p>With this method the column variables are centered (subtracting the column mean from each value in a column) and standardized (dividing by the column standard deviation).</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_xgb_sc_cen</span><span> </span><span>&lt;-</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span>=</span><span>val_train_data</span><span>,</span><span>
</span><span>method</span><span>=</span><span>&quot;xgbTree&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;scale&quot;</span><span>,</span><span> </span><span>&quot;center&quot;</span><span>),</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_xgb_sc_cen</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>]),</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Death Recover
##    Death       5       4
##    Recover     4      10
##                                           
##                Accuracy : 0.6522          
##                  95% CI : (0.4273, 0.8362)
##     No Information Rate : 0.6087          
##     P-Value [Acc > NIR] : 0.4216          
##                                           
##                   Kappa : 0.2698          
##  Mcnemar's Test P-Value : 1.0000          
##                                           
##             Sensitivity : 0.5556          
##             Specificity : 0.7143          
##          Pos Pred Value : 0.5556          
##          Neg Pred Value : 0.7143          
##              Prevalence : 0.3913          
##          Detection Rate : 0.2174          
##    Detection Prevalence : 0.3913          
##       Balanced Accuracy : 0.6349          
##                                           
##        'Positive' Class : Death           
## 
</code></pre>
</div>
<p>Scaling and centering did not improve the predictions.</p>
<div>
<pre><code><span>pred_3</span><span> </span><span>&lt;-</span><span> </span><span>predict</span><span>(</span><span>model_xgb_sc_cen</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>])</span><span>
</span><span>pred_3b</span><span> </span><span>&lt;-</span><span> </span><span>round</span><span>(</span><span>predict</span><span>(</span><span>model_xgb_sc_cen</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span><span>result_3</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>case_ID</span><span> </span><span>=</span><span> </span><span>rownames</span><span>(</span><span>val_test_data</span><span>),</span><span>
</span><span>outcome</span><span> </span><span>=</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>label</span><span>,</span><span> </span><span>prediction</span><span> </span><span>=</span><span> </span><span>pred_3</span><span>,</span><span>
</span><span>pred_3b</span><span>)</span><span>
</span><span>result_3</span><span>$</span><span>prediction_eval</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>result_3</span><span>$</span><span>prediction</span><span> </span><span>!=</span><span> </span><span>result_3</span><span>$</span><span>outcome</span><span>,</span><span> </span><span>&quot;wrong&quot;</span><span>,</span><span> </span><span>&quot;correct&quot;</span><span>)</span><span>
</span><span>result_3</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##     case_ID outcome label prediction Death Recover prediction_eval
## 1  case_123   Death     1    Recover  0.04    0.96           wrong
## 2  case_127 Recover     0    Recover  0.04    0.96         correct
## 3  case_128 Recover     0      Death  0.81    0.19           wrong
## 4   case_14 Recover     0      Death  0.70    0.30           wrong
## 5   case_19   Death     1      Death  0.67    0.33         correct
## 6    case_2   Death     1      Death  0.66    0.34         correct
## 7   case_20 Recover     0      Death  0.76    0.24           wrong
## 8   case_21 Recover     0    Recover  0.17    0.83         correct
## 9   case_34   Death     1      Death  0.94    0.06         correct
## 10  case_37 Recover     0    Recover  0.02    0.98         correct
## 11   case_5 Recover     0    Recover  0.09    0.91         correct
## 12  case_51 Recover     0    Recover  0.41    0.59         correct
## 13  case_55 Recover     0    Recover  0.12    0.88         correct
## 14   case_6   Death     1    Recover  0.30    0.70           wrong
## 15  case_61   Death     1      Death  0.99    0.01         correct
## 16  case_65 Recover     0    Recover  0.01    0.99         correct
## 17  case_74 Recover     0      Death  0.85    0.15           wrong
## 18  case_78   Death     1    Recover  0.21    0.79           wrong
## 19  case_79 Recover     0    Recover  0.01    0.99         correct
## 20   case_8   Death     1    Recover  0.20    0.80           wrong
## 21  case_87   Death     1      Death  0.65    0.35         correct
## 22  case_91 Recover     0    Recover  0.01    0.99         correct
## 23  case_94 Recover     0    Recover  0.07    0.93         correct
</code></pre>
</div>
<p></p>
<h3>Box-Cox transformation</h3>
<p>The Box-Cox power transformation is used to normalize data.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_xgb_BoxCox</span><span> </span><span>&lt;-</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span>=</span><span>val_train_data</span><span>,</span><span>
</span><span>method</span><span>=</span><span>&quot;xgbTree&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>&quot;BoxCox&quot;</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_xgb_BoxCox</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>]),</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Death Recover
##    Death       5       4
##    Recover     4      10
##                                           
##                Accuracy : 0.6522          
##                  95% CI : (0.4273, 0.8362)
##     No Information Rate : 0.6087          
##     P-Value [Acc > NIR] : 0.4216          
##                                           
##                   Kappa : 0.2698          
##  Mcnemar's Test P-Value : 1.0000          
##                                           
##             Sensitivity : 0.5556          
##             Specificity : 0.7143          
##          Pos Pred Value : 0.5556          
##          Neg Pred Value : 0.7143          
##              Prevalence : 0.3913          
##          Detection Rate : 0.2174          
##    Detection Prevalence : 0.3913          
##       Balanced Accuracy : 0.6349          
##                                           
##        'Positive' Class : Death           
## 
</code></pre>
</div>
<p>Box-Cox transformation did not improve the predictions either.</p>
<div>
<pre><code><span>pred_4</span><span> </span><span>&lt;-</span><span> </span><span>predict</span><span>(</span><span>model_xgb_BoxCox</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>])</span><span>
</span><span>pred_4b</span><span> </span><span>&lt;-</span><span> </span><span>round</span><span>(</span><span>predict</span><span>(</span><span>model_xgb_BoxCox</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span><span>result_4</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>case_ID</span><span> </span><span>=</span><span> </span><span>rownames</span><span>(</span><span>val_test_data</span><span>),</span><span>
</span><span>outcome</span><span> </span><span>=</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>label</span><span>,</span><span> </span><span>prediction</span><span> </span><span>=</span><span> </span><span>pred_4</span><span>,</span><span>
</span><span>pred_4b</span><span>)</span><span>
</span><span>result_4</span><span>$</span><span>prediction_eval</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>result_4</span><span>$</span><span>prediction</span><span> </span><span>!=</span><span> </span><span>result_4</span><span>$</span><span>outcome</span><span>,</span><span> </span><span>&quot;wrong&quot;</span><span>,</span><span> </span><span>&quot;correct&quot;</span><span>)</span><span>
</span><span>result_4</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##     case_ID outcome label prediction Death Recover prediction_eval
## 1  case_123   Death     1    Recover  0.01    0.99           wrong
## 2  case_127 Recover     0    Recover  0.01    0.99         correct
## 3  case_128 Recover     0      Death  0.88    0.12           wrong
## 4   case_14 Recover     0      Death  0.81    0.19           wrong
## 5   case_19   Death     1      Death  0.88    0.12         correct
## 6    case_2   Death     1      Death  0.79    0.21         correct
## 7   case_20 Recover     0      Death  0.84    0.16           wrong
## 8   case_21 Recover     0    Recover  0.07    0.93         correct
## 9   case_34   Death     1      Death  0.97    0.03         correct
## 10  case_37 Recover     0    Recover  0.01    0.99         correct
## 11   case_5 Recover     0    Recover  0.06    0.94         correct
## 12  case_51 Recover     0    Recover  0.27    0.73         correct
## 13  case_55 Recover     0    Recover  0.09    0.91         correct
## 14   case_6   Death     1    Recover  0.30    0.70           wrong
## 15  case_61   Death     1      Death  1.00    0.00         correct
## 16  case_65 Recover     0    Recover  0.00    1.00         correct
## 17  case_74 Recover     0      Death  0.90    0.10           wrong
## 18  case_78   Death     1    Recover  0.11    0.89           wrong
## 19  case_79 Recover     0    Recover  0.00    1.00         correct
## 20   case_8   Death     1    Recover  0.13    0.87           wrong
## 21  case_87   Death     1      Death  0.61    0.39         correct
## 22  case_91 Recover     0    Recover  0.00    1.00         correct
## 23  case_94 Recover     0    Recover  0.04    0.96         correct
</code></pre>
</div>
<p></p>
<h4>Principal Component Analysis (PCA)</h4>
<p>PCA is used for dimensionality reduction. When applied as a preprocessing method the number of features are reduced by using the eigenvectors of the covariance matrix.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_xgb_pca</span><span> </span><span>&lt;-</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span>=</span><span>val_train_data</span><span>,</span><span>
</span><span>method</span><span>=</span><span>&quot;xgbTree&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>&quot;pca&quot;</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_xgb_pca</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>]),</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Death Recover
##    Death       4       2
##    Recover     5      12
##                                           
##                Accuracy : 0.6957          
##                  95% CI : (0.4708, 0.8679)
##     No Information Rate : 0.6087          
##     P-Value [Acc > NIR] : 0.2644          
##                                           
##                   Kappa : 0.3207          
##  Mcnemar's Test P-Value : 0.4497          
##                                           
##             Sensitivity : 0.4444          
##             Specificity : 0.8571          
##          Pos Pred Value : 0.6667          
##          Neg Pred Value : 0.7059          
##              Prevalence : 0.3913          
##          Detection Rate : 0.1739          
##    Detection Prevalence : 0.2609          
##       Balanced Accuracy : 0.6508          
##                                           
##        'Positive' Class : Death           
## 
</code></pre>
</div>
<p>PCA did improve the predictions slightly.</p>
<div>
<pre><code><span>pred_5</span><span> </span><span>&lt;-</span><span> </span><span>predict</span><span>(</span><span>model_xgb_pca</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>])</span><span>
</span><span>pred_5b</span><span> </span><span>&lt;-</span><span> </span><span>round</span><span>(</span><span>predict</span><span>(</span><span>model_xgb_pca</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span><span>result_5</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>case_ID</span><span> </span><span>=</span><span> </span><span>rownames</span><span>(</span><span>val_test_data</span><span>),</span><span>
</span><span>outcome</span><span> </span><span>=</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>label</span><span>,</span><span> </span><span>prediction</span><span> </span><span>=</span><span> </span><span>pred_5</span><span>,</span><span>
</span><span>pred_5b</span><span>)</span><span>
</span><span>result_5</span><span>$</span><span>prediction_eval</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>result_5</span><span>$</span><span>prediction</span><span> </span><span>!=</span><span> </span><span>result_5</span><span>$</span><span>outcome</span><span>,</span><span> </span><span>&quot;wrong&quot;</span><span>,</span><span> </span><span>&quot;correct&quot;</span><span>)</span><span>
</span><span>result_5</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##     case_ID outcome label prediction Death Recover prediction_eval
## 1  case_123   Death     1    Recover  0.02    0.98           wrong
## 2  case_127 Recover     0    Recover  0.29    0.71         correct
## 3  case_128 Recover     0    Recover  0.01    0.99         correct
## 4   case_14 Recover     0    Recover  0.44    0.56         correct
## 5   case_19   Death     1    Recover  0.07    0.93           wrong
## 6    case_2   Death     1    Recover  0.26    0.74           wrong
## 7   case_20 Recover     0    Recover  0.16    0.84         correct
## 8   case_21 Recover     0      Death  0.57    0.43           wrong
## 9   case_34   Death     1      Death  0.93    0.07         correct
## 10  case_37 Recover     0    Recover  0.03    0.97         correct
## 11   case_5 Recover     0    Recover  0.11    0.89         correct
## 12  case_51 Recover     0    Recover  0.19    0.81         correct
## 13  case_55 Recover     0    Recover  0.16    0.84         correct
## 14   case_6   Death     1      Death  0.55    0.45         correct
## 15  case_61   Death     1    Recover  0.22    0.78           wrong
## 16  case_65 Recover     0    Recover  0.08    0.92         correct
## 17  case_74 Recover     0      Death  0.67    0.33           wrong
## 18  case_78   Death     1      Death  0.87    0.13         correct
## 19  case_79 Recover     0    Recover  0.16    0.84         correct
## 20   case_8   Death     1    Recover  0.22    0.78           wrong
## 21  case_87   Death     1      Death  0.66    0.34         correct
## 22  case_91 Recover     0    Recover  0.01    0.99         correct
## 23  case_94 Recover     0    Recover  0.14    0.86         correct
</code></pre>
</div>
<p></p>
<h3>Median imputation</h3>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_xgb_medianImpute</span><span> </span><span>&lt;-</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span>=</span><span>val_train_data</span><span>,</span><span>
</span><span>method</span><span>=</span><span>&quot;xgbTree&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>&quot;medianImpute&quot;</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>confusionMatrix</span><span>(</span><span>predict</span><span>(</span><span>model_xgb_medianImpute</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>]),</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Death Recover
##    Death       5       4
##    Recover     4      10
##                                           
##                Accuracy : 0.6522          
##                  95% CI : (0.4273, 0.8362)
##     No Information Rate : 0.6087          
##     P-Value [Acc > NIR] : 0.4216          
##                                           
##                   Kappa : 0.2698          
##  Mcnemar's Test P-Value : 1.0000          
##                                           
##             Sensitivity : 0.5556          
##             Specificity : 0.7143          
##          Pos Pred Value : 0.5556          
##          Neg Pred Value : 0.7143          
##              Prevalence : 0.3913          
##          Detection Rate : 0.2174          
##    Detection Prevalence : 0.3913          
##       Balanced Accuracy : 0.6349          
##                                           
##        'Positive' Class : Death           
## 
</code></pre>
</div>
<p>Median imputation did not improve the predictions either.</p>
<div>
<pre><code><span>pred_6</span><span> </span><span>&lt;-</span><span> </span><span>predict</span><span>(</span><span>model_xgb_medianImpute</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>])</span><span>
</span><span>pred_6b</span><span> </span><span>&lt;-</span><span> </span><span>round</span><span>(</span><span>predict</span><span>(</span><span>model_xgb_medianImpute</span><span>,</span><span> </span><span>val_test_data</span><span>[,</span><span> </span><span>-1</span><span>],</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span><span>result_6</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>case_ID</span><span> </span><span>=</span><span> </span><span>rownames</span><span>(</span><span>val_test_data</span><span>),</span><span>
</span><span>outcome</span><span> </span><span>=</span><span> </span><span>val_test_data</span><span>$</span><span>outcome</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>label</span><span>,</span><span> </span><span>prediction</span><span> </span><span>=</span><span> </span><span>pred_6</span><span>,</span><span>
</span><span>pred_6b</span><span>)</span><span>
</span><span>result_6</span><span>$</span><span>prediction_eval</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>result_6</span><span>$</span><span>prediction</span><span> </span><span>!=</span><span> </span><span>result_6</span><span>$</span><span>outcome</span><span>,</span><span> </span><span>&quot;wrong&quot;</span><span>,</span><span> </span><span>&quot;correct&quot;</span><span>)</span><span>
</span><span>result_6</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##     case_ID outcome label prediction Death Recover prediction_eval
## 1  case_123   Death     1    Recover  0.04    0.96           wrong
## 2  case_127 Recover     0    Recover  0.04    0.96         correct
## 3  case_128 Recover     0      Death  0.81    0.19           wrong
## 4   case_14 Recover     0      Death  0.70    0.30           wrong
## 5   case_19   Death     1      Death  0.67    0.33         correct
## 6    case_2   Death     1      Death  0.66    0.34         correct
## 7   case_20 Recover     0      Death  0.76    0.24           wrong
## 8   case_21 Recover     0    Recover  0.17    0.83         correct
## 9   case_34   Death     1      Death  0.94    0.06         correct
## 10  case_37 Recover     0    Recover  0.02    0.98         correct
## 11   case_5 Recover     0    Recover  0.09    0.91         correct
## 12  case_51 Recover     0    Recover  0.41    0.59         correct
## 13  case_55 Recover     0    Recover  0.12    0.88         correct
## 14   case_6   Death     1    Recover  0.30    0.70           wrong
## 15  case_61   Death     1      Death  0.99    0.01         correct
## 16  case_65 Recover     0    Recover  0.01    0.99         correct
## 17  case_74 Recover     0      Death  0.85    0.15           wrong
## 18  case_78   Death     1    Recover  0.21    0.79           wrong
## 19  case_79 Recover     0    Recover  0.01    0.99         correct
## 20   case_8   Death     1    Recover  0.20    0.80           wrong
## 21  case_87   Death     1      Death  0.65    0.35         correct
## 22  case_91 Recover     0    Recover  0.01    0.99         correct
## 23  case_94 Recover     0    Recover  0.07    0.93         correct
</code></pre>
</div>
<p></p>
<h1>Comparison of extreme gradient boosting models</h1>
<h2>Combining results</h2>
<div>
<pre><code><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>result</span><span> </span><span>&lt;-</span><span> </span><span>left_join</span><span>(</span><span>result_1</span><span>[,</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>2</span><span>,</span><span> </span><span>6</span><span>)],</span><span> </span><span>result_2</span><span>[,</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>6</span><span>)],</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;case_ID&quot;</span><span>)</span><span>
</span><span>result</span><span> </span><span>&lt;-</span><span> </span><span>left_join</span><span>(</span><span>result</span><span>,</span><span> </span><span>result_3</span><span>[,</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>7</span><span>)],</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;case_ID&quot;</span><span>)</span><span>
</span><span>result</span><span> </span><span>&lt;-</span><span> </span><span>left_join</span><span>(</span><span>result</span><span>,</span><span> </span><span>result_4</span><span>[,</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>7</span><span>)],</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;case_ID&quot;</span><span>)</span><span>
</span><span>result</span><span> </span><span>&lt;-</span><span> </span><span>left_join</span><span>(</span><span>result</span><span>,</span><span> </span><span>result_5</span><span>[,</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>7</span><span>)],</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;case_ID&quot;</span><span>)</span><span>
</span><span>result</span><span> </span><span>&lt;-</span><span> </span><span>left_join</span><span>(</span><span>result</span><span>,</span><span> </span><span>result_6</span><span>[,</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>7</span><span>)],</span><span> </span><span>by</span><span> </span><span>=</span><span> </span><span>&quot;case_ID&quot;</span><span>)</span><span>
</span><span>colnames</span><span>(</span><span>result</span><span>)[</span><span>-</span><span>c</span><span>(</span><span>1</span><span>:</span><span>2</span><span>)]</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;pred_xgboost_gbtree&quot;</span><span>,</span><span> </span><span>&quot;pred_xgboost_gblinear&quot;</span><span>,</span><span> </span><span>&quot;model_xgb_sc_cen&quot;</span><span>,</span><span> </span><span>&quot;model_xgb_BoxCox&quot;</span><span>,</span><span> </span><span>&quot;pred_xgbTree_pca&quot;</span><span>,</span><span> </span><span>&quot;model_xgb_medianImpute&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<h3>What’s the rate of correctly predicted cases in the validation data?</h3>
<div>
<pre><code><span>round</span><span>(</span><span>sum</span><span>(</span><span>result</span><span>$</span><span>pred_xgboost_gbtree</span><span> </span><span>==</span><span> </span><span>&quot;correct&quot;</span><span>)</span><span>/</span><span>nrow</span><span>(</span><span>result</span><span>),</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 0.65
</code></pre>
</div>
<div>
<pre><code><span>round</span><span>(</span><span>sum</span><span>(</span><span>result</span><span>$</span><span>pred_xgboost_gblinear</span><span> </span><span>==</span><span> </span><span>&quot;correct&quot;</span><span>)</span><span>/</span><span>nrow</span><span>(</span><span>result</span><span>),</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 0.57
</code></pre>
</div>
<div>
<pre><code><span>round</span><span>(</span><span>sum</span><span>(</span><span>result</span><span>$</span><span>model_xgb_sc_cen</span><span> </span><span>==</span><span> </span><span>&quot;correct&quot;</span><span>)</span><span>/</span><span>nrow</span><span>(</span><span>result</span><span>),</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 0.65
</code></pre>
</div>
<div>
<pre><code><span>round</span><span>(</span><span>sum</span><span>(</span><span>result</span><span>$</span><span>model_xgb_BoxCox</span><span> </span><span>==</span><span> </span><span>&quot;correct&quot;</span><span>)</span><span>/</span><span>nrow</span><span>(</span><span>result</span><span>),</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 0.65
</code></pre>
</div>
<div>
<pre><code><span>round</span><span>(</span><span>sum</span><span>(</span><span>result</span><span>$</span><span>pred_xgbTree_pca</span><span> </span><span>==</span><span> </span><span>&quot;correct&quot;</span><span>)</span><span>/</span><span>nrow</span><span>(</span><span>result</span><span>),</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 0.7
</code></pre>
</div>
<div>
<pre><code><span>round</span><span>(</span><span>sum</span><span>(</span><span>result</span><span>$</span><span>model_xgb_medianImpute</span><span> </span><span>==</span><span> </span><span>&quot;correct&quot;</span><span>)</span><span>/</span><span>nrow</span><span>(</span><span>result</span><span>),</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## [1] 0.65
</code></pre>
</div>
<p>PCA preprocessing combined with caret’s implementation of extreme gradient boosting had the highest number of accurately predicted cases in the validation data set.</p>
<p></p>
<h1>Predicting unknown output</h1>
<p>Because PCA preprocessing and caret’s implementation of extreme gradient boosting had the highest prediction accuracy, this model will be used to predict the unknown cases from the original dataset.</p>
<p>Uncertainty of prediction is accounted for with a cutoff below 0.7.</p>
<div>
<pre><code><span>set.seed</span><span>(</span><span>27</span><span>)</span><span>
</span><span>model_xgb_pca</span><span> </span><span>&lt;-</span><span>train</span><span>(</span><span>outcome</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>train_data</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;xgbTree&quot;</span><span>,</span><span>
</span><span>preProcess</span><span> </span><span>=</span><span> </span><span>&quot;pca&quot;</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span> </span><span>number</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span> </span><span>repeats</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>verboseIter</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>pred</span><span> </span><span>&lt;-</span><span> </span><span>predict</span><span>(</span><span>model_xgb_pca</span><span>,</span><span> </span><span>test_data</span><span>)</span><span>
</span><span>predb</span><span> </span><span>&lt;-</span><span> </span><span>round</span><span>(</span><span>predict</span><span>(</span><span>model_xgb_pca</span><span>,</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span> </span><span>digits</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span><span>result</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>case_ID</span><span> </span><span>=</span><span> </span><span>rownames</span><span>(</span><span>test_data</span><span>),</span><span>
</span><span>prediction</span><span> </span><span>=</span><span> </span><span>pred</span><span>,</span><span>
</span><span>predb</span><span>)</span><span>
</span><span>result</span><span>$</span><span>predicted_outcome</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>result</span><span>$</span><span>Death</span><span> </span><span>></span><span> </span><span>0.7</span><span>,</span><span> </span><span>&quot;Death&quot;</span><span>,</span><span>
</span><span>ifelse</span><span>(</span><span>result</span><span>$</span><span>Recover</span><span> </span><span>></span><span> </span><span>0.7</span><span>,</span><span> </span><span>&quot;Recover&quot;</span><span>,</span><span> </span><span>&quot;uncertain&quot;</span><span>))</span><span>
</span><span>result</span><span>
</span></code></pre>
</div>
<div>
<pre><code>##     case_ID prediction Death Recover predicted_outcome
## 1  case_100    Recover  0.08    0.92           Recover
## 2  case_101    Recover  0.11    0.89           Recover
## 3  case_102    Recover  0.04    0.96           Recover
## 4  case_103    Recover  0.01    0.99           Recover
## 5  case_104    Recover  0.19    0.81           Recover
## 6  case_105    Recover  0.01    0.99           Recover
## 7  case_108      Death  0.99    0.01             Death
## 8  case_109    Recover  0.30    0.70         uncertain
## 9  case_110    Recover  0.16    0.84           Recover
## 10 case_112      Death  0.54    0.46         uncertain
## 11 case_113    Recover  0.02    0.98           Recover
## 12 case_114    Recover  0.30    0.70         uncertain
## 13 case_115    Recover  0.02    0.98           Recover
## 14 case_118      Death  0.65    0.35         uncertain
## 15 case_120      Death  0.89    0.11             Death
## 16 case_122    Recover  0.00    1.00           Recover
## 17 case_126    Recover  0.04    0.96           Recover
## 18 case_130    Recover  0.42    0.58         uncertain
## 19 case_132    Recover  0.47    0.53         uncertain
## 20 case_136      Death  0.86    0.14             Death
## 21  case_15      Death  0.99    0.01             Death
## 22  case_16      Death  0.91    0.09             Death
## 23  case_22      Death  0.80    0.20             Death
## 24  case_28      Death  0.91    0.09             Death
## 25  case_31      Death  1.00    0.00             Death
## 26  case_32      Death  0.93    0.07             Death
## 27  case_38      Death  0.87    0.13             Death
## 28  case_39      Death  0.88    0.12             Death
## 29   case_4      Death  0.99    0.01             Death
## 30  case_40      Death  1.00    0.00             Death
## 31  case_41      Death  0.95    0.05             Death
## 32  case_42    Recover  0.49    0.51         uncertain
## 33  case_47      Death  0.91    0.09             Death
## 34  case_48      Death  0.95    0.05             Death
## 35  case_52      Death  0.62    0.38         uncertain
## 36  case_54    Recover  0.43    0.57         uncertain
## 37  case_56      Death  0.81    0.19             Death
## 38  case_62      Death  0.58    0.42         uncertain
## 39  case_63    Recover  0.44    0.56         uncertain
## 40  case_66    Recover  0.18    0.82           Recover
## 41  case_67    Recover  0.02    0.98           Recover
## 42  case_68    Recover  0.27    0.73           Recover
## 43  case_69    Recover  0.35    0.65         uncertain
## 44  case_70    Recover  0.02    0.98           Recover
## 45  case_71    Recover  0.17    0.83           Recover
## 46  case_80    Recover  0.05    0.95           Recover
## 47  case_84    Recover  0.02    0.98           Recover
## 48  case_85      Death  0.99    0.01             Death
## 49  case_86    Recover  0.02    0.98           Recover
## 50  case_88    Recover  0.02    0.98           Recover
## 51   case_9      Death  0.98    0.02             Death
## 52  case_90    Recover  0.08    0.92           Recover
## 53  case_92    Recover  0.04    0.96           Recover
## 54  case_93    Recover  0.43    0.57         uncertain
## 55  case_95    Recover  0.11    0.89           Recover
## 56  case_96      Death  0.68    0.32         uncertain
## 57  case_99    Recover  0.04    0.96           Recover
</code></pre>
</div>
<p></p>
<h2>Comparison with predicted outcome from last week’s analyses</h2>
<p></p>
<h3>Plotting predicted outcome</h3>
<p>Results from last week’s analysis come from the following part of the analysis:</p>
<div>
<pre><code><span>results</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>randomForest</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_rf</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>glmnet</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_glmnet</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>kknn</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_kknn</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>pda</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_pda</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>slda</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_slda</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>pam</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_pam</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>C</span><span>5.0</span><span>Tree</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_C5.0Tree</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>),</span><span>
</span><span>pls</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>model_pls</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test_data</span><span>,</span><span> </span><span>type</span><span>=</span><span>&quot;prob&quot;</span><span>))</span><span>
</span><span>results</span><span>$</span><span>sum_Death</span><span> </span><span>&lt;-</span><span> </span><span>rowSums</span><span>(</span><span>results</span><span>[,</span><span> </span><span>grep</span><span>(</span><span>&quot;Death&quot;</span><span>,</span><span> </span><span>colnames</span><span>(</span><span>results</span><span>))])</span><span>
</span><span>results</span><span>$</span><span>sum_Recover</span><span> </span><span>&lt;-</span><span> </span><span>rowSums</span><span>(</span><span>results</span><span>[,</span><span> </span><span>grep</span><span>(</span><span>&quot;Recover&quot;</span><span>,</span><span> </span><span>colnames</span><span>(</span><span>results</span><span>))])</span><span>
</span><span>results</span><span>$</span><span>log2_ratio</span><span> </span><span>&lt;-</span><span> </span><span>log2</span><span>(</span><span>results</span><span>$</span><span>sum_Recover</span><span>/</span><span>results</span><span>$</span><span>sum_Death</span><span>)</span><span>
</span><span>results</span><span>$</span><span>predicted_outcome</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>results</span><span>$</span><span>log2_ratio</span><span> </span><span>></span><span> </span><span>1.5</span><span>,</span><span> </span><span>&quot;Recover&quot;</span><span>,</span><span> </span><span>ifelse</span><span>(</span><span>results</span><span>$</span><span>log2_ratio</span><span> </span><span>&lt;</span><span> </span><span>-1.5</span><span>,</span><span> </span><span>&quot;Death&quot;</span><span>,</span><span> </span><span>&quot;uncertain&quot;</span><span>))</span><span>
</span><span>results</span><span>[,</span><span> </span><span>-</span><span>c</span><span>(</span><span>1</span><span>:</span><span>16</span><span>)]</span><span>
</span><span>results</span><span> </span><span>&lt;-</span><span> </span><span>results_last_week</span><span>
</span></code></pre>
</div>
<p>Combining the table with predicted outcomes from this and last week with the original data.</p>
<div>
<pre><code><span>result</span><span> </span><span>&lt;-</span><span> </span><span>merge</span><span>(</span><span>result</span><span>[,</span><span> </span><span>c</span><span>(</span><span>1</span><span>,</span><span> </span><span>5</span><span>)],</span><span> </span><span>results_last_week</span><span>[,</span><span> </span><span>ncol</span><span>(</span><span>results_last_week</span><span>),</span><span> </span><span>drop</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>],</span><span> </span><span>by.x</span><span> </span><span>=</span><span> </span><span>&quot;case_ID&quot;</span><span>,</span><span> </span><span>by.y</span><span> </span><span>=</span><span> </span><span>&quot;row.names&quot;</span><span>)</span><span>
</span><span>colnames</span><span>(</span><span>result</span><span>)[</span><span>2</span><span>:</span><span>3</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;predicted_outcome_xgboost&quot;</span><span>,</span><span> </span><span>&quot;predicted_outcome_last_week&quot;</span><span>)</span><span>
</span><span>results_combined</span><span> </span><span>&lt;-</span><span> </span><span>merge</span><span>(</span><span>result</span><span>,</span><span> </span><span>fluH7N9.china.2013</span><span>[</span><span>which</span><span>(</span><span>fluH7N9.china.2013</span><span>$</span><span>case.ID</span><span> </span><span>%in%</span><span> </span><span>result</span><span>$</span><span>case_ID</span><span>),</span><span> </span><span>],</span><span> </span><span>by.x</span><span> </span><span>=</span><span> </span><span>&quot;case_ID&quot;</span><span>,</span><span> </span><span>by.y</span><span> </span><span>=</span><span> </span><span>&quot;case.ID&quot;</span><span>)</span><span>
</span><span>results_combined</span><span> </span><span>&lt;-</span><span> </span><span>results_combined</span><span>[,</span><span> </span><span>-</span><span>c</span><span>(</span><span>6</span><span>,</span><span>7</span><span>)]</span><span>
</span></code></pre>
</div>
<p>For plotting with ggplot2, the dataframe needs to be gathered.</p>
<div>
<pre><code><span>library</span><span>(</span><span>tidyr</span><span>)</span><span>
</span><span>results_combined_gather</span><span> </span><span>&lt;-</span><span> </span><span>results_combined</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>group_dates</span><span>,</span><span> </span><span>date</span><span>,</span><span> </span><span>date.of.onset</span><span>:</span><span>date.of.hospitalisation</span><span>)</span><span>
</span><span>results_combined_gather</span><span>$</span><span>group_dates</span><span> </span><span>&lt;-</span><span> </span><span>factor</span><span>(</span><span>results_combined_gather</span><span>$</span><span>group_dates</span><span>,</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;date.of.onset&quot;</span><span>,</span><span> </span><span>&quot;date.of.hospitalisation&quot;</span><span>))</span><span>
</span><span>results_combined_gather</span><span>$</span><span>group_dates</span><span> </span><span>&lt;-</span><span> </span><span>mapvalues</span><span>(</span><span>results_combined_gather</span><span>$</span><span>group_dates</span><span>,</span><span> </span><span>from</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;date.of.onset&quot;</span><span>,</span><span> </span><span>&quot;date.of.hospitalisation&quot;</span><span>),</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Date of onset&quot;</span><span>,</span><span> </span><span>&quot;Date of hospitalisation&quot;</span><span>))</span><span>
</span><span>results_combined_gather</span><span>$</span><span>gender</span><span> </span><span>&lt;-</span><span> </span><span>mapvalues</span><span>(</span><span>results_combined_gather</span><span>$</span><span>gender</span><span>,</span><span> </span><span>from</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;f&quot;</span><span>,</span><span> </span><span>&quot;m&quot;</span><span>),</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Female&quot;</span><span>,</span><span> </span><span>&quot;Male&quot;</span><span>))</span><span>
</span><span>levels</span><span>(</span><span>results_combined_gather</span><span>$</span><span>gender</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>levels</span><span>(</span><span>results_combined_gather</span><span>$</span><span>gender</span><span>),</span><span> </span><span>&quot;unknown&quot;</span><span>)</span><span>
</span><span>results_combined_gather</span><span>$</span><span>gender</span><span>[</span><span>is.na</span><span>(</span><span>results_combined_gather</span><span>$</span><span>gender</span><span>)]</span><span> </span><span>&lt;-</span><span> </span><span>&quot;unknown&quot;</span><span>
</span><span>results_combined_gather</span><span> </span><span>&lt;-</span><span> </span><span>results_combined_gather</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>group_pred</span><span>,</span><span> </span><span>prediction</span><span>,</span><span> </span><span>predicted_outcome_xgboost</span><span>:</span><span>predicted_outcome_last_week</span><span>)</span><span>
</span><span>results_combined_gather</span><span>$</span><span>group_pred</span><span> </span><span>&lt;-</span><span> </span><span>mapvalues</span><span>(</span><span>results_combined_gather</span><span>$</span><span>group_pred</span><span>,</span><span> </span><span>from</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;predicted_outcome_xgboost&quot;</span><span>,</span><span> </span><span>&quot;predicted_outcome_last_week&quot;</span><span>),</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Predicted outcome from XGBoost&quot;</span><span>,</span><span> </span><span>&quot;Predicted outcome from last week&quot;</span><span>))</span><span>
</span></code></pre>
</div>
<p>Setting a custom theme for plotting:</p>
<div>
<pre><code><span>library</span><span>(</span><span>ggplot2</span><span>)</span><span>
</span><span>my_theme</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>base_size</span><span> </span><span>=</span><span> </span><span>12</span><span>,</span><span> </span><span>base_family</span><span> </span><span>=</span><span> </span><span>&quot;sans&quot;</span><span>){</span><span>
</span><span>theme_minimal</span><span>(</span><span>base_size</span><span> </span><span>=</span><span> </span><span>base_size</span><span>,</span><span> </span><span>base_family</span><span> </span><span>=</span><span> </span><span>base_family</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>
</span><span>axis.text</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>12</span><span>),</span><span>
</span><span>axis.text.x</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>angle</span><span> </span><span>=</span><span> </span><span>45</span><span>,</span><span> </span><span>vjust</span><span> </span><span>=</span><span> </span><span>0.5</span><span>,</span><span> </span><span>hjust</span><span> </span><span>=</span><span> </span><span>0.5</span><span>),</span><span>
</span><span>axis.title</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>14</span><span>),</span><span>
</span><span>panel.grid.major</span><span> </span><span>=</span><span> </span><span>element_line</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>),</span><span>
</span><span>panel.grid.minor</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>panel.background</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;aliceblue&quot;</span><span>),</span><span>
</span><span>strip.background</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;lightgrey&quot;</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1</span><span>),</span><span>
</span><span>strip.text</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>face</span><span> </span><span>=</span><span> </span><span>&quot;bold&quot;</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>12</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;black&quot;</span><span>),</span><span>
</span><span>legend.position</span><span> </span><span>=</span><span> </span><span>&quot;bottom&quot;</span><span>,</span><span>
</span><span>legend.justification</span><span> </span><span>=</span><span> </span><span>&quot;top&quot;</span><span>,</span><span> </span><span>legend.box</span><span> </span><span>=</span><span> </span><span>&quot;horizontal&quot;</span><span>,</span><span>
</span><span>legend.box.background</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>colour</span><span> </span><span>=</span><span> </span><span>&quot;grey50&quot;</span><span>),</span><span>
</span><span>legend.background</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>panel.border</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>NA</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>0.5</span><span>),</span><span>
</span><span>panel.spacing</span><span> </span><span>=</span><span> </span><span>unit</span><span>(</span><span>1</span><span>,</span><span> </span><span>&quot;lines&quot;</span><span>)</span><span>
</span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>results_combined_gather</span><span>$</span><span>province</span><span> </span><span>&lt;-</span><span> </span><span>mapvalues</span><span>(</span><span>results_combined_gather</span><span>$</span><span>province</span><span>,</span><span> </span><span>from</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Anhui&quot;</span><span>,</span><span> </span><span>&quot;Beijing&quot;</span><span>,</span><span> </span><span>&quot;Fujian&quot;</span><span>,</span><span> </span><span>&quot;Guangdong&quot;</span><span>,</span><span> </span><span>&quot;Hebei&quot;</span><span>,</span><span> </span><span>&quot;Henan&quot;</span><span>,</span><span> </span><span>&quot;Hunan&quot;</span><span>,</span><span> </span><span>&quot;Jiangxi&quot;</span><span>,</span><span> </span><span>&quot;Shandong&quot;</span><span>,</span><span> </span><span>&quot;Taiwan&quot;</span><span>),</span><span> </span><span>to</span><span> </span><span>=</span><span> </span><span>rep</span><span>(</span><span>&quot;Other&quot;</span><span>,</span><span> </span><span>10</span><span>))</span><span>
</span><span>levels</span><span>(</span><span>results_combined_gather</span><span>$</span><span>gender</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>levels</span><span>(</span><span>results_combined_gather</span><span>$</span><span>gender</span><span>),</span><span> </span><span>&quot;unknown&quot;</span><span>)</span><span>
</span><span>results_combined_gather</span><span>$</span><span>gender</span><span>[</span><span>is.na</span><span>(</span><span>results_combined_gather</span><span>$</span><span>gender</span><span>)]</span><span> </span><span>&lt;-</span><span> </span><span>&quot;unknown&quot;</span><span>
</span><span>results_combined_gather</span><span>$</span><span>province</span><span> </span><span>&lt;-</span><span> </span><span>factor</span><span>(</span><span>results_combined_gather</span><span>$</span><span>province</span><span>,</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Jiangsu&quot;</span><span>,</span><span> </span><span>&quot;Shanghai&quot;</span><span>,</span><span> </span><span>&quot;Zhejiang&quot;</span><span>,</span><span> </span><span>&quot;Other&quot;</span><span>))</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>ggplot</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>subset</span><span>(</span><span>results_combined_gather</span><span>,</span><span> </span><span>group_dates</span><span> </span><span>==</span><span> </span><span>&quot;Date of onset&quot;</span><span>),</span><span> </span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>date</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>age</span><span>),</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>prediction</span><span>))</span><span> </span><span>+</span><span>
</span><span>stat_density2d</span><span>(</span><span>aes</span><span>(</span><span>alpha</span><span> </span><span>=</span><span> </span><span>..level..</span><span>),</span><span> </span><span>geom</span><span> </span><span>=</span><span> </span><span>&quot;polygon&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_jitter</span><span>(</span><span>aes</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>prediction</span><span>,</span><span> </span><span>shape</span><span> </span><span>=</span><span> </span><span>gender</span><span>),</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_rug</span><span>(</span><span>aes</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>prediction</span><span>))</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>
</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;Predicted outcome&quot;</span><span>,</span><span>
</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;Predicted outcome&quot;</span><span>,</span><span>
</span><span>alpha</span><span> </span><span>=</span><span> </span><span>&quot;Level&quot;</span><span>,</span><span>
</span><span>shape</span><span> </span><span>=</span><span> </span><span>&quot;Gender&quot;</span><span>,</span><span>
</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;Date of onset in 2013&quot;</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;Age&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;2013 Influenza A H7N9 cases in China&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;Predicted outcome of cases with unknown outcome&quot;</span><span>,</span><span>
</span><span>caption</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>
</span><span>)</span><span> </span><span>+</span><span>
</span><span>facet_grid</span><span>(</span><span>group_pred</span><span> </span><span>~</span><span> </span><span>province</span><span>)</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_shape_manual</span><span>(</span><span>values</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>15</span><span>,</span><span> </span><span>16</span><span>,</span><span> </span><span>17</span><span>))</span><span> </span><span>+</span><span>
</span><span>scale_color_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>,</span><span> </span><span>na.value</span><span> </span><span>=</span><span> </span><span>&quot;grey50&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span>=</span><span>&quot;Set1&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<p><img src="https://i2.wp.com/shiring.github.io/machine_learning/2016/12/02/flu_outcome_ML_2_post_files/figure-markdown_github/unnamed-chunk-37-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>There is much less uncertainty in the XGBoost data, even tough I used slightly different methods for classifying uncertainty: In last week’s analysis I based uncertainty on the ratio of combined prediction values from all analyses, this week uncertainty is based on the prediction value from one analysis.</p>
<p></p>
<hr>
<div>
<pre><code><span>sessionInfo</span><span>()</span><span>
</span></code></pre>
</div>
<div>
<pre><code>## R version 3.3.2 (2016-10-31)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 7 x64 (build 7601) Service Pack 1
## 
## locale:
## [1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252 LC_NUMERIC=C                           LC_TIME=English_United States.1252    
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] tidyr_0.6.0     plyr_1.8.4      xgboost_0.6-0   caret_6.0-73    ggplot2_2.2.0   lattice_0.20-34 mice_2.25       Rcpp_0.12.8     dplyr_0.5.0     outbreaks_1.0.0
## 
## loaded via a namespace (and not attached):
##  [1] Ckmeans.1d.dp_3.4.6-4 RColorBrewer_1.1-2    compiler_3.3.2        nloptr_1.0.4          class_7.3-14          iterators_1.0.8       tools_3.3.2           rpart_4.1-10          digest_0.6.10         lme4_1.1-12           evaluate_0.10         tibble_1.2            gtable_0.2.0          nlme_3.1-128          mgcv_1.8-16           Matrix_1.2-7.1        foreach_1.4.3         DBI_0.5-1             parallel_3.3.2        yaml_2.1.14           SparseM_1.74          e1071_1.6-7           stringr_1.1.0         knitr_1.15            MatrixModels_0.4-1    stats4_3.3.2          grid_3.3.2            nnet_7.3-12           data.table_1.9.6      R6_2.2.0              survival_2.40-1       rmarkdown_1.1         minqa_1.2.4           reshape2_1.4.2        car_2.1-3             magrittr_1.5          scales_0.4.1          codetools_0.2-15      ModelMetrics_1.1.0    htmltools_0.3.5       MASS_7.3-45           splines_3.3.2         assertthat_0.1        pbkrtest_0.4-6        colorspace_1.3-0      labeling_0.3          quantreg_5.29         stringi_1.1.2         lazyeval_0.2.0        munsell_0.4.3         chron_2.3-47
</code></pre>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/extreme-gradient-boosting-and-preprocessing-in-machine-learning-addendum-to-predicting-flu-outcome-with-r/"></div></div><div><a data-url="https://www.r-bloggers.com/extreme-gradient-boosting-and-preprocessing-in-machine-learning-addendum-to-predicting-flu-outcome-with-r/" data-counturl="https://www.r-bloggers.com/extreme-gradient-boosting-and-preprocessing-in-machine-learning-addendum-to-predicting-flu-outcome-with-r/" data-count="vertical" data-text="Extreme Gradient Boosting and Preprocessing in Machine Learning – Addendum to predicting flu outcome with R" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://shiring.github.io/machine_learning/2016/12/02/flu_outcome_ML_2_post"> Shirin's playgRound</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/I1oKFpTMWoE"></a></div>