<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/weighting-tricks-for-machine-learning-with-icarus-part-1/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://nc233.com/2018/07/weighting-tricks-for-machine-learning-with-icarus-part-1/"> R – NC233</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div class="twitter-share"><a rel="nofollow" href="https://twitter.com/intent/tweet?url=http%3A%2F%2Fnc233.com%2F2018%2F07%2Fweighting-tricks-for-machine-learning-with-icarus-part-1%2F&via=nc233" class="twitter-share-button" target="_blank">Tweet</a></div>
<p><a rel="nofollow" href="https://www.tandfonline.com/doi/abs/10.1080/01621459.1992.10475217" target="_blank">Calibration in survey sampling</a> is a wonderful tool, and today I want to show you how we can use it in some Machine Learning applications, using the R package <a rel="nofollow" href="https://cran.r-project.org/web/packages/icarus/" target="_blank">Icarus</a>. And because <a rel="nofollow" href="https://en.wikipedia.org/wiki/2018_FIFA_World_Cup" target="_blank">’tis the season</a>, what better than a soccer dataset to illustrate this? The data and code are located on this gitlab repo: <a rel="nofollow" href="https://gitlab.com/haroine/weighting-ml" target="_blank">https://gitlab.com/haroine/weighting-ml</a></p>
<figure class="wp-caption"><a rel="nofollow" href="https://gitlab.com/haroine/weighting-ml" target="_blank"><img sizes="(max-width: 300px) 100vw, 300px" src="https://i0.wp.com/nc233.com/wp-content/uploads/2018/07/gitlab-300x83.png?resize=300%2C83" alt="weighting ML gitlab" width="300" class="wp-image-1481" data-recalc-dims="1" height="83"></a><figcaption class="wp-caption-text">https://gitlab.com/haroine/weighting-ml</figcaption></figure>
<p>First, let’s start by installing and loading icarus and nnet, the two packages needed in this tutorial, from CRAN (if necessary):</p>
<pre>install.packages(c(&quot;icarus&quot;,&quot;nnet&quot;))
library(icarus)
library(nnet)
</pre>
<p>Then load the data:</p>
<pre>load(&quot;data/weighting_ML_part1.RData&quot;)
</pre>
<p>The RData file contains two dataframes, one for the training set and one for the test set. They contain results of some international soccer games, from 01/2008 to 12/2016 for the training set, and from 01/2017 to 11/2017 for the test. Along with the team names and goals scored for each side, a few descriptive variables that we’re going to use as features of our ML models:</p>
<pre tabindex="0"><span>> </span><span>head(train_soccer)
</span>        Date                   team opponent_team home_field elo_team
1 2010-10-12                Belarus       Albania          1      554
2 2010-10-08 Bosnia and Herzegovina       Albania          0      544
3 2011-06-07 Bosnia and Herzegovina       Albania          0      594
4 2011-06-20              Argentina       Albania          1     1267
5 2011-08-10             Montenegro       Albania          0      915
6 2011-09-02                 France       Albania          0      918
  opponent_elo importance goals_for goals_against outcome year
1          502          1         2             0     WIN 2010
2          502          1         1             1    DRAW 2010
3          564          1         2             0     WIN 2011
4          564          1         4             0     WIN 2011
5          524          1         2             3    LOSS 2011
6          546          1         2             1     WIN 2011</pre>
<p><em>elo_team</em> and <em>opponent_elo</em> are quantitative variables indicative of the level of the team at the date of the game ; <em>importance</em> is a measure of high-profile the game played was (a friendly match rates 1 while a World Cup game rates 4). The other variables are imo self-descriptive.</p>
<p>Then we can train a multinomial logistic regression, with <em>outcome</em> being the predicted variable, and compute the predictions from the model:</p>
<pre>outcome_model_unw &lt;- multinom(outcome ~ elo_team + opponent_elo + home_field + importance,
data = train_soccer)

test_soccer$pred_outcome_unw &lt;- predict(outcome_model_unw, newdata = test_soccer)</pre>
<p>The sheer accuracy of this predictor is kinda good:</p>
<pre tabindex="0"><span>> </span><span>## Accuracy
</span><span>> </span><span>sum(test_soccer$pred_outcome_unw == test_soccer$outcome) / nrow(test_soccer)
</span>[1] 0.5526316</pre>
<p>but it has a problem: it never predicts draws!</p>
<pre tabindex="0"><span>> </span><span>summary(test_soccer$pred_outcome_unw)
</span>DRAW LOSS  WIN 
   0  208  210</pre>
<p>And indeed, draws being less common than other results, it seems more profitable for the algorithm that optimizes accuracy never to predict them. As a consequence, the probabilities of the game being a draw is always lesser than the probability of one team winning it. We could show that the probabilities are <a rel="nofollow" href="http://scikit-learn.org/stable/modules/calibration.html" target="_blank">not well calibrated</a>.</p>
<p>A common solution to this problem is to use reweighting to correct the imbalances in the sample, which we’ll now tackle. It is important to note that the weighting trick has to happen in the training set to avoid “data leaks”. A very good piece on this subject has been written by Max Kuhn <a rel="nofollow" href="https://topepo.github.io/caret/subsampling-for-class-imbalances.html" target="_blank">in the documentation of caret</a>.</p>
<figure class="wp-caption"><a rel="nofollow" href="https://topepo.github.io/caret/subsampling-for-class-imbalances.html" target="_blank"><img sizes="(max-width: 300px) 100vw, 300px" src="https://i0.wp.com/nc233.com/wp-content/uploads/2018/07/caret-300x134.png?resize=300%2C134" alt="R package caret" width="300" class="wp-image-1480" data-recalc-dims="1" height="134"></a><figcaption class="wp-caption-text">https://topepo.github.io/caret/</figcaption></figure>
<p>Commonly, you would do:</p>
<pre tabindex="0">train_soccer$weight &lt;- 1
train_soccer[train_soccer$outcome == &quot;DRAW&quot;,]$weight &lt;- (nrow(train_soccer)/table(train_soccer$outcome)[1]) * 1/3
train_soccer[train_soccer$outcome == &quot;LOSS&quot;,]$weight &lt;- (nrow(train_soccer)/table(train_soccer$outcome)[2]) * 1/3
train_soccer[train_soccer$outcome == &quot;WIN&quot;,]$weight &lt;- (nrow(train_soccer)/table(train_soccer$outcome)[3]) * 1/3

</pre>
<pre tabindex="0"><span>> </span><span>table(train_soccer$weight)
</span>
0.916067146282974  1.22435897435897 
             3336              1248</pre>
<p>The draws are reweighted with a factor greater than 1 and the other games with a factor lesser than 1. This balances the predicted outcomes and thus improves the quality of the probabilities …</p>
<pre tabindex="0"><span>outcome_model &lt;-multinom(outcome ~ elo_team + opponent_elo + home_field + importance,
data = train_soccer,
weights = train_soccer$weight)
test_soccer$pred_outcome &lt;-predict(outcome_model, newdata = test_soccer)
</span></pre>
<pre tabindex="0"><span>> </span><span>summary(test_soccer$pred_outcome)
</span>DRAW LOSS  WIN 
  96  167  155</pre>
<p>… though at a loss in accuracy:</p>
<pre tabindex="0"><span>> </span><span>## Accuracy
</span><span>> </span><span>sum(test_soccer$pred_outcome == test_soccer$outcome) / nrow(test_soccer)
</span>[1] 0.5263158</pre>
<p>Now let’s look at the balance of our training sample on other variables:</p>
<pre tabindex="0"><span>> </span><span>round(table(test_soccer$importance) / nrow(test_soccer),2)
</span>
   1    2    3    4 
0.26 0.08 0.54 0.12 
<span>> </span><span>round(table(train_soccer$importance) / nrow(train_soccer),2)
</span>
   1    2    3    4 
0.56 0.08 0.23 0.12</pre>
<p>It seems that the test set features a lot more important matches than the training set. Let’s look further, in particular at the dates the matches of the training set were played:</p>
<pre tabindex="0"><span>> </span><span>round(table(train_soccer$year) / nrow(train_soccer),2)
</span>
2008 2009 2010 2011 2012 2013 2014 2015 2016 
0.10 0.11 0.11 0.10 0.11 0.13 0.11 0.11 0.12</pre>
<p>Thus the matches of each year between 2008 and 2016 have the same influence on the final predictor. A better idea would be to give the most recent games a slightly higher influence, for example by increasing their weight and thus reducing the weights of the older games:</p>
<pre tabindex="0">nyears &lt;- length(unique(train_soccer$year))
year_tweak &lt;- rep(1/nyears,nyears) * 1:nyears
year_tweak &lt;- year_tweak * 1/sum(year_tweak) ## Normalization

</pre>
<pre tabindex="0"><span>> </span><span>year_tweak
</span>[1] 0.02222222 0.04444444 0.06666667 0.08888889 0.11111111 0.13333333
[7] 0.15555556 0.17777778 0.20000000</pre>
<p>We determine it is thus a good idea to balance on these two additional variables (<em>year</em> and <em>importance</em>). Now how should we do this? A solution could be to create an indicator variable containing all the values of the cross product between the variables <em>outcome</em>, <em>year</em> and <em>importance</em>, and use the same reweighting technique as before. But this would not be very practical and more importantly, some of the sub-categories would be nearly empty, making the procedure not very robust. A better solution is to use survey sampling calibration and Icarus <img alt="🙂" src="https://i1.wp.com/s.w.org/images/core/emoji/2.4/72x72/1f642.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"></p>
<pre tabindex="0">train_soccer$weight_cal &lt;- 1
importance_pct_test &lt;- unname(
table(test_soccer$importance) / nrow(test_soccer),
)

marginMatrix &lt;- matrix(, nrow = 0, ncol = 1) %>% ## Will be replaced by newMarginMatrix() in icarus 0.3.2
addMargin(&quot;outcome&quot;, c(0.333,0.333,0.333)) %>%
addMargin(&quot;importance&quot;, importance_pct_test) %>%
addMargin(&quot;year&quot;, year_tweak)

train_soccer$weight_cal &lt;- calibration(data=train_soccer, marginMatrix=marginMatrix,
colWeights=&quot;weight_cal&quot;, pct=TRUE, description=TRUE,
popTotal = nrow(train_soccer), method=&quot;raking&quot;)

outcome_model_cal &lt;- multinom(outcome ~ elo_team + opponent_elo + home_field + importance,
data = train_soccer,
weights = train_soccer$weight_cal)

test_soccer$pred_outcome_cal &lt;- predict(outcome_model_cal, newdata = test_soccer)</pre>
<p>icarus gives a summary of the calibration procedure in the log (too long to reproduce here). We then observe a slight improvement in accuracy compared to the previous reweighting technique:</p>
<pre tabindex="0"><span>> </span><span>sum(test_soccer$pred_outcome_cal == test_soccer$outcome) / nrow(test_soccer)
</span>[1] 0.5478469</pre>
<p>But more importantly we have reason to believe that the we improved the quality of the probabilities assigned to each event (we could check this using metrics such as the Brier score or calibration plots) <img alt="🙂" src="https://i1.wp.com/s.w.org/images/core/emoji/2.4/72x72/1f642.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"></p>
<p>It is also worth noting that some algorithms (especially those who rely on bagging, boosting, or more generally on ensemble methods) naturally do a good job at balancing samples. You could for example rerun the whole code and replace the logit regressions by boosted algorithms. You would then observe fewer differences between the unweighted algorithm and its weighted counterparts.</p>
<p>Stay tuned for the part 2, where we’ll show a trick to craft better probabilities (particularly for simulations) using external knowledge on probabilities.</p>
<div class="twitter-share"><a rel="nofollow" href="https://twitter.com/intent/tweet?url=http%3A%2F%2Fnc233.com%2F2018%2F07%2Fweighting-tricks-for-machine-learning-with-icarus-part-1%2F&via=nc233" class="twitter-share-button" target="_blank">Tweet</a></div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://nc233.com/2018/07/weighting-tricks-for-machine-learning-with-icarus-part-1/"> R – NC233</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/R4cXp-JTQig" height="1"></a></div>