<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/handling-class-imbalance-with-r-and-caret-an-introduction/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://dpmartin42.github.io/blogposts/r/imbalanced-classes-part-1"> Wicked Good Data - r</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>When faced with classification tasks in the real world, it can be challenging to deal with an outcome where one class heavily outweighs the other (a.k.a., imbalanced classes). The following will be a two-part post on some of the techniques that can help to improve prediction performance in the case of imbalanced classes using R and caret. This first post provides a general overview of how these techniques can be implemented in practice, and the second post highlights some caveats to keep in mind when using these methods.</p>
<h2>Evaluation metrics for classifiers</h2>
<p>After building a classifier, you need to decide how to tell if it is doing a good job or not. Many evaluation metrics for classifiers exist, and can generally be divided into two main groups:</p>
<ol>
<li>
<p>Threshold-dependent: This includes metrics like accuracy, precision, recall, and F1 score, which all require a confusion matrix to be calculated using a hard cutoff on predicted probabilities. These metrics are typically quite poor in the case of imbalanced classes, as statistical software inappropriately uses a default threshold of 0.50 resulting in the model predicting that all observations belong in the majority class.</p>
</li>
<li>
<p>Threshold-invariant: This includes metrics like area under the ROC curve (AUC), which quantifies true positive rate as a function of false positive rate for a variety of classification thresholds. Another way to interpret this metric is the probability that a random positive instance will have a higher estimated probability than a random negative instance.</p>
</li>
</ol>
<h2>Methods to improve performance on imbalanced data</h2>
<p>A few of the more popular techniques to deal with class imbalance will be covered below, but the following list is nowhere near exhaustive. For brevity, a quick overview is provided. For a more substantial overview, I highly recommend this <a target="_blank" rel="nofollow" href="https://svds.com/learning-imbalanced-classes/">Silicon Valley Data Science blog post</a>.</p>
<ul>
<li>
<p>Class weights: impose a heavier cost when errors are made in the minority class</p>
</li>
<li>
<p>Down-sampling: randomly remove instances in the majority class</p>
</li>
<li>
<p>Up-sampling: randomly replicate instances in the minority class</p>
</li>
<li>
<p>Synthetic minority sampling technique (SMOTE): down samples the majority class and synthesizes new minority instances by interpolating between existing ones</p>
</li>
</ul>
<p>It is important to note that these weighting and sampling techniques have the biggest impact on threshold-dependent metrics like accuracy, because they artificially move the threshold to be closer to what might be considered as the “optimal” location on a ROC curve. Threshold-invariant metrics can still be improved using these methods, but the effect will not be as pronounced.</p>
<h2>Simulation set-up</h2>
<p>To simulate class imbalance, the <code>twoClassSim</code> function from caret is used. Here, we simulate a separate training set and test set, each with 5000 observations. Additionally, we include 20 meaningful variables and 10 noise variables. The intercept argument controls the overall level of class imbalance and has been selected to yield a class imbalance of around 50:1.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>dplyr</span><span>)</span><span> </span><span># for data manipulation
</span><span>library</span><span>(</span><span>caret</span><span>)</span><span> </span><span># for model-building
</span><span>library</span><span>(</span><span>DMwR</span><span>)</span><span> </span><span># for smote implementation
</span><span>library</span><span>(</span><span>purrr</span><span>)</span><span> </span><span># for functional programming (map)
</span><span>library</span><span>(</span><span>pROC</span><span>)</span><span> </span><span># for AUC calculations
</span><span>
</span><span>set.seed</span><span>(</span><span>2969</span><span>)</span><span>
</span><span>imbal_train</span><span> </span><span>&lt;-</span><span> </span><span>twoClassSim</span><span>(</span><span>5000</span><span>,</span><span>
</span><span>intercept</span><span> </span><span>=</span><span> </span><span>-25</span><span>,</span><span>
</span><span>linearVars</span><span> </span><span>=</span><span> </span><span>20</span><span>,</span><span>
</span><span>noiseVars</span><span> </span><span>=</span><span> </span><span>10</span><span>)</span><span>
</span><span>imbal_test</span><span> </span><span>&lt;-</span><span> </span><span>twoClassSim</span><span>(</span><span>5000</span><span>,</span><span>
</span><span>intercept</span><span> </span><span>=</span><span> </span><span>-25</span><span>,</span><span>
</span><span>linearVars</span><span> </span><span>=</span><span> </span><span>20</span><span>,</span><span>
</span><span>noiseVars</span><span> </span><span>=</span><span> </span><span>10</span><span>)</span><span>
</span><span>prop.table</span><span>(</span><span>table</span><span>(</span><span>imbal_train</span><span>$</span><span>Class</span><span>))</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## 
## Class1 Class2 
## 0.9796 0.0204</code></pre>
</figure>
<h2>Initial results</h2>
<p>To model these data, a gradient boosting machine (gbm) is used as it can easily handle potential interactions and non-linearities that have been simulated above. Model hyperparameters are tuned using repeated cross-validation on the training set, repeating five times with ten folds used in each repeat. The AUC is used to evaluate the classifier to avoid having to make decisions about the classification threshold. Note that this code takes a little while to run due to the repeated cross-validation, so reduce the number of repeats to speed things up and/or use the <code>verboseIter = TRUE</code> argument in the <code>trainControl</code> function to keep track of the progress.</p>
<figure>
<pre><code data-lang="r"><span># Set up control function for training
</span><span>
</span><span>ctrl</span><span> </span><span>&lt;-</span><span> </span><span>trainControl</span><span>(</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;repeatedcv&quot;</span><span>,</span><span>
</span><span>number</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span>
</span><span>repeats</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span>
</span><span>summaryFunction</span><span> </span><span>=</span><span> </span><span>twoClassSummary</span><span>,</span><span>
</span><span>classProbs</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span># Build a standard classifier using a gradient boosted machine
</span><span>
</span><span>set.seed</span><span>(</span><span>5627</span><span>)</span><span>
</span><span>orig_fit</span><span> </span><span>&lt;-</span><span> </span><span>train</span><span>(</span><span>Class</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>imbal_train</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;gbm&quot;</span><span>,</span><span>
</span><span>verbose</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span>
</span><span>metric</span><span> </span><span>=</span><span> </span><span>&quot;ROC&quot;</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>ctrl</span><span>)</span><span>
</span><span># Build custom AUC function to extract AUC
# from the caret model object
</span><span>
</span><span>test_roc</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>model</span><span>,</span><span> </span><span>data</span><span>)</span><span> </span><span>{</span><span>
</span><span>roc</span><span>(</span><span>data</span><span>$</span><span>Class</span><span>,</span><span>
</span><span>predict</span><span>(</span><span>model</span><span>,</span><span> </span><span>data</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>&quot;prob&quot;</span><span>)[,</span><span> </span><span>&quot;Class2&quot;</span><span>])</span><span>
</span><span>}</span><span>
</span><span>orig_fit</span><span> </span><span>%>%</span><span>
</span><span>test_roc</span><span>(</span><span>data</span><span> </span><span>=</span><span> </span><span>imbal_test</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>auc</span><span>()</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## Area under the curve: 0.9575</code></pre>
</figure>
<p>Overall, the final model yields an AUC of 0.96 which is quite good. Can we improve it using the techniques outlined above?</p>
<h2>Handling class imbalance with weighted or sampling methods</h2>
<p>Both weighting and sampling methods are easy to employ in caret. Incorporating weights into the model can be handled by using the weights argument in the <code>train</code> function (assuming the model can handle weights in caret, see the list <a target="_blank" rel="nofollow" href="https://topepo.github.io/caret/train-models-by-tag.html#Accepts_Case_Weights">here</a>), while the sampling methods mentioned above can be implemented using the sampling argument in the <code>trainControl</code> function. Note that the same seeds were used for each model to ensure that results from the same cross-validation folds are being used.</p>
<p>Also keep in mind that for sampling methods, it is vital that you only sample the training set and not the test set as well. This means that when doing cross-validation, the sampling step must be done inside of the cross-validation procedure. Max Kuhn of the caret package gives a good overview of what happens when you don’t take this precaution in <a target="_blank" rel="nofollow" href="https://topepo.github.io/caret/subsampling-for-class-imbalances.html">this caret documentation</a>. Using the sampling argument in the <code>trainControl</code> function implements sampling correctly in the cross-validation procedure.</p>
<figure>
<pre><code data-lang="r"><span># Create model weights (they sum to one)
</span><span>
</span><span>model_weights</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>imbal_train</span><span>$</span><span>Class</span><span> </span><span>==</span><span> </span><span>&quot;Class1&quot;</span><span>,</span><span>
</span><span>(</span><span>1</span><span>/</span><span>table</span><span>(</span><span>imbal_train</span><span>$</span><span>Class</span><span>)[</span><span>1</span><span>])</span><span> </span><span>*</span><span> </span><span>0.5</span><span>,</span><span>
</span><span>(</span><span>1</span><span>/</span><span>table</span><span>(</span><span>imbal_train</span><span>$</span><span>Class</span><span>)[</span><span>2</span><span>])</span><span> </span><span>*</span><span> </span><span>0.5</span><span>)</span><span>
</span><span># Use the same seed to ensure same cross-validation splits
</span><span>
</span><span>ctrl</span><span>$</span><span>seeds</span><span> </span><span>&lt;-</span><span> </span><span>orig_fit</span><span>$</span><span>control</span><span>$</span><span>seeds</span><span>
</span><span># Build weighted model
</span><span>
</span><span>weighted_fit</span><span> </span><span>&lt;-</span><span> </span><span>train</span><span>(</span><span>Class</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>imbal_train</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;gbm&quot;</span><span>,</span><span>
</span><span>verbose</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span>
</span><span>weights</span><span> </span><span>=</span><span> </span><span>model_weights</span><span>,</span><span>
</span><span>metric</span><span> </span><span>=</span><span> </span><span>&quot;ROC&quot;</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>ctrl</span><span>)</span><span>
</span><span># Build down-sampled model
</span><span>
</span><span>ctrl</span><span>$</span><span>sampling</span><span> </span><span>&lt;-</span><span> </span><span>&quot;down&quot;</span><span>
</span><span>down_fit</span><span> </span><span>&lt;-</span><span> </span><span>train</span><span>(</span><span>Class</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>imbal_train</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;gbm&quot;</span><span>,</span><span>
</span><span>verbose</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span>
</span><span>metric</span><span> </span><span>=</span><span> </span><span>&quot;ROC&quot;</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>ctrl</span><span>)</span><span>
</span><span># Build up-sampled model
</span><span>
</span><span>ctrl</span><span>$</span><span>sampling</span><span> </span><span>&lt;-</span><span> </span><span>&quot;up&quot;</span><span>
</span><span>up_fit</span><span> </span><span>&lt;-</span><span> </span><span>train</span><span>(</span><span>Class</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>imbal_train</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;gbm&quot;</span><span>,</span><span>
</span><span>verbose</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span>
</span><span>metric</span><span> </span><span>=</span><span> </span><span>&quot;ROC&quot;</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>ctrl</span><span>)</span><span>
</span><span># Build smote model
</span><span>
</span><span>ctrl</span><span>$</span><span>sampling</span><span> </span><span>&lt;-</span><span> </span><span>&quot;smote&quot;</span><span>
</span><span>smote_fit</span><span> </span><span>&lt;-</span><span> </span><span>train</span><span>(</span><span>Class</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>imbal_train</span><span>,</span><span>
</span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;gbm&quot;</span><span>,</span><span>
</span><span>verbose</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span>
</span><span>metric</span><span> </span><span>=</span><span> </span><span>&quot;ROC&quot;</span><span>,</span><span>
</span><span>trControl</span><span> </span><span>=</span><span> </span><span>ctrl</span><span>)</span></code></pre>
</figure>
<p>Examining the AUC calculated on the test set shows a clear distinction between the original model implementation and those that incorporated either a weighting or sampling technique. The weighted method possessed the highest AUC value, followed by the sampling methods, with the original model implementation performing the worst.</p>
<figure>
<pre><code data-lang="r"><span># Examine results for test set
</span><span>
</span><span>model_list</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>original</span><span> </span><span>=</span><span> </span><span>orig_fit</span><span>,</span><span>
</span><span>weighted</span><span> </span><span>=</span><span> </span><span>weighted_fit</span><span>,</span><span>
</span><span>down</span><span> </span><span>=</span><span> </span><span>down_fit</span><span>,</span><span>
</span><span>up</span><span> </span><span>=</span><span> </span><span>up_fit</span><span>,</span><span>
</span><span>SMOTE</span><span> </span><span>=</span><span> </span><span>smote_fit</span><span>)</span><span>
</span><span>model_list_roc</span><span> </span><span>&lt;-</span><span> </span><span>model_list</span><span> </span><span>%>%</span><span>
</span><span>map</span><span>(</span><span>test_roc</span><span>,</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>imbal_test</span><span>)</span><span>
</span><span>model_list_roc</span><span> </span><span>%>%</span><span>
</span><span>map</span><span>(</span><span>auc</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## $original
## Area under the curve: 0.9575
## 
## $weighted
## Area under the curve: 0.9804
## 
## $down
## Area under the curve: 0.9705
## 
## $up
## Area under the curve: 0.9759
## 
## $SMOTE
## Area under the curve: 0.976</code></pre>
</figure>
<p>We can examine the actual ROC curve to get a better idea of where the weighted and sampling models are outperforming the original model at a variety of classification thresholds. Here, we see that the weighted model seems to dominate the others throughout, while the original model lags between a false positive rate between 0% and 25%. This indicates that the other models have better early retrieval numbers. That is, the algorithm better identifies the true positives as a function of false positives for instances that are predicted as having a high probability of being in the minority class.</p>
<figure>
<pre><code data-lang="r"><span>results_list_roc</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>NA</span><span>)</span><span>
</span><span>num_mod</span><span> </span><span>&lt;-</span><span> </span><span>1</span><span>
</span><span>for</span><span>(</span><span>the_roc</span><span> </span><span>in</span><span> </span><span>model_list_roc</span><span>){</span><span>
</span><span>results_list_roc</span><span>[[</span><span>num_mod</span><span>]]</span><span> </span><span>&lt;-</span><span> </span><span>data_frame</span><span>(</span><span>tpr</span><span> </span><span>=</span><span> </span><span>the_roc</span><span>$</span><span>sensitivities</span><span>,</span><span>
</span><span>fpr</span><span> </span><span>=</span><span> </span><span>1</span><span> </span><span>-</span><span> </span><span>the_roc</span><span>$</span><span>specificities</span><span>,</span><span>
</span><span>model</span><span> </span><span>=</span><span> </span><span>names</span><span>(</span><span>model_list</span><span>)[</span><span>num_mod</span><span>])</span><span>
</span><span>num_mod</span><span> </span><span>&lt;-</span><span> </span><span>num_mod</span><span> </span><span>+</span><span> </span><span>1</span><span>
</span><span>}</span><span>
</span><span>results_df_roc</span><span> </span><span>&lt;-</span><span> </span><span>bind_rows</span><span>(</span><span>results_list_roc</span><span>)</span><span>
</span><span># Plot ROC curve for all 5 models
</span><span>
</span><span>custom_col</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;#000000&quot;</span><span>,</span><span> </span><span>&quot;#009E73&quot;</span><span>,</span><span> </span><span>&quot;#0072B2&quot;</span><span>,</span><span> </span><span>&quot;#D55E00&quot;</span><span>,</span><span> </span><span>&quot;#CC79A7&quot;</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>fpr</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>tpr</span><span>,</span><span> </span><span>group</span><span> </span><span>=</span><span> </span><span>model</span><span>),</span><span> </span><span>data</span><span> </span><span>=</span><span> </span><span>results_df_roc</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>(</span><span>aes</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>model</span><span>),</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_color_manual</span><span>(</span><span>values</span><span> </span><span>=</span><span> </span><span>custom_col</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_abline</span><span>(</span><span>intercept</span><span> </span><span>=</span><span> </span><span>0</span><span>,</span><span> </span><span>slope</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;gray&quot;</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme_bw</span><span>(</span><span>base_size</span><span> </span><span>=</span><span> </span><span>18</span><span>)</span></code></pre>
</figure>
<p><img alt="center" src="https://i1.wp.com/dpmartin42.github.io/figs/2016-12-10-imbalanced-classes-part-1/unnamed-chunk-5-1.png?w=456" data-recalc-dims="1"></p>
<h2>Final thoughts</h2>
<p>In the above post, I outline some steps to help improve classification performance when you have imbalanced classes. Although weighting outperformed the sampling techniques in this simulation, this may not always be the case. Because of this, it is important to compare different techniques to see which works best for your data. I have actually found that in many cases, there is no huge benefit in using either weighting or sampling techniques when classes are moderately imbalanced (i.e., no worse than 10:1) in conjunction with a threshold-invariant metric like the AUC. In the next post, I will go over some caveats to keep in mind when using the AUC in the case of imbalanced classes and how other metrics can be more informative. Stay tuned!</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://dpmartin42.github.io/blogposts/r/imbalanced-classes-part-1"> Wicked Good Data - r</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/nKqhvBtfKrY"></a></div>