<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/sample-size-and-class-balance-on-model-performance/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://blog.datascienceheroes.com/sample-size-and-class-balance-on-model-performance/"> R - Data Science Heroes Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div><img alt="Sample size and class balance on model performance" src="https://i1.wp.com/blog.datascienceheroes.com/content/images/2018/09/Screen-Shot-2018-09-05-at-13.38.33.png?w=456&ssl=1" data-recalc-dims="1">
<p>tl;dr: This post shows the relationship between the sample size and the accuracy in a classification model.</p>
<p>I hope this little research I did may help you in your classification problems. An LSTM model created in Keras was used to produce the results.</p>
<p>The metric we are tracking is <code>categorical_accuracy</code> (equivalent to <code>accuracy</code> for multi-class), which is <strong>biased towards the values that have more representativeness</strong>.</p>
<p>For example, if we are predicting fraud, which occurs only 1 time in 1000 (0.1%); then assigning all 100% of the cases as &quot;Non-fraud&quot; will lead us to be correct 99.9% of the times. High, but utterly useless.</p>
<h2>Introducing our case</h2>
<p>I decided to study the output of this model, as a sanity check on the performance.</p>
<p>Each row represents a class to predict, (42 unique in total).</p>
<p><img alt="Sample size and class balance on model performance" src="https://i2.wp.com/blog.datascienceheroes.com/content/images/2018/09/final_count_42.gif?w=400&ssl=1" data-recalc-dims="1"></p>
<p>This is an important point, since the total different classes to predict are around 900, and the <strong>model only predicts 42</strong>. This should be a warning of the input data and/or architecture we are using, (either in Deep Learning or the &quot;classic&quot; Machine Learning).</p>
<p>Columns:</p>
<ul>
<li><code>TP</code>: True positive (the model correctly predicted the class)</li>
<li><code>rank</code>: Indicates the rank of frequency apparition. Just like in a running competence, the fastest runner ends at the 1st place; this rank measures the order according to the class representativeness.</li>
</ul>
<p>Loading the libraries:</p>
<pre><code># install.packages(&quot;corrr&quot;)
# install.packages(&quot;funModeling&quot;)
# install.packages(&quot;tidyverse&quot;)

library(corrr) 
library(funModeling)
library(tidyverse)
</code></pre>
<p>The data we are going to use looks like:</p>
<pre><code>d_tp_rank=readr::read_delim(&quot;https://s3.amazonaws.com/datascienceheroes.com/img/tp_rank.csv&quot;, delim = &quot;;&quot;)
</code></pre>
<pre><code>## Parsed with column specification:
## cols(
##   TP = col_double(),
##   rank = col_integer()
## )
</code></pre>
<pre><code>d_tp_rank
</code></pre>
<pre><code>## # A tibble: 42 x 2
##       TP  rank
##    </code></pre></div>