<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/data-discretization-made-easy-with-funmodeling/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://blog.datascienceheroes.com/data-discretization-made-easy-with-funmodeling/"> R - Data Science Heroes Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img alt="Data discretization made easy with funModeling" src="https://i1.wp.com/blog.datascienceheroes.com/content/images/2017/12/data_discretization_2.png?w=456" data-recalc-dims="1"></p>
<p><strong>tl;dr</strong>: Convert numerical variables into categorical, as it is shown in the next image.</p>
<p><img alt="Data discretization made easy with funModeling" src="https://i1.wp.com/datascienceheroes.com/img/blog/data_discretization_1.png?w=80%25&ssl=1" data-recalc-dims="1"></p>
<p>⏳ <em>Reading time ~ 6 min.</em> </p>
<h2>Let’s start!</h2>
<p>The package <code>funModeling</code> (from version > 1.6.6) introduces two <br>
functions— <code>discretize_get_bins</code> &amp; <code>discretize_df</code> —that work together <br>
in order to help us in the discretization task.</p>
<pre><code># First we load the libraries
# install.packages(&quot;funModeling&quot;)
library(funModeling)
library(dplyr)
</code></pre>
<p>Let’s see an example. First, we check current data types:</p>
<pre><code>df_status(heart_disease, print_results = F) %>% select(variable, type, unique, q_na) %>% arrange(type)

##                  variable    type unique q_na
## 1                  gender  factor      2    0
## 2              chest_pain  factor      4    0
## 3     fasting_blood_sugar  factor      2    0
## 4         resting_electro  factor      3    0
## 5                    thal  factor      3    2
## 6            exter_angina  factor      2    0
## 7       has_heart_disease  factor      2    0
## 8                     age integer     41    0
## 9  resting_blood_pressure integer     50    0
## 10      serum_cholestoral integer    152    0
## 11         max_heart_rate integer     91    0
## 12            exer_angina integer      2    0
## 13                  slope integer      3    0
## 14      num_vessels_flour integer      4    4
## 15 heart_disease_severity integer      5    0
## 16                oldpeak numeric     40    0
</code></pre>
<p>We’ve got factor, integer, and numeric variables: a good mix! The <br>
transformation has two steps. First, it gets the cuts or threshold <br>
values from which each segment begins. The second step is using the <br>
threshold to obtain the variables as categoricals.</p>
<p>Two variables will be discretized in the following example: <br>
<code>max_heart_rate</code> and <code>oldpeak</code>. Also, we’ll introduce some <code>NA</code> values<br>
into <code>oldpeak</code> to test how the function works with missing data.</p>
<pre><code># Introducing some missing values in the first 30 rows of the oldpeak variable
heart_disease$oldpeak[1:30]=NA
</code></pre>
<p>Step 1) Getting the bin thresholds for each input variable:</p>
<p><code>discretize_get_bins</code> returns a data frame that needs to be used in the<br>
<code>discretize_df</code> function, which returns the final processed data frame.</p>
<pre><code>d_bins=discretize_get_bins(data=heart_disease, input=c(&quot;max_heart_rate&quot;, &quot;oldpeak&quot;), n_bins=5)

## [1] &quot;Variables processed: max_heart_rate, oldpeak&quot;

# Checking `d_bins` object:
d_bins

##         variable                     cuts
## 1 max_heart_rate -Inf|131|147|160|171|Inf
## 2        oldpeak   -Inf|0.1|0.3|1.1|2|Inf
</code></pre>
<p>Parameters:</p>
<ul>
<li><code>data</code>: the data frame containing the variables to be processed.</li>
<li><code>input</code>: vector of strings containing the variable names.</li>
<li><code>n_bins</code>: the number of bins/segments to have in the discretized<br>
data.</li>
</ul>
<p>We can see each threshold point (or upper boundary) for each variable. <br>
<em>Please note, <code>-Inf</code> is not an actual upper boundary: more info in the<br>
upcoming example.</em></p>
<p>Step 2) Applying the thresholds for each variable:</p>
<pre><code># Now it can be applied on the same data frame or in a new one (for example, in a predictive model that changes data over time)
heart_disease_discretized=discretize_df(data=heart_disease, data_bins=d_bins, stringsAsFactors=T)

## [1] &quot;Variables processed: max_heart_rate, oldpeak&quot;
</code></pre>
<p>Parameters:</p>
<ul>
<li><code>data</code>: data frame containing the numerical variables to be<br>
discretized.</li>
<li><code>data_bins</code>: data frame returned by <code>discretize_get_bins</code>. If it is<br>
changed by the user, then each upper boundary must be separated by a<br>
pipe character (<code>|</code>) as shown in the example.</li>
<li><code>stringsAsFactors</code>: <code>TRUE</code> by default, final variables will be<br>
factor (instead of a character) and useful when plotting.</li>
</ul>
<h4>Final results and their plots</h4>
<p>Before and after</p>
<p><img alt="Data discretization made easy with funModeling" src="https://i1.wp.com/datascienceheroes.com/img/blog/data_discretization_1.png?w=80%25&ssl=1" data-recalc-dims="1"></p>
<p>Final distribution: </p>
<pre><code>describe(heart_disease_discretized %>% select(max_heart_rate,oldpeak))

## heart_disease_discretized %>% select(max_heart_rate, oldpeak) 
## 
##  2  Variables      303  Observations
## ---------------------------------------------------------------------------
## max_heart_rate 
##        n  missing distinct 
##      303        0        5 
##                                                                       
## Value      [-Inf, 131) [ 131, 147) [ 147, 160) [ 160, 171) [ 171, Inf]
## Frequency           63          59          62          62          57
## Proportion       0.208       0.195       0.205       0.205       0.188
## ---------------------------------------------------------------------------
## oldpeak 
##        n  missing distinct 
##      303        0        6 
##                                                                       
## Value      [-Inf, 0.1) [ 0.1, 0.3) [ 0.3, 1.1) [ 1.1, 2.0) [ 2.0, Inf]
## Frequency           97          18          54          54          50
## Proportion       0.320       0.059       0.178       0.178       0.165
##                       
## Value              NA.
## Frequency           30
## Proportion       0.099
## ---------------------------------------------------------------------------

p5=ggplot(heart_disease_discretized, aes(max_heart_rate)) + geom_bar(fill=&quot;#0072B2&quot;) + theme_bw() + theme(axis.text.x = element_text(angle = 45, vjust = 1, hjust=1))
p6=ggplot(heart_disease_discretized, aes(oldpeak)) + geom_bar(fill=&quot;#CC79A7&quot;) + theme_bw() + theme(axis.text.x = element_text(angle = 45, vjust = 1, hjust=1))

gridExtra::grid.arrange(p5, p6, ncol=2)
</code></pre>
<p>Showing final variable distribution:</p>
<p><img alt="Data discretization made easy with funModeling" src="https://i1.wp.com/datascienceheroes.com/img/blog/data_discretization_2.png?w=80%25&ssl=1" data-recalc-dims="1"></p>
<p>Sometimes, it is not possible to get the same number of cases per bucket <br>
when computing <strong>equal frequency</strong> as is shown in the <code>oldpeak</code> <br>
variable.</p>
<h4>NA handling</h4>
<p>Regarding the <code>NA</code> values, the new <code>oldpeak</code> variable has six <br>
categories: five categories defined in <code>n_bins=5</code> plus the <code>NA.</code> value. <br>
Note the point at the end indicating the presence of missing values.</p>
<h4>More info</h4>
<ul>
<li><code>discretize_df</code> will never return an <code>NA</code> value without transforming<br>
it to the string <code>NA.</code>.</li>
<li><code>n_bins</code> sets the number of bins for all the variables.</li>
<li>If <code>input</code> is missing, then it will run for all numeric/integer<br>
variables whose number of unique values is greater than the number<br>
of bins (<code>n_bins</code>).</li>
<li>Only the variables defined in <code>input</code> will be processed while<br>
remaining variables will <strong>not be modified at all</strong>.</li>
<li><code>discretize_get_bins</code> returns just a data frame that can be changed<br>
by hand as needed, either in a text file or in the R session.</li>
</ul>
<h4>Discretization with new data</h4>
<p>In our data, the minimum value for <code>max_heart_rate</code> is 71. The data <br>
preparation must be robust with new data; e.g., if a new patient arrives <br>
whose <code>max_heart_rate</code> is 68, then the current process will assign <br>
her/him to the lowest category.</p>
<p>In other functions from other packages, this preparation may return an <br>
<code>NA</code> because it is out of the segment.</p>
<p>As we pointed out before, if new data comes over time, it’s likely to <br>
get new min/max value/s. This can break our process. To solve this, <br>
<code>discretize_df</code> will always have as min/max the values <code>-Inf</code>/<code>Inf</code>;<br>
thus, any new value falling below/above the minimum/maximum will be <br>
added to the lowest or highest segment as applicable.</p>
<p>The data frame returned by <code>discretize_get_bins</code> must be saved in order <br>
to apply it to new data. If the discretization is not intended to run <br>
with new data, then there is no sense in having two functions: it can be <br>
only one. In addition, there would be no need to save the results of <br>
<code>discretize_get_bins</code>.</p>
<p>Having this two-step approach, we can handle both cases.</p>
<h4>Conclusions about two-step discretization</h4>
<p>The usage of <code>discretize_get_bins</code> + <code>discretize_df</code> provides quick data <br>
preparation, with a clean data frame that is ready to use. Clearly <br>
showing where each segment begin and end, indispensable when making <br>
statistical reports.</p>
<p>The decision of <em>not fail</em> when dealing with a new min/max in new data <br>
is <strong>just a decision</strong>. In some contexts, failure would be the desired <br>
behavior.</p>
<p><strong>The human intervention</strong>: The easiest way to discretize a data frame<br>
is to select the same number of bins to apply to every variable—just <br>
like the example we saw—however, if tuning is needed, then some <br>
variables may need a <strong>different number of bins</strong>. For example, a <br>
variable with less dispersion can work well with a low number of bins.</p>
<p>Common values for the number of segments could be 3, 5, 10, or 20 (but <br>
no more). It is up to the data scientist to make this decision.</p>
<p></p>
<h4>Bonus track: The trade-off art ⚖️</h4>
<ul>
<li>A high number of bins => More noise captured.</li>
<li>A low number of bins => Oversimplification, less variance.</li>
</ul>
<p>Do these terms sound similar to any other ones in machine learning?</p>
<p>The answer: <strong>Yes!</strong>. Just to mention one example: the trade-off between <br>
adding or subtracting variables from a predictive model.</p>
<ul>
<li>More variables: Overfitting alert (too detailed predictive model).</li>
<li>Fewer variables: Underfitting danger (not enough information to<br>
capture general patterns).</li>
</ul>
<p><em>Just like oriental philosophy has pointed out for thousands of years, there is an art in finding the right balance between one value and its opposite.</em></p>
<hr>
<p><img alt="📌" src="https://s.w.org/images/core/emoji/2.3/72x72/1f4cc.png" class="wp-smiley"> This article was adapted from the <strong>Data Science Live Book</strong> – <em>Handling Data Types</em> chapter: <a rel="nofollow" href="https://livebook.datascienceheroes.com/data-preparation.html#data_types" target="_blank">https://livebook.datascienceheroes.com/data-preparation.html#data_types</a>. Please go there for a deeper coverage.</p>
<p><img alt="Data discretization made easy with funModeling" src="https://i2.wp.com/datascienceheroes.com/img/blog/data-science-live-book-cover-2.png?w=30%25&ssl=1" data-recalc-dims="1"></p>
<p><a rel="nofollow" href="https://livebook.datascienceheroes.com/download-book.html" target="_blank">Book’s download page</a> <img alt="📥" src="https://s.w.org/images/core/emoji/2.3/72x72/1f4e5.png" class="wp-smiley"><img alt="📘" src="https://s.w.org/images/core/emoji/2.3/72x72/1f4d8.png" class="wp-smiley"></p>
<hr>
<p><code>Keep in touch:</code> <a rel="nofollow" href="https://twitter.com/pabloc_ds" target="_blank">@pabloc_ds</a>.</p>
<p>~ Thanks for reading <img alt="🚀" src="https://s.w.org/images/core/emoji/2.3/72x72/1f680.png" class="wp-smiley">. </p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://blog.datascienceheroes.com/data-discretization-made-easy-with-funmodeling/"> R - Data Science Heroes Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/dsKrMY5ylT0" height="1"></a></div>