<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/woe-and-iv-variable-screening-with-information-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.exactness.net/post/164667039305"> The Exactness of Mind</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>A short note on information-theoretic variable screening in R w. {<a rel="nofollow" href="https://cran.r-project.org/web/packages/Information/index.html" target="_blank">information</a>}.</p>
<figure data-orig-width="450" data-orig-height="600"><img data-orig-width="450" src="https://i1.wp.com/68.media.tumblr.com/87b30596966260fe39644f84da127559/tumblr_inline_ovc7qnZCPG1qa0hyw_540.jpg?resize=450%2C600" data-recalc-dims="1" data-orig-height="600"></figure>
<p>Variable screening comes as an important step in the<br>
contemporary EDA for predictive modeling: what can we tell about the<br>
nature of the relationships between a set of predictors and the<br>
dependent before entering the modeling phase? Can we infer something about the predictive power of the independent variables before we<br>
start rolling them into a predictive model? In this blog post I will<br>
discuss two information-theoretic measures that are common in variable<br>
screening for binary classification and regression models in the credit<br>
risk arena (the fact being completely unrelated to the simple truth that<br>
they could do you some good in any other application of predictive<br>
modeling as well). I will first introduce the Weight of Evidence (WoE) and Information Value<br>
(IV) of a variable in respect to a binary outcome. Then I will<br>
illustrate their computation (it’s fairly easy, in fact) from the<br>
{Information} package in R.</p>
<p><b>Weight of Evidence</b></p>
<p>Take the common Bayesian hypothesis test (or a <a rel="nofollow" href="http://wwwf.imperial.ac.uk/~das01/MyWeb/SCBI/Papers/KassRaftery.pdf" target="_blank">Bayes factor</a>, if you prefer):</p>
<figure data-orig-height="45" data-orig-width="107"><img data-orig-width="107" src="https://i1.wp.com/68.media.tumblr.com/3fda8e4225a364e0a3c41c8a18d591a2/tumblr_inline_ovc7seHyiy1qa0hyw_540.png?resize=107%2C45" data-recalc-dims="1" data-orig-height="45"></figure>
<p>and assume your models M1, M2 of the world*<br>
are simply two discrete possible states of a binary variable Y, while<br>
the data are given by discrete distributions of some predictor X (or, X<br>
stands for a binned continuous distribution); for every category j in X, j = 1, 2,.. n,  take the log:</p>
<figure data-orig-height="46" data-orig-width="131"><img data-orig-width="131" src="https://i1.wp.com/68.media.tumblr.com/0dad429aac952bd6e508853965d65073/tumblr_inline_ovc7t1ZjTC1qa0hyw_540.png?resize=131%2C46" data-recalc-dims="1" data-orig-height="46"></figure>
<p>and you will get to simple a measure of evidence in favor of M1 against M2 that <a rel="nofollow" href="http://www.cs.tufts.edu/~nr/cs257/archive/jack-good/weight-of-evidence.pdf" target="_blank">Good has described as Weight of Evidence</a><br>
(WoE). In theory, any monotonic transformation of the odds would do,<br>
but the logarithm brings an intuitive advantage of obtaining a negative<br>
WoE when the odds are less than one and a positive one when they are<br>
higher than one. To simplify the setting where the analysis under<br>
consideration encompasses a binary dependent Y and a discrete (or binned<br>
continuous) predictor X, we are simply inspecting the conditional distribution of X given Y:</p>
<figure data-orig-height="179" data-orig-width="139"><img data-orig-width="139" src="https://i1.wp.com/68.media.tumblr.com/e20c8829a4db7bca09aa790afc6f7145/tumblr_inline_ovc7uif5kC1qa0hyw_540.png?resize=139%2C179" data-recalc-dims="1" data-orig-height="179"></figure>
<p>where f denotes counts.</p>
<p>Let’s illustrate the computation of WoE in this setting for a variable from a well-known dataset**. We have one categorical, binary dependent:</p>
<blockquote><p>dataSet &lt;-read.table(‘bank-additional-full.csv’,<br>                     header = T,<br>                     strip.white = F,<br>                     sep = “;”)</p>
<p>str(dataSet)<br>table(dataSet$y)</p>
<p>dataSet$y &lt;-recode(dataSet$y,<br>                   ‘yes’ = 1,<br>                    ‘no’ = 0)</p>
</blockquote>
<p>and we want to compute the WoE for, say, the age variable. Here it goes:</p>
<blockquote><p># – compute WOE for: dataSet$age<br>bins &lt;-10</p>
<p>q &lt;-quantile(dataSet$age,<br>             probs = c(1:(bins – 1)/bins),<br>             na.rm = TRUE,<br>             type = 3)</p>
<p>cuts &lt;-unique(q)</p>
<p>aggAge &lt;-table(findInterval(dataSet$age,<br>                            vec = cuts,<br>                            rightmost.closed = FALSE),<br>                            dataSet$y)</p>
<p>aggAge &lt;-as.data.frame.matrix(aggAge)<br>aggAge$N &lt;-rowSums(aggAge)<br>aggAge$WOE &lt;-log((aggAge$`1`*sum(aggAge$`0`))/(aggAge$`0`*sum(aggAge$`1`)))</p>
</blockquote>
<p>In the previous example I have used exactly the approach to bin X (age, in this case) that is used in the R package {Information} whose application I want to illustrate later. The table()<br>
call provides for the conditional distributions like the ones shown in<br>
the table above. The computation of WoE is then straightforward – as<br>
exemplified in the last line. However, you want to spare yourself from<br>
computing the WoE in this way for many variables in the dataset, and<br>
that’s where {<a rel="nofollow" href="https://cran.r-project.org/web/packages/Information/index.html" target="_blank">Information</a>} in R comes handy; for the same dataset:</p>
<blockquote><p># – Information value: all variables</p>
<p>infoTables &lt;-create_infotables(data = dataSet,</p>
<p>                                y = “y”,<br>                               bins = 10,<br>                               parallel = T)</p>
<p># – WOE table:<br>infoTables$Tables$age$WOE</p>
</blockquote>
<p>with the respective data frames in infoTables$Tables standing for the variables in the dataset. </p>
<p><b>Information Value</b></p>
<p>A straightforward definition of the Information Value (IV)of a variable is provided in the {Information} package <a rel="nofollow" href="https://cran.r-project.org/web/packages/Information/vignettes/Information-vignette.html" target="_blank">vignette</a>:</p>
<figure data-orig-width="433" data-orig-height="53"><img data-orig-width="433" src="https://i2.wp.com/68.media.tumblr.com/35c57ac5bf0399e8d0db42fa28c33fa1/tumblr_inline_ovc80hhWcp1qa0hyw_540.png?resize=433%2C53" data-recalc-dims="1" data-orig-height="53"></figure>
<p>In effect, this means that we are summing across the individual WoE values (i.e. for each bin j of X) and weighting them by the respective differences between P(xj|Y=1) and P(xj|Y=0). The IV of a variable measures its predictive power, and variables with IV &lt; .05 are generally considered to have a low predictive power.</p>
<p>Using {Information} in R, for the dataset under consideration:</p>
<p># – Information value: all variables</p>
<blockquote><p>infoTables &lt;-create_infotables(data = dataSet,</p>
<p>                                y = “y”,<br>                               bins = 10,<br>                               parallel = T)</p>
<p># – Plot IV</p>
<p>plotFrame &lt;-infoTables$Summary[order(-infoTables$Summary$IV), ]<br>plotFrame$Variable &lt;-factor(plotFrame$Variable,</p>
<p>                             levels = plotFrame$Variable[order(-plotFrame$IV)])</p>
<p>ggplot(plotFrame, aes(x = Variable, y = IV)) +<br>geom_bar(width = .35, stat = “identity”, color = “darkblue”, fill = “white”) +<br>ggtitle(“Information Value”) +<br>theme_bw() +<br>theme(plot.title = element_text(size = 10)) +<br>theme(axis.text.x = element_text(angle = 90))</p>
</blockquote>
<figure data-orig-width="450" data-orig-height="403"><img data-orig-width="450" src="https://i2.wp.com/68.media.tumblr.com/7fc52f0445f02172488a29cc1b6a670d/tumblr_inline_ovc81lFgpg1qa0hyw_540.png?resize=450%2C403" data-recalc-dims="1" data-orig-height="403"></figure>
<p>You may have noted the usage of parallel = T in the create_infotables()<br>
call; the {Information} package will try to use all available cores to<br>
speed up the computations by default. Besides the basic package<br>
functionality that I have illustrated, the package provides a natural<br>
way of dealing with uplift models, where the computation of the IVs for<br>
the control vs. treatment designs is nicely automated. Cross-validation<br>
procedures are also built-in.</p>
<p>However, now that we know that we have a nice,<br>
working package for WoE and IV estimation in R, let’s restrain ourselves<br>
from using it to perform automatic feature<br>
selection for models like binary logistic regression. While the<br>
information-theoretic measures discussed here truly assess the<br>
predictive power of a predictor in binary classification, building a<br>
model that encompasses multiple terms model is another story. Do not get<br>
disappointed if you start figuring out how the AICs for the full models<br>
are still lower then those for the nested models obtained by feature<br>
selection based on the IV values; although they can provide useful<br>
guidelines, WoE and IV are not even meant to be used that way (I’ve<br>
tried… once with the dataset used in the previous examples, and then<br>
with the two {Information} built-in datasets; not too much of a success,<br>
as you may have guessed).</p>
<p><b>References</b></p>
<p>* For parametric models, you would need to integrate over the full<br>
parameter space, of course; taking the MLEs would result in obtaining<br>
the standard LR test.</p>
<p>** The dataset is considered in S. Moro, P. Cortez and P. Rita<br>
(2014). A Data-Driven Approach to Predict the Success of Bank<br>
Telemarketing. Decision Support Systems, Elsevier, 62:22-31, June 2014. I<br>
have obtained it from: <a rel="nofollow" href="https://archive.ics.uci.edu/ml/datasets/Bank+Marketing" target="_blank">https://archive.ics.uci.edu/ml/datasets/Bank+Marketing</a><br>
(N.B. <a rel="nofollow" href="https://archive.ics.uci.edu/ml/machine-learning-databases/00222/," target="_blank">https://archive.ics.uci.edu/ml/machine-learning-databases/00222/,</a><br>
file: bank-additional.zip); a nice description of the dataset is found<br>
at:<br>
<a rel="nofollow" href="http://www2.1010data.com/documentationcenter/beta/Tutorials/MachineLearningExamples/BankMarketingDataSet.html" target="_blank">http://www2.1010data.com/documentationcenter/beta/Tutorials/MachineLearningExamples/BankMarketingDataSet.html</a>)</p>
<p></p>
<p>Goran S. Milovanović, Phd</p>
<p>Data Science Consultant, SmartCat</p>
<figure data-orig-height="180" data-orig-width="180"><img data-orig-width="180" src="https://i0.wp.com/68.media.tumblr.com/a5d2491db08ab66670efe27e60c1ee9c/tumblr_inline_ovc8awRMTx1qa0hyw_540.png?resize=180%2C180" data-recalc-dims="1" data-orig-height="180"></figure>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.exactness.net/post/164667039305"> The Exactness of Mind</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/O7Zbe9w5fj0" height="1"></a></div>