A short note on information-theoretic variable screening in R w. {information}.
Variable screening comes as an important step in the contemporary EDA for predictive modeling: what can we tell about the nature of the relationships between a set of predictors and the dependent before entering the modeling phase? Can we infer something about the predictive power of the independent variables before we
start rolling them into a predictive model? In this blog post I will discuss two information-theoretic measures that are common in variable screening for binary classification and regression models in the credit risk arena (the fact being completely unrelated to the simple truth that
they could do you some good in any other application of predictive modeling as well). I will first introduce the Weight of Evidence (WoE) and Information Value
(IV) of a variable in respect to a binary outcome. Then I will illustrate their computation (it’s fairly easy, in fact) from the {Information} package in R.
Weight of Evidence
Take the common Bayesian hypothesis test (or a Bayes factor, if you prefer):
and assume your models M1, M2 of the world*
are simply two discrete possible states of a binary variable Y, while the data are given by discrete distributions of some predictor X (or, X stands for a binned continuous distribution); for every category j in X, j = 1, 2,.. n,  take the log:
and you will get to simple a measure of evidence in favor of M1 against M2 that Good has described as Weight of Evidence
(WoE). In theory, any monotonic transformation of the odds would do, but the logarithm brings an intuitive advantage of obtaining a negative WoE when the odds are less than one and a positive one when they are higher than one. To simplify the setting where the analysis under consideration encompasses a binary dependent Y and a discrete (or binned
continuous) predictor X, we are simply inspecting the conditional distribution of X given Y:
where f denotes counts.
Let’s illustrate the computation of WoE in this setting for a variable from a well-known dataset**. We have one categorical, binary dependent:
dataSet