<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/vtreat-variable-importance/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2018/12/vtreat-variable-importance/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a rel="nofollow" href="https://github.com/WinVector/vtreat" target="_blank"><code>vtreat</code></a>‘s purpose is to produce pure numeric <a rel="nofollow" href="https://www.r-project.org/" target="_blank"><code>R</code></a> <code>data.frame</code>s that are ready for <a rel="nofollow" href="https://en.wikipedia.org/wiki/Supervised_learning" target="_blank">supervised predictive modeling</a> (predicting a value from other values). By ready we mean: a purely numeric data frame with no missing values and a reasonable number of columns (missing-values re-encoded with indicators, and high-degree categorical re-encode by effects codes or impact codes).</p>
<p>In this note we will discuss a small aspect of the <code>vtreat</code> package: variable screening.</p>
<p><span></span></p>
<p>Part of the <code>vtreat</code> philosophy is to assume after the <code>vtreat</code> variable processing the next step is a sophisticated <a rel="nofollow" href="https://en.wikipedia.org/wiki/Supervised_learning" target="_blank">supervised machine learning</a> method. Under this assumption we assume the machine learning methodology (be it regression, tree methods, random forests, boosting, or neural nets) will handle issues of redundant variables, joint distributions of variables, overall regularization, and joint dimension reduction.</p>
<p>However, an important exception is: variable screening. In practice we have seen wide data-warehouses with hundreds of columns overwhelm and defeat state of the art machine learning algorithms due to over-fitting. We have some synthetic examples of this (<a rel="nofollow" href="http://www.win-vector.com/blog/2014/02/bad-bayes-an-example-of-why-you-need-hold-out-testing/" target="_blank">here</a> and <a rel="nofollow" href="http://www.win-vector.com/blog/tag/talks/" target="_blank">here</a>).</p>
<p>The upshot is: even in 2018 you can not treat every column you find in a data warehouse as a variable. You must at least perform some basic screening.</p>
<p>To help with this <code>vtreat</code> incorporates a per-variable linear significance report. This report shows how useful each variable is taken alone in a linear or generalized linear model (some details can be found <a rel="nofollow" href="https://arxiv.org/abs/1611.09477" target="_blank">here</a>). However, this sort of calculation was optimized for speed, not discovery power.</p>
<p><code>vtreat</code> now includes a direct variable valuation system that works very well with complex numeric relationships. It is a function called <a rel="nofollow" href="https://winvector.github.io/vtreat/reference/value_variables_N.html" target="_blank"><code>vtreat::value_variables_N()</code></a> for numeric or regression problems and <a rel="nofollow" href="https://winvector.github.io/vtreat/reference/value_variables_C.html" target="_blank"><code>vtreat::value_variables_C()</code></a> for binomial classification problems. It works by fitting two transformed copies of each numeric variable to the outcome. One transform is a low frequency transform realized as an optimal <code>k</code>-segment linear model for a moderate choice of <code>k</code>. The other fit is a high-frequency trasnform realized as a <code>k</code>-nearest neighbor average for moderate choice of <code>k</code>. Some of the methodology is shown <a rel="nofollow" href="https://github.com/WinVector/vtreat/blob/master/extras/SegFitter.md" target="_blank">here</a>.</p>
<p>We recommend using <code>vtreat::value_variables_*()</code> as an initial variable screen.</p>
<p>Let’s demonstrate this using the data from the segment fitter example. In our case the value to be predicted (&quot;<code>y</code>&quot;) is a noisy copy of <code>sin(x)</code>. Let’s set up our example data:</p>
<div>
<pre><code><span>set.seed</span>(<span>1999</span>)
d &lt;-<span> </span><span>data.frame</span>(<span>x =</span> <span>seq</span>(<span>0</span>, <span>15</span>, <span>by =</span> <span>0.25</span>))
d$y_ideal &lt;-<span> </span><span>sin</span>(d$x)
d$x_noise &lt;-<span> </span>d$x[<span>sample.int</span>(<span>nrow</span>(d), <span>nrow</span>(d), <span>replace =</span> <span>FALSE</span>)]
d$y &lt;-<span> </span>d$y_ideal +<span> </span><span>0.5</span>*<span>rnorm</span>(<span>nrow</span>(d))
<span>dim</span>(d)</code></pre>
</div>
<pre><code>## [1] 61  4</code></pre>
<p>Now a simple linear valuation of the the variables can be produced as follows.</p>
<div>
<pre><code>cfe &lt;-<span> </span>vtreat::<span>mkCrossFrameNExperiment</span>(
  d, 
  <span>varlist =</span> <span>c</span>(<span>&quot;x&quot;</span>, <span>&quot;x_noise&quot;</span>), 
  <span>outcomename =</span> <span>&quot;y&quot;</span>)</code></pre>
</div>
<pre><code>## [1] &quot;vtreat 1.3.4 start initial treatment design Mon Dec 17 20:02:58 2018&quot;
## [1] &quot; start cross frame work Mon Dec 17 20:02:59 2018&quot;
## [1] &quot; vtreat::mkCrossFrameNExperiment done Mon Dec 17 20:02:59 2018&quot;</code></pre>
<div>
<pre><code>sf &lt;-<span> </span>cfe$treatments$scoreFrame
knitr::<span>kable</span>(sf[, <span>c</span>(<span>&quot;varName&quot;</span>, <span>&quot;rsq&quot;</span>, <span>&quot;sig&quot;</span>)])</code></pre>
</div>
<table>
<thead>
<tr>
<th align="left">varName</th>
<th align="right">rsq</th>
<th align="right">sig</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">x</td>
<td align="right">0.0050940</td>
<td align="right">0.5846527</td>
</tr>
<tr>
<td align="left">x_noise</td>
<td align="right">0.0155874</td>
<td align="right">0.3377146</td>
</tr>
</tbody>
</table>
<p>Notice the signal carrying variable did not score better (having a larger <code>r</code>-squared and a smaller (better) significance value) than the noise variable (that is unrelated to the outcome). This is because the relation between <code>x</code> and <code>y</code> is not linear.</p>
<p>Now let’s try <code>vtreat::value_variables_N()</code>.</p>
<div>
<pre><code>vf =<span> </span>vtreat::<span>value_variables_N</span>(
  d, 
  <span>varlist =</span> <span>c</span>(<span>&quot;x&quot;</span>, <span>&quot;x_noise&quot;</span>),
  <span>outcomename =</span> <span>&quot;y&quot;</span>)
<span>rownames</span>(vf) &lt;-<span> </span><span>NULL</span>

knitr::<span>kable</span>(vf[, <span>c</span>(<span>&quot;var&quot;</span>, <span>&quot;rsq&quot;</span>, <span>&quot;sig&quot;</span>)])</code></pre>
</div>
<table>
<thead>
<tr>
<th align="left">var</th>
<th align="right">rsq</th>
<th align="right">sig</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">x</td>
<td align="right">0.26719313</td>
<td align="right">0.0000600</td>
</tr>
<tr>
<td align="left">x_noise</td>
<td align="right">0.01591601</td>
<td align="right">0.9978984</td>
</tr>
</tbody>
</table>
<p>Now the difference is night and day. The important variable <code>x</code> is singled out (scores very well), and the unimportant variable <code>x_noise</code> doesn’t often score well. Though, as with all significance tests, useless variables can get lucky from time to time- (an issue that can be addressed by using a <a rel="nofollow" href="http://www.win-vector.com/blog/2017/09/remember-p-values-are-not-effect-sizes/" target="_blank">Cohen’s-<code>d</code> style calculation</a>).</p>
<p>Our modeling advice is:</p>
<ul>
<li>Use <code>vtreat::value_variables_*()</code></li>
<li>Pick all variables with <code>sig &lt;= 1/number_of_variables_being_considered</code>.</li>
</ul>
<p>The idea is: each &quot;pure noise&quot; (or purely useless) variable has a significance that is distributed uniformly between zero and one. So the expected number of useless variables that make it through the above screening is <code>number_of_useless_varaibles * P[useless_sig &lt;= 1/number_of_variables_being_considered]</code>. This equals <code>number_of_useless_varaibles * 1/number_of_variables_being_considered</code>. As <code>number_of_useless_varaibles &lt;= number_of_variables_being_considered</code> we get this quantity is no more than one. So we expect a constant number of useless variables to sneak through this filter. The hope is: this should not be enough useless variables to overwhelm the next stage supervised machine learning step.</p>
<p>Obviously there are situations where variable importance can not be discovered without considering joint distributions. The most famous one being &quot;xor&quot; where the concept to be learned is if an odd or even number of indicator variables are zero or one (each such variable is individual completely uninformative about the outcome until you have all of the variables simultaneously). However, for practical problems you often have that most variables have a higher marginal predictive power taken alone than they have in the final joint model (as other, better, variables consume some of common variables’ predictive power in the joint model). With this in mind single variable screening often at least gives an indication where to look.</p>
<p>In conclusion the <code>vtreat</code> package and <code>vtreat::value_variables_*()</code> can be a valuable addition to your supervised learning practice.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2018/12/vtreat-variable-importance/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/IOJAQsSDYxw" height="1"></a></div>