<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/a-demonstration-of-vtreat-data-preparation/"></div></div><div><a data-url="http://www.r-bloggers.com/a-demonstration-of-vtreat-data-preparation/" data-counturl="http://www.r-bloggers.com/a-demonstration-of-vtreat-data-preparation/" data-count="horizontal" data-text="A demonstration of vtreat data preparation" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2016/06/a-demonstration-of-vtreat-data-preparation/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>This article is a demonstration the use of the <a target="_blank" href="https://cran.r-project.org/">R</a> <a target="_blank" href="https://github.com/WinVector/vtreat">vtreat</a> variable preparation package followed by <a target="_blank" href="http://topepo.github.io/caret/index.html">caret</a> controlled training.</p>
<p>In previous writings we have gone to great lengths to <a target="_blank" href="http://winvector.github.io/vtreathtml/">document, explain and motivate <code>vtreat</code></a>. That necessarily gets long and unnecessarily feels complicated.</p>
<p>In this example we are going to show what building a predictive model using <code>vtreat</code> best practices looks like assuming you were somehow already in the habit of using vtreat for your data preparation step. We are deliberately not going to explain any steps, but just show the small number of steps we advise routinely using. This is a simple schematic, but not a guide. Of course we do not advise use without understanding (and we work hard to teach the concepts in our writing), but want what small effort is required to add <code>vtreat</code> to your predictive modeling practice.</p>
<p><span></span></p>
<p>First we set things up: load libraries, initialize parallel processing.</p>
<div>
<pre><code><span>library</span>(<span>'vtreat'</span>)
<span>library</span>(<span>'caret'</span>)
<span>library</span>(<span>'gbm'</span>)
<span>library</span>(<span>'doMC'</span>)
<span>library</span>(<span>'WVPlots'</span>) <span># see https://github.com/WinVector/WVPlots</span>

<span># parallel for vtreat</span>
ncores &lt;-<span> </span>parallel::<span>detectCores</span>()
parallelCluster &lt;-<span> </span>parallel::<span>makeCluster</span>(ncores)
<span># parallel for caret</span>
<span>registerDoMC</span>(<span>cores=</span>ncores)</code></pre>
</div>
<p>The we load our data for analysis. We are going to build a model predicting an income level from other demographic features. The data is taken from <a target="_blank" href="http://archive.ics.uci.edu/ml/machine-learning-databases/adult/">here</a> and you can perform all of the demonstrated steps if you download the contents of the <a target="_blank" href="https://github.com/WinVector/Examples/tree/master/CensusAdultIncomeExample">example git directory</a>. Obviously this has a lot of moving parts (R, R Markdown, Github, R packages, devtools)- but is very easy to do a second time (first time can be a bit of learning and preparation).</p>
<div>
<pre><code><span># load data</span>
<span># data from: http://archive.ics.uci.edu/ml/machine-learning-databases/adult/</span>
colnames &lt;-
<span> </span><span>c</span>(
    <span>'age'</span>,
    <span>'workclass'</span>,
    <span>'fnlwgt'</span>,
    <span>'education'</span>,
    <span>'education-num'</span>,
    <span>'marital-status'</span>,
    <span>'occupation'</span>,
    <span>'relationship'</span>,
    <span>'race'</span>,
    <span>'sex'</span>,
    <span>'capital-gain'</span>,
    <span>'capital-loss'</span>,
    <span>'hours-per-week'</span>,
    <span>'native-country'</span>,
    <span>'class'</span>
  )
dTrain &lt;-<span> </span><span>read.table</span>(
  <span>'adult.data.txt'</span>,
  <span>header =</span> <span>FALSE</span>,
  <span>sep =</span> <span>','</span>,
  <span>strip.white =</span> <span>TRUE</span>,
  <span>stringsAsFactors =</span> <span>FALSE</span>,
  <span>na.strings =</span> <span>c</span>(<span>'NA'</span>, <span>'?'</span>, <span>''</span>)
)
<span>colnames</span>(dTrain) &lt;-<span> </span>colnames
dTest &lt;-<span> </span><span>read.table</span>(
  <span>'adult.test.txt'</span>,
  <span>skip =</span> <span>1</span>,
  <span>header =</span> <span>FALSE</span>,
  <span>sep =</span> <span>','</span>,
  <span>strip.white =</span> <span>TRUE</span>,
  <span>stringsAsFactors =</span> <span>FALSE</span>,
  <span>na.strings =</span> <span>c</span>(<span>'NA'</span>, <span>'?'</span>, <span>''</span>)
)
<span>colnames</span>(dTest) &lt;-<span> </span>colnames</code></pre>
</div>
<p>Now we use <code>vtreat</code> to prepare the data for analysis. The goal of vtreat is to ensure a ready-to-dance data frame in a statistically valid manner. We are respecting the test/train split and building our data preparation plan only on the training data (though we do apply it to the test data). This step helps with a huge number of potential problems through automated repairs:</p>
<ul>
<li>re-encoding missing values</li>
<li>dealing with large cardinality categorical variables</li>
<li>dealing with novel levels</li>
<li>fixing variable/column names to be &quot;R safe&quot;</li>
<li>looking for strange column types</li>
</ul>
<div>
<pre><code><span># define problem</span>
yName &lt;-<span> 'class'</span>
yTarget &lt;-<span> '>50K'</span>
varNames &lt;-<span> </span><span>setdiff</span>(colnames,yName)

<span># build variable encoding plan and prepare simulated out of sample</span>
<span># training fame (cross-frame) </span>
<span># http://www.win-vector.com/blog/2016/05/vtreat-cross-frames/</span>
<span>system.time</span>({
  cd &lt;-<span> </span>vtreat::<span>mkCrossFrameCExperiment</span>(dTrain,varNames,yName,yTarget,
                                        <span>parallelCluster=</span>parallelCluster)
  scoreFrame &lt;-<span> </span>cd$treatments$scoreFrame
  dTrainTreated &lt;-<span> </span>cd$crossFrame
  <span># pick our variables</span>
  newVars &lt;-<span> </span>scoreFrame$varName[scoreFrame$sig&lt;<span>1</span>/<span>nrow</span>(scoreFrame)]
  dTestTreated &lt;-<span> </span>vtreat::<span>prepare</span>(cd$treatments,dTest,
                                  <span>pruneSig=</span><span>NULL</span>,<span>varRestriction=</span>newVars)
})</code></pre>
</div>
<pre><code>##    user  system elapsed 
##  11.340   2.760  30.872</code></pre>
<div>
<pre><code><span>#print(newVars)</span></code></pre>
</div>
<p>Now we train our model. In this case we are using the caret package to tune parameters.</p>
<div>
<pre><code><span># train our model using caret</span>
<span>system.time</span>({
  yForm &lt;-<span> </span><span>as.formula</span>(<span>paste</span>(yName,<span>paste</span>(newVars,<span>collapse=</span><span>' + '</span>),<span>sep=</span><span>' ~ '</span>))
  <span># from: http://topepo.github.io/caret/training.html</span>
  fitControl &lt;-<span> </span><span>trainControl</span>(## 10-fold CV
    <span>method =</span> <span>&quot;cv&quot;</span>,
    <span>number =</span> <span>3</span>)
  model &lt;-<span> </span><span>train</span>(yForm,
                 <span>data =</span> dTrainTreated,
                 <span>method =</span> <span>&quot;gbm&quot;</span>,
                 <span>trControl =</span> fitControl,
                 <span>verbose =</span> <span>FALSE</span>)
  <span>print</span>(model)
  dTest$pred &lt;-<span> </span><span>predict</span>(model,<span>newdata=</span>dTestTreated,<span>type=</span><span>'prob'</span>)[,yTarget]
})</code></pre>
</div>
<pre><code>## Stochastic Gradient Boosting 
## 
## 32561 samples
##    64 predictor
##     2 classes: '&lt;=50K', '>50K' 
## 
## No pre-processing
## Resampling: Cross-Validated (3 fold) 
## Summary of sample sizes: 21707, 21708, 21707 
## Resampling results across tuning parameters:
## 
##   interaction.depth  n.trees  Accuracy   Kappa    
##   1                   50      0.8476398  0.5083558
##   1                  100      0.8556555  0.5561726
##   1                  150      0.8577746  0.5699958
##   2                   50      0.8560855  0.5606650
##   2                  100      0.8593102  0.5810931
##   2                  150      0.8625042  0.5930111
##   3                   50      0.8593717  0.5789289
##   3                  100      0.8649919  0.6017707
##   3                  150      0.8660975  0.6073645
## 
## Tuning parameter 'shrinkage' was held constant at a value of 0.1
## 
## Tuning parameter 'n.minobsinnode' was held constant at a value of 10
## Accuracy was used to select the optimal model using  the largest value.
## The final values used for the model were n.trees = 150,
##  interaction.depth = 3, shrinkage = 0.1 and n.minobsinnode = 10.

##    user  system elapsed 
##  61.908   2.227  36.850</code></pre>
<p>Finally we take a look at the results on the held-out test data.</p>
<div>
<pre><code>WVPlots::<span>ROCPlot</span>(dTest,<span>'pred'</span>,yName,<span>'predictions on test'</span>)</code></pre>
</div>
<div>
<img title="score-1.png" alt="Score 1" src="http://i1.wp.com/www.win-vector.com/blog/wp-content/uploads/2016/06/score-1.png?w=660" border="0" data-recalc-dims="1">
</div>
<div>
<pre><code>WVPlots::<span>DoubleDensityPlot</span>(dTest,<span>'pred'</span>,yName,<span>'predictions on test'</span>)</code></pre>
</div>
<div>
<img title="score-2.png" alt="Score 2" src="http://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2016/06/score-2.png?w=660" border="0" data-recalc-dims="1">
</div>
<div>
<pre><code>confusionMatrix &lt;-<span> </span><span>table</span>(<span>truth=</span>dTest[[yName]],<span>pred=</span>dTest$pred>=<span>0.5</span>)
<span>print</span>(confusionMatrix)</code></pre>
</div>
<pre><code>##         pred
## truth    FALSE  TRUE
##   &lt;=50K. 11684   751
##   >50K.   1406  2440</code></pre>
<div>
<pre><code>testAccuarcy &lt;-<span> </span>(confusionMatrix[<span>1</span>,<span>1</span>]+confusionMatrix[<span>2</span>,<span>2</span>])/<span>sum</span>(confusionMatrix)
testAccuarcy</code></pre>
</div>
<pre><code>## [1] 0.8675143</code></pre>
<p>Notice the achieved test accuracy is in the ballpark of what was reported for this dataset.</p>
<pre><code>(From [adult.names description](http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.names) )
Error Accuracy reported as follows, after removal of unknowns from
 |    train/test sets):
 |    C4.5       : 84.46+-0.30
 |    Naive-Bayes: 83.88+-0.30
 |    NBTree     : 85.90+-0.28</code></pre>
<p>We can also compare accuracy on the &quot;complete cases&quot;:</p>
<div>
<pre><code>dTestComplete &lt;-<span> </span>dTest[<span>complete.cases</span>(dTest[,varNames]),]
confusionMatrixComplete &lt;-<span> </span><span>table</span>(<span>truth=</span>dTestComplete[[yName]],
                                 <span>pred=</span>dTestComplete$pred>=<span>0.5</span>)
<span>print</span>(confusionMatrixComplete)</code></pre>
</div>
<pre><code>##         pred
## truth    FALSE  TRUE
##   &lt;=50K. 10618   742
##   >50K.   1331  2369</code></pre>
<div>
<pre><code>testAccuarcyComplete &lt;-<span> </span>(confusionMatrixComplete[<span>1</span>,<span>1</span>]+confusionMatrixComplete[<span>2</span>,<span>2</span>])/
<span> </span><span>sum</span>(confusionMatrixComplete)
testAccuarcyComplete</code></pre>
</div>
<pre><code>## [1] 0.8623506</code></pre>
<div>
<pre><code><span># clean up</span>
parallel::<span>stopCluster</span>(parallelCluster)</code></pre>
</div>
<p>This is consistent with our experience that missingness is often actually informative, so in addition to imputing missing values you would like to preserver some notation indicating the missingness (which <code>vtreat</code> does in fact do).</p>
<p>And that is all there is to this example. I’d like to emphasize that vtreat steps were only a few lines in one of the blocks of code. <code>vtreat</code> treatment can take some time, but it is usually bearable. By design it is easy to add vtreat to your predictive analytics projects.</p>
<p>The point is: we got competitive results on real world data, in a single try (using vtreat to prepare data and caret to tune parameters). The job of the data scientist is to actually work longer on a problem and do better. But having a good start helps.</p>
<p>The theory behind vtreat is fairly important to the correctness of our implementation, and we would love for you to read through some of it:</p>
<ul>
<li><a target="_blank" href="http://www.win-vector.com/blog/2014/08/vtreat-designing-a-package-for-variable-treatment/">vtreat: designing a package for variable treatment</a></li>
<li><a target="_blank" href="http://www.win-vector.com/blog/2016/05/vtreat-cross-frames/">vtreat Cross Frames</a></li>
<li><a target="_blank" href="http://www.win-vector.com/blog/2016/04/on-nested-models/">On Nested Models</a></li>
<li><a target="_blank" href="http://winvector.github.io/vtreathtml/">vtreat manuals</a></li>
<li><a target="_blank" href="http://winvector.github.io/DataPrep/EN-CNTNT-Whitepaper-Data-Prep-Using-R.pdf">Preparing data for analysis using R</a></li>
<li><a target="_blank" href="http://www.win-vector.com/blog/2016/03/more-on-preparing-data/">More on preparing data</a></li>
<li><a target="_blank" href="https://cran.r-project.org/package=vtreat">vtreat on Cran</a></li>
<li><a target="_blank" href="https://github.com/WinVector/vtreat">vtreat on Github</a></li>
</ul>
<p>But operationally, please think of <code>vtreat</code> as just adding a couple of lines to your analysis scripts. Again, the raw R markdown source can be found <a target="_blank" href="https://github.com/WinVector/Examples/blob/master/CensusAdultIncomeExample/ExampleRun.Rmd">here</a> and a rendered copy (with results and graphs) <a target="_blank" href="https://github.com/WinVector/Examples/blob/master/CensusAdultIncomeExample/ExampleRun.md">here</a>.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/a-demonstration-of-vtreat-data-preparation/"></div></div><div><a data-url="http://www.r-bloggers.com/a-demonstration-of-vtreat-data-preparation/" data-counturl="http://www.r-bloggers.com/a-demonstration-of-vtreat-data-preparation/" data-count="vertical" data-text="A demonstration of vtreat data preparation" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2016/06/a-demonstration-of-vtreat-data-preparation/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/Gct7hw6cfKk"></a></div>