<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/data-preparation-long-form-and-tldr-form/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2016/12/data-preparation-long-form-and-tldr-form/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Data preparation and cleaning are some of <em>the</em> most important steps of predictive analytic and data science tasks. They are laborious, where most of the errors are made, your last line of defense against a wild data, and hold the biggest opportunities for outcome improvement. No matter how much time you spend on then, they still seem like a neglected topic. Data preparation isn’t as self contained or genteel as tweaking machine learning models or hyperparameter tuning; and that is one of the reasons data preparation represents such an important practical opportunity for improvement.</p>
<p><center><br>
<a target="_blank" rel="nofollow" href="http://www.picserver.org/c/cleaning.html"><img title="NewImage.png" alt="NewImage" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2016/12/NewImage-9.png?resize=349%2C303" border="0" data-recalc-dims="1">
<p>Photo: NY – http://nyphotographic.com/, License: Creative Commons 3 – CC BY-SA 3.0</a><br>
</center></p>
<p>Our group is distributing a detailed write up of the theory and operation behind our <a target="_blank" rel="nofollow" href="https://cran.r-project.org/">R</a> realization of a set of sound data preparation and cleaning procedures called <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=vtreat">vtreat</a> here: <a target="_blank" rel="nofollow" href="https://arxiv.org/abs/1611.09477">arXiv:1611.09477 [stat.AP]</a>. This is where you can find out what <code>vtreat</code> does, decide if it is appropriate for your problem, or even find a specification allowing the use of the techniques in non-<code>R</code> environments (such as <code>Python</code>/<code>Pandas</code>/<code>scikit-learn</code>, <code>Spark</code>, and many others).</p>
<p>We have submitted this article for formal publication, so it is our intent you can cite this article (as it stands) in scientific work as a pre-print, and later cite it from a formally refereed source.</p>
<p>Or alternately, below is the tl;dr (“too long; didn’t read”) form.<span></span></p>
<p>
Our concrete advice is: when building a supervised model (regression or classification) in <code>R</code>, prepare your training, test, and application data by doing the following.
<pre>
# load the vtreat package
library(&quot;vtreat&quot;)

# use your training data to design 
# data treatment plan
ce &lt;- mkCrossFrameCExperiment(trainData, 
                              vars,
                              yName, yTarget)

# look at the variable scores
varScores &lt;- ce$treatments$scoreFrame
print(varScores)

# prune variables based on significance
pruneSig &lt;- 1/nrow(varScores)
modelVars &lt;- varScores$varName[varScores$sig&lt;=pruneSig]

# instead of preparing training data, use 
# &quot;simulated out of sample data&quot; to reduce modeling bias
treatedTrainData &lt;- ce$crossFrame

# prepare any other data (test, future application)
# using the treatment plan
treatedTestData &lt;- prepare(ce$treatments, 
                           testData, 
                           varRestriction= modelVars, 
                           pruneSig= NULL)
</pre>
<p>Then <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/vtreat/vignettes/vtreat.html">work through our examples</a> to find out what all these steps are doing for you.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2016/12/data-preparation-long-form-and-tldr-form/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/9FT7SKRe6kI"></a></div>