<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/prepare-real-world-data-for-analysis-with-the-vtreat-package/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2017/04/vtreat.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>As anyone who's tried to analyze real-world data knows, there are any number of problems that may be lurking in the data that can prevent you from being able to fit a useful predictive model:</p>
<ul>
<li>Categorical variables can <a target="_blank" rel="nofollow" href="https://mran.revolutionanalytics.com/web/packages/vtreat/vignettes/vtreatRareLevels.html">include infrequently-used levels</a>, which will cause problems if sampling leaves them unrepresented in the training set.</li>
<li>Numerical variables can be in <a target="_blank" rel="nofollow" href="https://mran.revolutionanalytics.com/web/packages/vtreat/vignettes/vtreatScaleMode.html">wildly different scales</a>, which can cause instability when fitting models.</li>
<li>The data set may include several highly-correlated columns, some of which could be <a target="_blank" rel="nofollow" href="https://mran.revolutionanalytics.com/web/packages/vtreat/vignettes/vtreatSignificance.html">pruned from the data</a> without sacrificing predictive power.</li>
<li>The data set may include missing values that need to be dealt with before analysis can begin.</li>
<li>… and many others</li>
</ul>
<p>The <a target="_blank" rel="nofollow" href="https://mran.revolutionanalytics.com/package/vtreat/">vtreat package</a> is designed to counter common data problems like these in a statistically sound manner. It's a data frame preprocessor which applies a number of data cleaning processes to the input data before analysis, using techniques such as <a target="_blank" rel="nofollow" href="http://www.win-vector.com/blog/2012/07/modeling-trick-impact-coding-of-categorical-variables-with-many-levels/">impact coding</a> and <a target="_blank" rel="nofollow" href="http://www.win-vector.com/blog/2014/12/a-comment-on-preparing-data-for-classifiers/">categorical variable encoding</a> (the methods are described in detail in <a target="_blank" rel="nofollow" href="https://arxiv.org/abs/1611.09477">this paper</a>). Further details can be found on the <a target="_blank" rel="nofollow" href="https://github.com/WinVector/vtreat">vtreat github page</a>, where authors John Mount and Nina Zumel note:</p>
<blockquote>
<p>Even with modern machine learning techniques (random forests, support vector machines, neural nets, gradient boosted trees, and so on) or standard statistical methods (regression, generalized regression, generalized additive models) there are <em>common</em> data issues that can cause modeling to fail. vtreat deals with a number of these in a principled and automated fashion.</p>
</blockquote>
<p><img title="Vtreat" alt="Vtreat" src="http://a2.typepad.com/6a0105360ba1c6970c01bb098f866a970d-800wi" border="0"></p>
<p>One final note: the main function in the package, <a target="_blank" rel="nofollow" href="https://mran.revolutionanalytics.com/web/packages/vtreat/vignettes/vtreat.html">prepare</a>, is a little like <a target="_blank" rel="nofollow" href="https://www.rdocumentation.org/packages/stats/versions/3.3.3/topics/model.matrix">model.matrix</a> in that categorical variables are converted into numeric variables using contrast codings. This means that the output is suitable for many machine-learning functions (like <a target="_blank" rel="nofollow" href="https://mran.revolutionanalytics.com/package/xgboost/">xgboost</a>) that don't accept categorical variables.</p>
<p>The vtreat package is available on CRAN now, and you can find a worked example using vtreat in the blog post linked below.</p>
<p>Win-Vector Blog: <a target="_blank" rel="nofollow" href="http://www.win-vector.com/blog/2017/03/vtreat-prepare-data/">vtreat: prepare data</a></p>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2017/04/vtreat.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/WvLIS6r21JY"></a></div>