<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/r-tip-use-the-vtreat-package-for-data-preparation/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2018/03/r-tip-use-the-vtreat-package-for-data-preparation/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>If you are working with predictive modeling or machine learning in <a rel="nofollow" href="https://www.r-project.org/" target="_blank"><code>R</code></a> this is the <a rel="nofollow" href="http://www.win-vector.com/blog/tag/r-tips/" target="_blank"><code>R</code> tip</a> that is going to save you the <em>most</em> time and deliver the <em>biggest</em> improvement in your results.</p>
<p><a rel="nofollow" href="http://www.win-vector.com/blog/tag/r-tips/" target="_blank">R Tip</a>: Use the <a rel="nofollow" href="https://winvector.github.io/vtreat/" target="_blank"><code>vtreat</code> package</a> for data preparation in predictive analytics and machine learning projects.</p>
<p><img border="0" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2018/03/vtreat.png?resize=317%2C245" alt="Vtreat" width="317" title="vtreat.png" data-recalc-dims="1" height="245"></p>
<p>When attempting predictive modeling with real-world data you <em>quickly</em> run into difficulties beyond what is typically emphasized in machine learning coursework:</p>
<ul>
<li>Missing, invalid, or out of range values.</li>
<li>Categorical variables with large sets of possible levels.</li>
<li>Novel categorical levels discovered during test, cross-validation, or model application/deployment.</li>
<li>Large numbers of columns to consider as potential modeling variables (both statistically hazardous and time consuming).</li>
<li>Nested model bias poisoning results in non-trivial data processing pipelines.</li>
</ul>
<p>Any <em>one</em> of these issues can add to project time and decrease the predictive power and reliability of a machine learning project. Many real world projects encounter <em>all</em> of these issues, which are often ignored leading to degraded performance in production.</p>
<p><code>vtreat</code> systematically and correctly deals with <em>all</em> of the above issues in a documented, automated, parallel, and statistically sound manner.</p>
<p><code>vtreat</code> can fix or mitigate these <em>domain independent</em> issues much more reliably and much faster than by-hand ad-hoc methods.<br>
This leaves the data scientist or analyst more time to research and apply critical <em>domain dependent</em> (or knowledge based) steps and checks.</p>
<p>If you are attempting high-value predictive modeling in <code>R</code>, you should try out <code>vtreat</code> and consider adding it to your workflow.</p>
<p><span></span></p>
<p>Both the software and the write-up have citable DOIs to make them easier to include in your methods sections and other write-ups.</p>
<ul>
<li>Package: <a rel="nofollow" href="https://github.com/WinVector/vtreat" target="_blank">https://github.com/WinVector/vtreat</a>, DOI: <a rel="nofollow" href="https://doi.org/10.5281/zenodo.1173318" target="_blank"><img alt="DOI" src="https://zenodo.org/badge/DOI/10.5281/zenodo.1173318.svg"></a>.</li>
<li>Article DOI: <a rel="nofollow" href="https://arxiv.org/abs/1611.09477" target="_blank">arXiv:1611.09477 stat.AP</a> <a rel="nofollow" href="https://doi.org/10.5281/zenodo.1173314" target="_blank"><img src="https://zenodo.org/badge/DOI/10.5281/zenodo.1173314.svg"></a>.</li>
</ul>
<p><a rel="nofollow" href="https://winvector.github.io/vtreat/" target="_blank"><code>vtreat</code> <code>1.0.3</code></a> is now available for <a rel="nofollow" href="https://www.r-project.org/" target="_blank"><code>R</code></a> users <a rel="nofollow" href="https://cran.r-project.org/package=vtreat" target="_blank"> through <code>CRAN</code></a>.<br>
This <code>vtreat</code> release adds some parallel performance improvements and <a rel="nofollow" href="https://winvector.github.io/vtreat/reference/novel_value_summary.html" target="_blank">new methods to track and characterize novel variable levels</a>.</p>
<p><a rel="nofollow" href="http://www.win-vector.com/site/" target="_blank">Win-Vector LLC</a> offers semi-custom on-site training in the <code>vtreat</code> methodology (and support). Please reach out to us if your group is interested in such training.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2018/03/r-tip-use-the-vtreat-package-for-data-preparation/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/xr9kHv0msDk" height="1"></a></div>