<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/a-theory-of-nested-cross-simulation/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2017/01/a-theory-of-nested-cross-simulation/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>[Reader’s Note. Some of our articles are applied and some of our articles are more theoretical. The following article is more theoretical, and requires fairly formal notation to even work through. However, it should be of interest as it touches on some of the fine points of cross-validation that are quite hard to perceive or discuss without the notational framework. We thought about including some “simplifying explanatory diagrams” but so many entities are being introduced and manipulated by the processes we are describing we found equation notation to be in fact cleaner than the diagrams we attempted and rejected.]</p>
<p>Please consider either of the following common predictive modeling tasks:</p>
<ul>
<li>Picking hyper-parameters, fitting a model, and then evaluating the model.</li>
<li>Variable preparation/pruning, fitting a model, and then evaluating the model.</li>
</ul>
<p>In each case you are building a pipeline where <a target="_blank" rel="nofollow" href="http://www.win-vector.com/blog/2016/06/y-aware-scaling-in-context/">“y-aware”</a> (or outcome aware) choices and transformations made at each stage affect later stages. This can introduce undesirable <a target="_blank" rel="nofollow" href="http://www.win-vector.com/blog/2016/04/on-nested-models/">nested model bias</a> and over-fitting.</p>
<p>Our <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/vtreat/vignettes/vtreatCrossFrames.html">current standard advice to avoid nested model bias</a> is either:</p>
<ul>
<li>Split your data into 3 or more disjoint pieces, such as separate variable preparation/pruning, model fitting, and model evaluation.</li>
<li>Reserve a test-set for evaluation and use “simulated out of sample data” or <a target="_blank" rel="nofollow" href="http://www.win-vector.com/blog/2016/05/vtreat-cross-frames/">“cross-frame”/“cross simulation”</a> techniques to simulate dividing data among the first two model construction stages.</li>
</ul>
<p>The first practice is simple and computationally efficient, but statistically inefficient. This may not matter if you have a lot of data, as in “big data”. The second procedure is more statistically efficient, but is also more complicated and has some computational cost. For convenience the cross simulation method is supplied as a <a target="_blank" rel="nofollow" href="http://www.win-vector.com/blog/2016/12/data-preparation-long-form-and-tldr-form/">ready to go procedure</a> in our <a target="_blank" rel="nofollow" href="https://cran.r-project.org/"><code>R</code></a> data cleaning and preparation package <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=vtreat"><code>vtreat</code></a>.</p>
<p>What would it look like if we insisted on using cross simulation or simulated out of sample techniques for all three (or more) stages? <a target="_blank" rel="nofollow" href="http://winvector.github.io/CrossSimulation/NestedCrossSimulation.html">Please read on to find out.</a></p>
<p><img title="responsibility12(alternate)-2.png" alt="CleanAllTheThings" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2016/12/responsibility12alternate-2.png?resize=400%2C300" border="0" data-recalc-dims="1"></p>
<p>Hyperbole and a Half copyright Allie Brosh (use allowed <a target="_blank" rel="nofollow" href="http://hyperboleandahalf.blogspot.com/p/faq_10.html">in some situations with attribution</a>)<br>
</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2017/01/a-theory-of-nested-cross-simulation/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/q5etAloR96c"></a></div>