<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/easy-leave-one-out-cross-validation-with-pipelearner/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://drsimonj.svbtle.com/easy-leave-one-out-cross-validation-with-pipelearner"> blogR</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a target="_blank" rel="nofollow" href="https://twitter.com/drsimonj">@drsimonj</a> here to show you how to do <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Cross-validation_(statistics)">leave-one-out cross validation</a> using pipelearner.</p>
<h2><a target="_blank" rel="nofollow" href="http://drsimonj.svbtle.com/easy-leave-one-out-cross-validation-with-pipelearner#leaveoneout-cross-validation_2"> </a>Leave-one-out cross validation</h2>
<p>Leave-one-out is a type of cross validation whereby the following is done for each observation in the data:</p>
<ul>
<li> Run model on all other observations</li>
<li> Use model to predict value for observation</li>
</ul>
<p>This means that a model is fitted, and a predicted is made <em>n</em> times where <em>n</em> is the number of observations in your data.</p>
<h2><a target="_blank" rel="nofollow" href="http://drsimonj.svbtle.com/easy-leave-one-out-cross-validation-with-pipelearner#leaveoneout-in-pipelearner_2"> </a>Leave-one-out in pipelearner</h2>
<p><a target="_blank" rel="nofollow" href="https://github.com/drsimonj/pipelearner">pipelearner</a> is a package for streamlining machine learning pipelines, including cross validation. If you’re new to it, check out <a target="_blank" rel="nofollow" href="https://drsimonj.svbtle.com/">blogR</a> for other relevant posts.</p>
<p>To demonstrate, let’s use regression to predict horsepower (<code>hp</code>) with all other variables in the <code>mtcars</code> data set. Set this up in pipelearner as follows:</p>
<pre><code>library(pipelearner)

pl &lt;- pipelearner(mtcars, lm, hp ~ .)
</code></pre>
<p>How cross validation is done is handled by <code>learn_cvpairs()</code>. For leave-one-out, specify <em>k = number of rows</em>:</p>
<pre><code>pl &lt;- learn_cvpairs(pl, k = nrow(mtcars))
</code></pre>
<p>Finally, <code>learn()</code> the model on all folds:</p>
<pre><code>pl &lt;- learn(pl)
</code></pre>
<p>This can all be written in a pipeline:</p>
<pre><code>pl &lt;- pipelearner(mtcars, lm, hp ~ .) %>% 
  learn_cvpairs(k = nrow(mtcars)) %>% 
  learn()

pl
#> # A tibble: 32 × 9
#>    models.id cv_pairs.id train_p      fit target model     params
#>        &lt;chr>       &lt;chr>   &lt;dbl>   &lt;list>  &lt;chr> &lt;chr>     &lt;list>
#> 1          1          01       1 &lt;S3: lm>     hp    lm &lt;list [1]>
#> 2          1          02       1 &lt;S3: lm>     hp    lm &lt;list [1]>
#> 3          1          03       1 &lt;S3: lm>     hp    lm &lt;list [1]>
#> 4          1          04       1 &lt;S3: lm>     hp    lm &lt;list [1]>
#> 5          1          05       1 &lt;S3: lm>     hp    lm &lt;list [1]>
#> 6          1          06       1 &lt;S3: lm>     hp    lm &lt;list [1]>
#> 7          1          07       1 &lt;S3: lm>     hp    lm &lt;list [1]>
#> 8          1          08       1 &lt;S3: lm>     hp    lm &lt;list [1]>
#> 9          1          09       1 &lt;S3: lm>     hp    lm &lt;list [1]>
#> 10         1          10       1 &lt;S3: lm>     hp    lm &lt;list [1]>
#> # ... with 22 more rows, and 2 more variables: train &lt;list>, test &lt;list>
</code></pre>
<h2><a target="_blank" rel="nofollow" href="http://drsimonj.svbtle.com/easy-leave-one-out-cross-validation-with-pipelearner#evaluating-performance_2"> </a>Evaluating performance</h2>
<p>Performance can be evaluated in many ways depending on your model. We will calculate R<sup>2</sup>:</p>
<pre><code>library(tidyverse)

# Extract true and predicted values of hp for each observation
pl &lt;- pl %>% 
  mutate(true = map2_dbl(test, target, ~as.data.frame(.x)[[.y]]),
         predicted = map2_dbl(fit, test, predict)) 

# Summarise results
results &lt;- pl %>% 
  summarise(
    sse = sum((predicted - true)^2),
    sst = sum(true^2)
  ) %>% 
  mutate(r_squared = 1 - sse / sst)

results
#> # A tibble: 1 × 3
#>        sse    sst r_squared
#>      &lt;dbl>  &lt;dbl>     &lt;dbl>
#> 1 41145.56 834278 0.9506812
</code></pre>
<p>Using leave-one-out cross validation, the regression model obtains an R<sup>2</sup> of 0.95 when generalizing to predict horsepower in new data.</p>
<p>We’ll conclude with a plot of each true data point and it’s predicted value:</p>
<pre><code>pl %>% 
  ggplot(aes(true, predicted)) +
      geom_point(size = 2) +
      geom_abline(intercept = 0, slope = 1, linetype = 2)  +
      theme_minimal() +
      labs(x = &quot;True value&quot;, y = &quot;Predicted value&quot;) +
      ggtitle(&quot;True against predicted values based\non leave-one-one cross validation&quot;)
</code></pre>
<p><a target="_blank" rel="nofollow" href="https://i2.wp.com/svbtleusercontent.com/ukzojupcyh78q.jpeg?ssl=1"><img alt="unnamed-chunk-7-1.jpeg" src="https://i1.wp.com/svbtleusercontent.com/ukzojupcyh78q_small.jpeg?w=456&ssl=1" data-recalc-dims="1"></a></p>
<h2><a target="_blank" rel="nofollow" href="http://drsimonj.svbtle.com/easy-leave-one-out-cross-validation-with-pipelearner#sign-off_2"> </a>Sign off</h2>
<p>Thanks for reading and I hope this was useful for you.</p>
<p>For updates of recent blog posts, follow <a target="_blank" rel="nofollow" href="https://twitter.com/drsimonj">@drsimonj</a> on Twitter, or email me at <a target="_blank" rel="nofollow" href="mailto:drsimonjackson@gmail.com">drsimonjackson@gmail.com</a> to get in touch.</p>
<p>If you’d like the code that produced this blog, check out the <a target="_blank" rel="nofollow" href="https://github.com/drsimonj/blogR">blogR GitHub repository</a>.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://drsimonj.svbtle.com/easy-leave-one-out-cross-validation-with-pipelearner"> blogR</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/5EdmdRRn_Ak"></a></div>