<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/loo-2-0-is-loose/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://andrewgelman.com/2018/04/16/loo-2-0-loose/"> R – Statistical Modeling, Causal Inference, and Social Science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>This post is by Jonah and Aki.</p>
<p>We’re happy to announce the release of v2.0.0 of the <a rel="nofollow" href="http://mc-stan.org/loo" target="_blank"><strong>loo</strong></a> R package for efficient approximate leave-one-out cross-validation (and more). For anyone unfamiliar with the package, the original motivation for its development is in our paper:</p>
<blockquote>
<p>Vehtari, A., Gelman, A., and Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. <em>Statistics and Computing</em>. 27(5), 1413–1432. doi:10.1007/s11222-016-9696-4. (<a rel="nofollow" href="http://link.springer.com/article/10.1007%2Fs11222-016-9696-4" target="_blank">published version</a>, <a rel="nofollow" href="http://arxiv.org/abs/1507.04544" target="_blank">arXiv preprint</a>)</p>
</blockquote>
<p>Version 2.0.0 is a major update <a rel="nofollow" href="http://mc-stan.org/loo/news" target="_blank">(release notes)</a> to the package that we’ve been working on for quite some time and in this post we’ll highlight some of the most important improvements. Soon I (Jonah) will follow up with a post about important new developments in our various other R packages.</p>
<p><strong>New interface, vignettes, and more helper functions to make the package easier to use</strong></p>
<p>Because of certain improvements to the algorithms and diagnostics (summarized below), the interfaces, i.e., the <a rel="nofollow" href="http://mc-stan.org/loo/reference/loo" target="_blank"><code>loo()</code></a> and <a rel="nofollow" href="http://mc-stan.org/loo/reference/psis" target="_blank"><code>psis()</code></a> functions and the objects they return, also needed some improvement. (Click on the function names in the previous sentence to see their new documentation pages.) Other related packages in the Stan R ecosystem, e.g., <strong>rstanarm</strong>, <strong>brms</strong>, <strong>bayesplot</strong>, and others have also been updated to integrate seamlessly with <strong>loo</strong> v2.0.0. (Apologies to anyone who happened to install the update during the short window between the <strong>loo</strong> release and when the compatible rstanarm/brms binaries became available on CRAN.)</p>
<p>Three vignettes now come with the <strong>loo</strong> package package and are also available (and more nicely formatted) online at <a rel="nofollow" href="http://mc-stan.org/loo/articles/index.html" target="_blank">mc-stan.org/loo/articles</a>:</p>
<ul>
<li><em>Using the loo package (version >= 2.0.0)</em> (<a rel="nofollow" href="http://mc-stan.org/loo/articles/loo2-example.html" target="_blank">view</a>)</li>
<li><em>Bayesian Stacking and Pseudo-BMA weights using the loo package</em> (<a rel="nofollow" href="http://mc-stan.org/loo/articles/loo2-weights.html" target="_blank">view</a>)</li>
<li><em>Writing Stan programs for use with the loo package</em> (<a rel="nofollow" href="http://mc-stan.org/loo/articles/loo2-with-rstan.html" target="_blank">view</a>)</li>
</ul>
<p>A vignette about K-fold cross-validation using new K-fold helper functions will be included in a subsequent update. Since the last release of <strong>loo</strong> we have also written a paper, <a rel="nofollow" href="http://arxiv.org/abs/1709.01449" target="_blank">Visualization in Bayesian workflow</a>, that includes several visualizations based on computations from <strong>loo</strong>.</p>
<p><strong>Improvements to the PSIS algorithm, effective sample sizes and MC errors</strong></p>
<p>The approximate leave-one-out cross-validation performed by the <strong>loo</strong> package depends on Pareto smoothed importance sampling (PSIS). In <strong>loo</strong> v2.0.0, the PSIS algorithm (<a rel="nofollow" href="http://mc-stan.org/loo/reference/psis" target="_blank"><code>psis()</code></a> function) corresponds to the algorithm in the most recent update to our <a rel="nofollow" href="https://arxiv.org/abs/1507.02646" target="_blank">PSIS paper</a>, including adapting the Pareto fit with respect to the effective sample size and using a weakly informative prior to reduce the variance for small effective sample sizes. (I believe we’ll be updating the paper again with some proofs from new coauthors.)</p>
<p>For users of the <strong>loo</strong> package for PSIS-LOO cross-validation and not just the PSIS algorithm for importance sampling, an even more important update is that the latest version of the same PSIS paper referenced above describes how to compute the effective sample size estimate and Monte Carlo error for the PSIS estimate of <code>elpd_loo</code> (expected log predictive density for new data). Thus, in addition to the Pareto k diagnostic (an indicator of convergence rate – see <a rel="nofollow" href="https://arxiv.org/abs/1507.02646" target="_blank">paper</a>) already available in previous <strong>loo</strong> versions, we now also report an effective sample size that takes into account both the MCMC efficiency and the importance sampling efficiency. Here’s an example of what the diagnostic output table from <strong>loo</strong> v2.0.0 looks like (the particular intervals chosen for binning are explained in the papers and also the package <a rel="nofollow" href="http://mc-stan.org/loo/reference/pareto-k-diagnostic.html#details" target="_blank">documentation</a>) for the diagnostics:</p>
<pre>Pareto k diagnostic values:
                         Count Pct.    Min. n_eff
(-Inf, 0.5]   (good)     240   91.6%   205
 (0.5, 0.7]   (ok)         7    2.7%   48
   (0.7, 1]   (bad)        8    3.1%   7
   (1, Inf)   (very bad)   7    2.7%   1
</pre>
<p>We also compute and report the Monte Carlo SE of <code>elpd_loo</code> to give an estimate of the accuracy. If some k>1 (which means the PSIS-LOO approximation is not reliable, as in the example above) NA will be reported for the Monte Carlo SE. We hope that showing the relationship between the k diagnostic, effective sample size, and and MCSE of <code>elpd_loo</code> will make it easier to interpret the diagnostics than in previous versions of <strong>loo</strong> that only reported the k diagnostic.<strong> </strong>This particular example is taken from one of the new <a rel="nofollow" href="http://mc-stan.org/loo/articles/loo2-example.html" target="_blank">vignettes</a>, which uses it as part of a comparison of unstable and stable PSIS-LOO behavior.</p>
<p><strong>Weights for model averaging: Bayesian stacking, pseudo-BMA and pseudo-BMA+</strong></p>
<p>Another major addition is the <a rel="nofollow" href="http://mc-stan.org/loo/reference/loo_model_weights.html" target="_blank"><code>loo_model_weights()</code></a> function, which, thanks to the contributions of Yuling Yao, can be used to compute weights for model averaging or selection. <code>loo_model_weights()</code> provides a user friendly interface to the new <code>stacking_weights()</code> and <code>pseudobma_weights()</code>, which are implementations of the methods from <a rel="nofollow" href="https://projecteuclid.org/euclid.ba/1516093227" target="_blank"> Using stacking to average Bayesian predictive distributions</a> (Yao et al., 2018). As shown in the paper, Bayesian stacking (the default for <code>loo_model_weights()</code>) provides better model averaging performance than “Akaike style“ weights, however, the <strong>loo </strong>package does also include Pseudo-BMA weights (PSIS-LOO based “Akaike style“ weights) and Pseudo-BMA+ weights, which are similar to Pseudo-BMA weights but use a so-called Bayesian bootstrap procedure to  better account for the uncertainties. We recommend the Pseudo-BMA+ method instead of, for example, WAIC weights, although we prefer the stacking method to both. In addition to the Yao et al. paper, the <a rel="nofollow" href="http://mc-stan.org/loo/articles/loo2-weights.html" target="_blank">new vignette</a> about computing model weights demonstrates some of the motivation for our preference for stacking when appropriate.</p>
<p><strong>Give it a try</strong></p>
<p>You can install <strong>loo</strong> v2.0.0 from CRAN with <code>install.packages(&quot;loo&quot;)</code>. Additionally, reinstalling an interface that provides <strong>loo</strong> functionality (e.g., <strong>rstanarm</strong>,<strong> </strong><strong>brms</strong>) will automatically update your <strong>loo</strong> installation. The <strong>loo</strong> website with online documentation is <a rel="nofollow" href="http://mc-stan.org/loo" target="_blank">mc-stan.org/loo</a> and you can report a bug or request a feature on <a rel="nofollow" href="http://github.com/stan-dev/loo" target="_blank">GitHub</a>.</p>
<p>The post <a rel="nofollow" href="http://andrewgelman.com/2018/04/16/loo-2-0-loose/" target="_blank">loo 2.0 is loose</a> appeared first on <a rel="nofollow" href="http://andrewgelman.com/" target="_blank">Statistical Modeling, Causal Inference, and Social Science</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://andrewgelman.com/2018/04/16/loo-2-0-loose/"> R – Statistical Modeling, Causal Inference, and Social Science</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/Zw1jLcX6mEs" height="1"></a></div>