<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/ensemble-methods-are-doomed-to-fail-in-high-dimensions/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://andrewgelman.com/2017/03/15/ensemble-methods-doomed-fail-high-dimensions/"> R – Statistical Modeling, Causal Inference, and Social Science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><b>Ensemble methods</b><br>
<a target="_blank" rel="nofollow" href="http://westsidetheatrereviews.blogspot.com/2013/07/broadway-rose-cats-done-right.html"><img src="https://i0.wp.com/andrewgelman.com/wp-content/uploads/2017/03/cat-ensemble.jpg?w=400" data-recalc-dims="1"></a></p>
<p>By ensemble methods, I (Bob, not Andrew) mean approaches that scatter points in parameter space and then make moves by inteprolating or extrapolating among subsets of them. Two prominent examples are:</p>
<ul>
<li> <a target="_blank" rel="nofollow" href="http://library.wur.nl/WebQuery/wurpubs/fulltext/26336">Ter Braak’s differential evolution</a> <br>  
<li> <a target="_blank" rel="nofollow" href="http://msp.org/camcos/2010/5-1/camcos-v5-n1-p04-s.pdf"> Goodman and Weare’s walkers</a>
</ul>
<p>There are extensions and computer implementations of these algorithms. For example, the <a target="_blank" rel="nofollow" href="http://dan.iel.fm/emcee/current/">Python package emcee</a> implements Goodman and Weare’s walker algorithm and is popular in astrophysics.</p>
<p><b>Typical sets in high dimensions</b></p>
<p>If you want to get the background on typical sets, I’d highly recommend Michael Betancourt’s video lectures on MCMC in general and HMC in particular; they both focus on typical sets and their relation to the integrals we use MCMC to calculate:</p>
<ul>
<li>StanCon 2017: <a target="_blank" rel="nofollow" href="https://www.youtube.com/watch?v=DJ0c7Bm5Djk">Everything you <i>should</i> have learned about Markov chain Monte Carlo</a> (starts at 4:40 or so; sorry for crummy video—the conference was DIY on a tight budget) <br>  
<li>Tokyo Stan: <a target="_blank" rel="nofollow" href="https://www.youtube.com/watch?v=VnNdhsm0rJQ">Scalable Bayesian Inference with Hamiltonian Monte Carlo</a>
</ul>
<p>It was Michael who made a doughnut in the air, pointed at the middle of it and said, “It’s obvious ensemble methods won’t work.” This is just fleshing out the details with some code for the rest of us without such sharp geometric intuitions.</p>
<p><a target="_blank" rel="nofollow" href="http://www.inference.phy.cam.ac.uk/itprnn/book.html">MacKay’s information theory book</a> is another excellent source on typical sets. Don’t bother with the Wikipedia on this one.</p>
<p><b>Why ensemble methods fail: Executive summary</b></p>
<ol>
<li>We want to draw a sample from the typical set
<li>The typical set is a thin shell a fixed radius from the mode in a multivariate normal
<li>Interpolating or extrapolating two points in this shell is unlikely to fall in this shell
<li>The only steps that get accepted will be near one of the starting points
<li>The samplers devolve to a random walk with poorly biased choice of direction
</ol>
<p>Several years ago, Peter Li built the Goodman and Weare walker methods for Stan (all they need is log density) on a branch for evaluation. They failed in practice exactly the way the theory says they will fail. Which is too bad, because the ensemble methods are very easy to implement and <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Embarrassingly_parallel">embarassingly parallel</a>.</p>
<p><b>Why ensemble methods fail: R simulation</b></p>
<p>OK, so let’s see why they fail in practice. I’m going to write some simple R code to do the job for us. Here’s an R function to generate a 100-dimensional standard isotropic normal variate (each element is generated normal(0, 1) independently):</p>
<pre>
normal_rng &lt;- function(K) rnorm(K);
</pre>
<p>This function computes the log density of a draw:</p>
<pre>
normal_lpdf &lt;- function(y) sum(dnorm(y, log=TRUE));
</pre>
<p>Next, generate two draws from a 100-dimesnional version:</p>
<pre>
K &lt;- 100;
y1 &lt;- normal_rng(K);
y2 &lt;- normal_rng(K);
</pre>
<p>and then interpolate by choosing a point between them:</p>
<pre>
lambda = 0.5;
y12 &lt;- lambda * y1 + (1 - lambda) * y2;
</pre>
<p>Now let's see what we get:</p>
<pre>
print(normal_lpdf(y1), digits=1);
print(normal_lpdf(y2), digits=1);
print(normal_lpdf(y12), digits=1);
</pre>
<p><i>
<pre>
[1] -153
[1] -142
[1] -123
</pre>
<p></p></i>
<p>Hmm. Why is the log density of the interpolated vector so much higher? Given that it's a multivariate normal, the answer is that it's closer to the mode. That should be a good thing, right? No, it's not. The typical set is defined as an area within &quot;typical&quot; density bounds. When I take a random draw from a 100-dimensional standard normal, I expect log densities that hover between -140 and -160 or so. That interpolated vector y12 with a log density of -123 <b><i>isn't in the typical set</i></b>!!! It's a bad draw, even though it's closer to the mode. Still confused? Watch Michael's videos above. Ironically, there's a description in the Goodman and Weare paper in a discussion of why they can use ensemble averages that also explains why their own sampler doesn't scale---the variance of averages is lower than the variance of individual draws; and we want to cover the actual posterior, not get closer to the mode.</p>
<p>So let's put this in a little sharper perspective by simulating thousands of draws from a multivariate normal and thousands of draws interpolating between pairs of draws and plot them in two histograms. First, draw them and print a summary:</p>
<pre>
lp1 &lt;- vector();
for (n in 1:1000) lp1[n] &lt;- normal_lpdf(normal_rng(K));
print(summary(lp1));

lp2 &lt;- vector()
for (n in 1:1000) lp2[n] &lt;- normal_lpdf((normal_rng(K) + normal_rng(K))/2);
print(summary(lp2));
</pre>
<p>from which we get:</p>
<p><i>
<pre>
  Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   -177    -146    -141    -142    -136    -121 

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   -129    -119    -117    -117    -114    -108 
</pre>
<p></p></i>
<p>That's looking bad. It's even clearer with a faceted histogram:</p>
<pre>
library(ggplot2);
df &lt;- data.frame(list(log_pdf = c(lp1, lp2),
                      case=c(rep(&quot;Y1&quot;, 1000), rep(&quot;(Y1 + Y2)/2&quot;, 1000))));
plot &lt;- ggplot(df, aes(log_pdf)) +
        geom_histogram(color=&quot;grey&quot;) +
        facet_grid(case ~ .);
</pre>
<p>Here's the plot:</p>
<p><img src="https://i0.wp.com/andrewgelman.com/wp-content/uploads/2017/03/ensemble-draws.jpg?w=450" data-recalc-dims="1"></p>
<p>The bottom plot shows the distribution of log densities in independent draws from the standard normal (these are pure Monte Carlo draws). The top plot shows the distribution of the log density of the vector resulting from interpolating two independent draws from the same distribution. Obviously, the log densities of the averaged draws are much higher. In other words, they are atypical of draws from the target standard normal density.</p>
<p><b>Exercise</b></p>
<p>Check out what happens as (1) the number of dimensions K varies, and (2) as lambda varies within or outside of [0, 1]. </p>
<p><i>Hint: What you should see is that as lambda approaches 0 or 1, the draws get more and more typical, and more and more like random walk Metropolis with a small step size. As dimensionality increases, the typical set becomes more attenuated and the problem becomes worse (and vice-versa as it decreases).</i></p>
<p><b>Does Hamiltonian Monte Carlo (HMC) have these problems?</b></p>
<p>Not so much. It scales much better with dimension. It'll slow down, but it won't break and devolve to a random walk like ensemble methods do.</p>
<p>The post <a target="_blank" rel="nofollow" href="http://andrewgelman.com/2017/03/15/ensemble-methods-doomed-fail-high-dimensions/">Ensemble Methods are Doomed to Fail in High Dimensions</a> appeared first on <a target="_blank" rel="nofollow" href="http://andrewgelman.com/">Statistical Modeling, Causal Inference, and Social Science</a>.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://andrewgelman.com/2017/03/15/ensemble-methods-doomed-fail-high-dimensions/"> R – Statistical Modeling, Causal Inference, and Social Science</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/JEJx5s3C1Yc"></a></div>