<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/partial-pooling-for-lower-variance-variable-encoding/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2017/09/partial-pooling-for-lower-variance-variable-encoding/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>

<div>
<p><br>
</p>
<p></p>
<figure><img border="0" alt="Terraces" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/09/terraces.jpg?w=660" data-recalc-dims="1"><br>
<figcaption align="center"><center><small>Banaue rice terraces. Photo: <a rel="nofollow" href="https://www.flickr.com/photos/london/319347366" target="_blank">Jon Rawlinson</a> </small></center></figcaption>
</figure>
<p>In a previous article, we showed the use of partial pooling, or hierarchical/multilevel models, for level coding high-cardinality categorical variables in <a rel="nofollow" href="https://winvector.github.io/vtreat/" target="_blank"><code>vtreat</code></a>. In this article, we will discuss a little more about the how and why of partial pooling in <a rel="nofollow" href="https://www.r-project.org/" target="_blank"><code>R</code></a>.</p>
<p>We will use the <code>lme4</code> package to fit the hierarchical models. The acronym “lme” stands for “linear mixed-effects” models: models that combine so-called “fixed effects” and “random effects” in a single (generalized) linear model. The <code>lme4</code> documentation uses the random/fixed effects terminology, but we are going to follow Gelman and Hill, and avoid the use of the terms “fixed” and “random” effects.</p>
<blockquote><p>The varying coefficients [corresponding to the levels of a categorical variable] in a multilevel model are sometimes called <em>random effects</em>, a term that refers to the randomness in the probability model for the group-level coefficients….</p>
</blockquote>
<blockquote><p>The term <em>fixed effects</em> is used in contrast to random effects – but not in a consistent way! … Because of the conflicting definitions and advice, we will avoid the terms “fixed” and “random” entirely, and focus on the description of the model itself…</p>
</blockquote>
<p>– Gelman and Hill 2007, Chapter 11.4</p>
<p>We will also restrict ourselves to the case that <code>vtreat</code> considers: partially pooled estimates of conditional group expectations, with no other predictors considered.</p>
<p><span></span></p>
<h2>The Data</h2>
<div>
<p>Let’s assume that the data is generated from a mixture of <span>\(M\)</span> populations; each population is normally distributed with (unknown) means <span>\(\mu_{gp}\)</span>, all with the same (unknown) standard deviation <span>\(\sigma_w\)</span>:</p>
<p><span>\[<br>
y_{gp} = N(\mu_{gp}, {\sigma_{w}}^2)<br>
\]</span></p>
<p>The population means themselves are normally distributed, with unknown mean <span>\(\mu_0\)</span> and unknown standard deviation <span>\(\sigma_b\)</span>:</p>
<p><span>\[<br>
\mu_{gp} = N(\mu_0, {\sigma_{b}}^2)<br>
\]</span></p>
<p>(The subscripts <em>w</em> and <em>b</em> stand for “within-group” and “between-group” standard deviations, respectively.)</p>
<p>We can generate a synthetic data set according to these assumptions, with distributions similar to the distributions observed in the radon data set that we used in our earlier post: 85 groups, sampled unevenly. We’ll use <span>\(\mu_0 = 0, \sigma_w = 0.7, \sigma_b = 0.5\)</span>. Here, we take a peek at our data, <code>df</code>.</p>
<pre><code>head(df)</code></pre>
<pre><code>##     gp          y
## 1 gp75  1.1622536
## 2 gp26 -1.0026492
## 3 gp26 -0.4317629
## 4 gp43  0.3547021
## 5 gp19 -0.5028478
## 6 gp41  0.1239806</code></pre>
<p><img border="0" alt="Unnamed chunk 2 1" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/09/unnamed-chunk-2-1-1.png?w=660" data-recalc-dims="1"></p>
<p>As the graph shows, some groups were heavily sampled, but most groups have only a handful of samples in the data set. Since this is synthetic data, we know the true population means (shown in red in the graph below), and we can compare them to the observed means <span>\(\bar{y}_i\)</span> of each group <span>\(i\)</span> (shown in black, with standard errors. The gray points are the actual observations). We’ve sorted the groups by the number of observations.</p>
<p><img border="0" alt="Unnamed chunk 3 1" src="https://i1.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/09/unnamed-chunk-3-1.png?resize=599%2C428" data-recalc-dims="1"></p>
<p>For groups with many observations, the observed group mean is near the true mean. For groups with few observations, the estimates are uncertain, and the observed group mean can be far from the true population mean.</p>
<p>Can we get better estimates of the conditional mean for groups with only a few observations?</p>
</div>
<div>
<h2>Partial Pooling</h2>
<p>If the data is generated by the process described above, and if we knew <span>\(\sigma_w\)</span> and <span>\(\sigma_b\)</span>, then a good estimate <span>\(\hat{y}_i\)</span> for the mean of group <span>\(i\)</span> is the weighted average of the grand mean over all the data, <span>\(\bar{y}\)</span>, and the observed mean of all the observations in group <span>\(i\)</span>, <span>\(\bar{y}_i\)</span>.</p>
<p><span>\[<br>
\large<br>
\hat{y_i} \approx \frac{\frac{n_i} {\sigma_w^2} \cdot \bar{y}_i + \frac{1}{\sigma_b^2} \cdot \bar{y}}<br>
{\frac{n_i} {\sigma_w^2} + \frac{1}{\sigma_b^2}}<br>
\]</span></p>
<p>where <span>\(n_i\)</span> is the number of observations for group <span>\(i\)</span>. In other words, for groups where you have a lot of observations, use an estimate close to the observed group mean. For groups where you have only a few observations, fall back to an estimate close to the grand mean.</p>
<p>Gelman and Hill call the grand mean the <em>complete-pooling</em> estimate, because the data from all the groups is pooled to create the estimate (which is the same for all <span>\(i\)</span>). The “raw” observed means are the <em>no-pooling</em> estimate, because no pooling occurs; only observations from group <span>\(i\)</span> contribute to <span>\(\hat{y_i}\)</span>. The weighted sum of the complete-pooling and the no-pooling estimate is hence the <em>partial-pooling</em> estimate.</p>
<p>Of course, in practice we don’t know <span>\(\sigma_w\)</span> and <span>\(\sigma_b\)</span>. The <code>lmer</code> function essentially solves for the restricted maximum likelihood (REML) estimates of the appropriate parameters in order to estimate <span>\(\hat{y_i}\)</span>. You can express multilevel models in <code>lme4</code> using the notation <code>| gp</code> in formulas to designate that <code>gp</code> is the grouping variable that you want conditional estimates for. The model that we are interested in is the simplest: outcome as a function of the grouping variable, with no other predictors.</p>
<pre><code>poolmod = lmer(y ~ (1 | gp), data=df)</code></pre>
<p>See section 2.2 of <a rel="nofollow" href="https://cran.r-project.org/web/packages/lme4/vignettes/lmer.pdf" target="_blank">this <code>lmer</code> vignette</a> for more discussion on writing formulas for models with additional predictors. Printing <code>poolmod</code> displays the REML estimates of the grand mean (The intercept), <span>\(\sigma_b\)</span> (the standard deviation of <span>\(gp\)</span>) and <span>\(\sigma_w\)</span> (the residual).</p>
<pre><code>poolmod</code></pre>
<pre><code>## Linear mixed model fit by REML ['lmerMod']
## Formula: y ~ (1 | gp)
##    Data: df
## REML criterion at convergence: 2282.939
## Random effects:
##  Groups   Name        Std.Dev.
##  gp       (Intercept) 0.5348  
##  Residual             0.7063  
## Number of obs: 1002, groups:  gp, 85
## Fixed Effects:
## (Intercept)  
##    -0.02761</code></pre>
<p>To pull these values out explicitly:</p>
<pre><code># the estimated grand mean 
(grandmean_est= fixef(poolmod))</code></pre>
<pre><code>## (Intercept) 
## -0.02760728</code></pre>
<pre><code># get the estimated between-group standard deviation
(sigma_b = as.data.frame(VarCorr(poolmod)) %>% 
  filter(grp==&quot;gp&quot;) %>% 
  pull(sdcor))</code></pre>
<pre><code>## [1] 0.5348401</code></pre>
<pre><code># get the estimated within-group standard deviation
(sigma_w = as.data.frame(VarCorr(poolmod)) %>% 
  filter(grp==&quot;Residual&quot;) %>% 
  pull(sdcor))</code></pre>
<pre><code>## [1] 0.7063342</code></pre>
<p><code>predict(poolmod)</code> will return the partial pooling estimates of the group means. Below, we compare the partial pooling estimates to the raw group mean expectations. The gray lines represent the true group means, the dark blue horizontal line is the observed grand mean, and the black dots are the estimates. We have again sorted the groups by number of observations, and laid them out (with a slight jitter) on a log10 scale.</p>
<p><img border="0" alt="Unnamed chunk 6 1" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/09/unnamed-chunk-6-1.png?w=660" data-recalc-dims="1"></p>
<p>For groups with only a few observations, the partial pooling “shrinks” the estimates towards the grand mean<sup>1</sup>, which often results in a better estimate of the true conditional population means. We can see the relationship between shrinkage (the raw estimate minus the partial pooling estimate) and the groups, ordered by sample size.</p>
<p><img border="0" alt="Unnamed chunk 7 1" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/09/unnamed-chunk-7-1.png?w=660" data-recalc-dims="1"></p>
<p>For this data set, the partial pooling estimates are on average closer to the true means than the raw estimates; we can see this by comparing the root mean squared errors of the two estimates.</p>
<table>
<thead>
<tr>
<th>estimate_type</th>
<th>rmse</th>
</tr>
</thead>
<tbody>
<tr>
<td>raw</td>
<td>0.3261321</td>
</tr>
<tr>
<td>partial pooling</td>
<td>0.2484646</td>
</tr>
</tbody>
</table>
<p>(1): To be precise, partial pooling shrinks estimates toward the <em>estimated</em> grand mean -0.0276, not to the <em>observed</em> grand mean 0.155.</p>
<div>
<h3>The Discrete Case</h3>
<p>For discrete (binary) outcomes or classification, use the function <code>glmer()</code> to fit multilevel logistic regression models. Suppose we want to predict <span>\(\mbox{P}(y > 0 \,|\, gp)\)</span>, the conditional probability that the outcome <span>\(y\)</span> is positive, as a function of <span>\(gp\)</span>.</p>
<pre><code>df$ispos = df$y > 0

# fit a logistic regression model
mod_glm = glm(ispos ~ gp, data=df, family=binomial)</code></pre>
<p>Again, the conditional probability estimates will be highly uncertain for groups with only a few observations. We can fit a multilevel model with <code>glmer</code> and compare the distributions of the resulting predictions in link space.</p>
<pre><code>mod_glmer = glmer(ispos ~ (1|gp), data=df, family=binomial)</code></pre>
<p><img border="0" alt="Unnamed chunk 11 1" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/09/unnamed-chunk-11-1.png?w=660" data-recalc-dims="1"></p>
<p>Note that the distribution of predictions for the standard logistic regression model is trimodal, and that for some groups, the logistic regression model predicts probabilities very close to 0 or to 1. In most cases, these predictions will correspond to groups with few observations, and are unlikely to be good estimates of the true conditional probability. The partial pooling model avoids making <em>unjustified</em> predictions near 0 or 1, instead “shrinking” the estimates to the estimated global probability that <span>\(y > 0\)</span>, which in this case is about 0.49.</p>
<p>We can see how the number of observations corresponds to the shrinkage (the difference between the logistic regression and the partial pooling estimates) in the graph below (this time in probability space). Points in orange correspond to groups where the logistic regression estimated probabilities of 0 or 1 (the two outer lobes of the response distribution). Multimodal densities are often symptoms of model flaws such as omitted variables or un-modeled mixtures, so it is exciting to see the partially pooled estimator avoid the “wings” seen in the simpler logistic regression estimator.</p>
<p><img border="0" alt="Unnamed chunk 12 1" src="https://i0.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/09/unnamed-chunk-12-1.png?w=660" data-recalc-dims="1"></p>
</div>
</div>
<div>
<h2>Partial Pooling Degrades Gracefully</h2>
<p>When there is enough data for each population to get a good estimate of the population means – for example, when the distribution of groups is fairly uniform, or at least not too skewed – the partial pooling estimates will converge to the the raw (no-pooling) estimates. When the variation between population means is very low, the partial pooling estimates will converge to the complete pooling estimate (the grand mean).</p>
<p>When there are only a few levels (Gelman and Hill suggest less than about five), there will generally not be enough information to make a good estimate of <span>\(\sigma_b\)</span>, so the partial pooled estimates likely won’t be much better than the raw estimates.</p>
<p>So partial pooling will be of the most potential value when the number of groups is large, and there are many rare levels. With respect to <code>vtreat</code>, this is exactly the situation when level coding is most useful!</p>
<p>Multilevel modeling assumes the data was generated from the mixture process above: each population is normally distributed, with the same standard deviation, and the population means are also normally distributed. Obviously, this may not be the case, but as Gelman and Hill argue, the additional inductive bias can be useful for those populations where you have little information.</p>
<p>Thanks to <a rel="nofollow" href="https://www.linkedin.com/in/geoffrey-simmons-bb675242/" target="_blank">Geoffrey Simmons</a>, Principal Data Scientist at Echo Global Logistics, for suggesting partial pooling based level coding for <code>vtreat</code>, introducing us to the references, and reviewing our articles.</p>
</div>
<div>
<h2>References</h2>
<p>Gelman, Andrew and Jennifer Hill. <em>Data Analysis Using Regression and Multilevel/Hierarchical Models</em>. Cambridge University Press, 2007.</p>
</div>
</div>
<p></p>
<p><br>
</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2017/09/partial-pooling-for-lower-variance-variable-encoding/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/AsUWcz4evl4" height="1"></a></div>