<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/custom-level-coding-in-vtreat/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2017/09/custom-level-coding-in-vtreat/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>One of the services that the <a rel="nofollow" href="https://www.r-project.org/" target="_blank"><code>R</code></a> package <a rel="nofollow" href="https://winvector.github.io/vtreat/" target="_blank"><code>vtreat</code></a> provides is <em>level coding</em> (what we sometimes call <em>impact coding</em>): converting the levels of a categorical variable to a meaningful and concise single numeric variable, rather than coding them as indicator variables (AKA &quot;one-hot encoding&quot;). Level coding can be computationally and statistically preferable to one-hot encoding for variables that have an extremely large number of possible levels.</p>
<figure align="center">
<img border="0" alt="Speed" src="https://i0.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/09/speed.gif?w=660" data-recalc-dims="1"><br>
<figcaption><small><br>
Level coding is like measurement: it summarizes categories of individuals into useful numbers. Source: <a rel="nofollow" href="https://pubs.usgs.gov/gip/dinosaurs/speed.html" target="_blank">USGS</a><br>
</small></figcaption>
</figure>
<p>
<p>By default, <code>vtreat</code> level codes to the difference between the conditional means and the grand mean (<code>catN</code> variables) when the outcome is numeric, and to the difference between the conditional log-likelihood and global log-likelihood of the target class (<code>catB</code> variables) when the outcome is categorical. These aren’t the only possible level codings. For example, the <code>ranger</code> package can encode categorical variables as ordinals, sorted by the conditional expectations/means. While this is not a completely faithful encoding for all possible models (it is not completely faithful for linear or logistic regression, for example), it is often invertible for tree-based methods, and has the advantage of keeping the original levels distinct, which impact coding may not. That is, two levels with the same conditional expectation would be conflated by <code>vtreat</code>‘s coding. This often isn’t a problem — but sometimes, it may be.</p>
<p>So the data scientist may want to use a level coding different from what <code>vtreat</code> defaults to. In this article, we will demonstrate how to implement custom level encoders in <code>vtreat</code>. We assume you are familiar with the basics of <code>vtreat</code>: the types of derived variables, how to create and apply a treatment plan, etc.</p>
<p><span></span></p>
<p>For our example, we will implement level coders based on partial pooling, or hierarchical/multilevel models (Gelman and Hill, 2007). We’ll leave the details of how partial pooling works to a subsequent article; for now, just think of it as a score that shrinks the estimate of the conditional mean to be closer to the unconditioned mean, and hence possibly closer to the unknown true values, when there are too few measurements to make an accurate estimate.</p>
<p>We’ll implement our partial pooling encoders using the <code>lmer()</code> (multilevel linear regression) and <code>glmer()</code> (multilevel generalized linear regression) functions from the <code>lme4</code> package. For our example data, we’ll use radon levels by county for the state of Minnesota (Gelman and Hill, 2007. You can find the original data <a rel="nofollow" href="http://www.stat.columbia.edu/~gelman/arm/software/" target="_blank">here</a>).</p>
<h2>The Data: Radon levels in Minnesota</h2>
<div>
<pre><code><span>library</span>(<span>&quot;vtreat&quot;</span>)
<span>library</span>(<span>&quot;lme4&quot;</span>)
<span>library</span>(<span>&quot;dplyr&quot;</span>)
<span>library</span>(<span>&quot;tidyr&quot;</span>)
<span>library</span>(<span>&quot;ggplot2&quot;</span>)

<span># example data</span>

srrs =<span> </span><span>read.table</span>(<span>&quot;srrs2.dat&quot;</span>, <span>header=</span><span>TRUE</span>, <span>sep=</span><span>&quot;,&quot;</span>, <span>stringsAsFactor=</span><span>FALSE</span>)

<span># target: log of radon activity (activity)</span>
<span># grouping variable: county</span>
radonMN =<span> </span><span>filter</span>(srrs, state==<span>&quot;MN&quot;</span>) %>%
<span> </span><span>select</span>(<span>&quot;county&quot;</span>, <span>&quot;activity&quot;</span>) %>%
<span> </span><span>filter</span>(activity ><span> </span><span>0</span>) %>%<span> </span>
<span> </span><span>mutate</span>(<span>activity =</span> <span>log</span>(activity),
         <span>county =</span> base::<span>trimws</span>(county)) %>%
<span> </span><span>mutate</span>(<span>critical =</span> activity><span>1.5</span>)

<span>str</span>(radonMN)</code></pre>
</div>
<pre><code>## 'data.frame':    916 obs. of  3 variables:
##  $ county  : chr  &quot;AITKIN&quot; &quot;AITKIN&quot; &quot;AITKIN&quot; &quot;AITKIN&quot; ...
##  $ activity: num  0.788 0.788 1.065 0 1.131 ...
##  $ critical: logi  FALSE FALSE FALSE FALSE FALSE FALSE ...</code></pre>
<p>For this example we have three columns of interest:</p>
<ul>
<li><code>county</code>: 85 possible values</li>
<li><code>activity</code>: the log of the radon reading (numerical outcome)</li>
<li><code>critical</code>: <code>TRUE</code> when activity > 1.5 (categorical outcome)</li>
</ul>
<p>The goal is to level code <code>county</code> for either the regression problem (predict the log radon reading) or the categorization problem (predict whether the radon level is &quot;critical&quot;).</p>
<p><img border="0" alt="Unnamed chunk 1 1" src="https://i0.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/09/unnamed-chunk-1-1.png?w=660" data-recalc-dims="1"></p>
<p>As the graph shows, the conditional mean of log radon activity by county ranges from nearly zero to about 3, and the conditional expectation of a critical reading ranges from zero to one. On the other hand, the number of readings per county is quite low for many counties — only one or two — though some counties have a large number of readings. That means some of the conditional expectations are quite uncertain.</p>
<h2>Implementing Level Coders for Partial Pooling</h2>
<p>Let’s implement level coders that use partial pooling to compute the level score.</p>
<p><strong>Regression</strong></p>
<p>For regression problems, the custom coder should be a function that takes as input:</p>
<ul>
<li><code>v</code>: a string with the name of the categorical variable</li>
<li><code>vcol</code>: the actual categorical column (assumed character)</li>
<li><code>y</code>: the numerical outcome column</li>
<li><code>weights</code>: a column of row weights</li>
</ul>
<p>The function should return a column of scores (the level codings). For our example, the function builds a <code>lmer</code> model to predict <code>y</code> as a function of <code>vcol</code>, then returns the predictions on the training data.</p>
<div>
<pre><code><span># @param v character variable name</span>
<span># @param vcol character, independent or input variable</span>
<span># @param y numeric, dependent or outcome variable to predict</span>
<span># @param weights row/example weights</span>
<span># @return scored training data column</span>
ppCoderN &lt;-<span> </span>function(v, vcol, 
                     y, 
                     weights) {
  <span># regression case y ~ vcol</span>
  d &lt;-<span> </span><span>data.frame</span>(<span>x =</span> vcol,
                  <span>y =</span> y,
                  <span>stringsAsFactors =</span> <span>FALSE</span>)
  m &lt;-<span> </span><span>lmer</span>(y ~<span> </span>(<span>1</span> |<span> </span>x), <span>data=</span>d, <span>weights=</span>weights)
  <span>predict</span>(m, <span>newdata=</span>d)
}</code></pre>
</div>
<p><strong>Categorization</strong></p>
<p>For categorization problems, the function should assume that <code>y</code> is a logical column, where <code>TRUE</code> is assumed to be the target outcome. This is because <code>vtreat</code> converts the outcome column to a logical while creating the treatment plan.</p>
<div>
<pre><code><span># @param v character variable name</span>
<span># @param vcol character, independent or input variable</span>
<span># @param y logical, dependent or outcome variable to predict</span>
<span># @param weights row/example weights</span>
<span># @return scored training data column</span>
ppCoderC &lt;-<span> </span>function(v, vcol, 
                     y, 
                     weights) {
  <span># classification case y ~ vcol</span>
  d &lt;-<span> </span><span>data.frame</span>(<span>x =</span> vcol,
                  <span>y =</span> y,
                  <span>stringsAsFactors =</span> <span>FALSE</span>)
  m =<span> </span><span>glmer</span>(y ~<span> </span>(<span>1</span> |<span> </span>x), <span>data=</span>d, <span>weights=</span>weights, <span>family=</span>binomial)
  <span>predict</span>(m, <span>newdata=</span>d, <span>type=</span><span>'link'</span>)
}</code></pre>
</div>
<p>You can then pass the functions in as a named list into either <code>designTreatmentsX</code> or <code>mkCrossFrameXExperiment</code> to build the treatment plan. The format of the key is <code>[n|c].levelName[.option]*</code>.</p>
<p>The prefacing picks the model type: numeric or regression starts with ‘n.’ and the categorical encoder starts with ‘c.’. Currently, the only supported option is ‘center,’ which directs <code>vtreat</code> to center the codes with respect to the estimated grand mean. The<code>catN</code> and <code>catB</code> level codings are centered in this way.</p>
<p>Our example coders can be passed in as shown below.</p>
<div>
<pre><code>customCoders =<span> </span><span>list</span>(<span>'n.poolN.center'</span> =<span> </span>ppCoderN, 
                    <span>'c.poolC.center'</span> =<span> </span>ppCoderC)</code></pre>
</div>
<h2>Using the Custom Coders</h2>
<p>Let’s build a treatment plan for the regression problem.</p>
<div>
<pre><code><span># I only want to create the cleaned numeric variables, the isBAD variables,</span>
<span># and the level codings (not the indicator variables or catP, etc.)</span>
vartypes_I_want =<span> </span><span>c</span>(<span>'clean'</span>, <span>'isBAD'</span>, <span>'catN'</span>, <span>'poolN'</span>)

treatplanN =<span> </span><span>designTreatmentsN</span>(radonMN, 
                               <span>varlist =</span> <span>c</span>(<span>'county'</span>),
                               <span>outcomename =</span> <span>'activity'</span>,
                              <span>codeRestriction =</span> vartypes_I_want,
                              <span>customCoders =</span> customCoders, 
                              <span>verbose=</span><span>FALSE</span>)

scoreFrame =<span> </span>treatplanN$scoreFrame
scoreFrame %>%<span> </span><span>select</span>(varName, sig, origName, code)</code></pre>
</div>
<pre><code>##        varName          sig origName  code
## 1 county_poolN 1.343072e-16   county poolN
## 2  county_catN 2.050811e-16   county  catN</code></pre>
<p>Note that the treatment plan returned both the <code>catN</code> variable (default level encoding) and the pooled level encoding (<code>poolN</code>). You can restrict to just using one coding or the other using the <code>codeRestriction</code> argument either during treatment plan creation, or in <code>prepare()</code>.</p>
<p>Let’s compare the two level encodings.</p>
<div>
<pre><code><span># create a frame with one row for every county,</span>
measframe =<span> </span><span>data.frame</span>(<span>county =</span> <span>unique</span>(radonMN$county),
                       <span>stringsAsFactors=</span><span>FALSE</span>)

outframe =<span> </span><span>prepare</span>(treatplanN, measframe)

<span># If we wanted only the new pooled level coding,</span>
<span># (plus any numeric/isBAD variables), we would</span>
<span># use a codeRestriction:</span>
<span>#</span>
<span># outframe = prepare(treatplanN, </span>
<span># measframe,</span>
<span># codeRestriction = c('clean', 'isBAD', 'poolN'))</span>


<span>gather</span>(outframe, <span>key=</span>scoreType, <span>value=</span>score, 
       county_poolN, county_catN) %>%
<span> </span><span>ggplot</span>(<span>aes</span>(<span>x=</span>score)) +<span> </span>
<span> </span><span>geom_density</span>(<span>adjust=</span><span>0.5</span>) +<span> </span><span>geom_rug</span>(<span>sides=</span><span>&quot;b&quot;</span>) +<span> </span>
<span> </span><span>facet_wrap</span>(~scoreType, <span>ncol=</span><span>1</span>, <span>scale=</span><span>&quot;free_y&quot;</span>) +<span> </span>
<span> </span><span>ggtitle</span>(<span>&quot;Distribution of scores&quot;</span>)</code></pre>
</div>
<p><img border="0" alt="Nex 1" src="https://i0.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/09/nex-1.png?w=660" data-recalc-dims="1"></p>
<p>Notice that the <code>poolN</code> scores are &quot;tucked in&quot; compared to the <code>catN</code> encoding. In a later article, we’ll show that the counties with the most tucking in (or <em>shrinkage</em>) tend to be those with fewer measurements.</p>
<p>We can also code for the categorical problem.</p>
<div>
<pre><code><span># For categorical problems, coding is catB</span>
vartypes_I_want =<span> </span><span>c</span>(<span>'clean'</span>, <span>'isBAD'</span>, <span>'catB'</span>, <span>'poolC'</span>)

treatplanC =<span> </span><span>designTreatmentsC</span>(radonMN, 
                               <span>varlist =</span> <span>c</span>(<span>'county'</span>),
                               <span>outcomename =</span> <span>'critical'</span>,
                               <span>outcometarget=</span> <span>TRUE</span>,
                               <span>codeRestriction =</span> vartypes_I_want,
                               <span>customCoders =</span> customCoders, 
                               <span>verbose=</span><span>FALSE</span>)

outframe =<span> </span><span>prepare</span>(treatplanC, measframe)

<span>gather</span>(outframe, <span>key=</span>scoreType, <span>value=</span>linkscore, 
       county_poolC, county_catB) %>%
<span> </span><span>ggplot</span>(<span>aes</span>(<span>x=</span>linkscore)) +<span> </span>
<span> </span><span>geom_density</span>(<span>adjust=</span><span>0.5</span>) +<span> </span><span>geom_rug</span>(<span>sides=</span><span>&quot;b&quot;</span>) +<span> </span>
<span> </span><span>facet_wrap</span>(~scoreType, <span>ncol=</span><span>1</span>, <span>scale=</span><span>&quot;free_y&quot;</span>) +<span> </span>
<span> </span><span>ggtitle</span>(<span>&quot;Distribution of link scores&quot;</span>)</code></pre>
</div>
<p><img border="0" alt="Unnamed chunk 2 1" src="https://i0.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/09/unnamed-chunk-2-1.png?w=660" data-recalc-dims="1"></p>
<p>Notice that the poolC link scores are even more tucked in compared to the catB link scores, and that the catB scores are multimodal. The smaller link scores mean that the pooled model avoids estimates of conditional expectation close to either zero or one, because, again, these estimates come from counties with few readings. Multimodal summaries can be evidence of modeling flaws, including omitted variables and un-modeled mixing of different example classes. Hence, we do not want our inference procedure to suggest such structure until there is a lot of evidence for it. And, as is common in machine learning, there are advantages to lower-variance estimators when they do not cost much in terms of bias.</p>
<h2>Other Considerations</h2>
<p>For this example, we used the <code>lme4</code> package to create custom level codings. Once calculated, <code>vtreat</code> stores the coding as a lookup table in the treatment plan. <em>This means <code>lme4</code> is not needed to prepare new data.</em> In general, using a treatment plan is not dependent on any special packages that might have been used to create it, so it can be shared with other users with no extra dependencies.</p>
<p>When using <code>mkCrossFrameXExperiment</code>, note that the resulting cross frame will have a slightly different distribution of scores than what the treatment plan produces. This is true even for <code>catB</code> and <code>catN</code> variables. This is because the treatment plan is built using all the data, while the cross frame is built using n-fold cross validation on the data. See <a rel="nofollow" href="https://winvector.github.io/vtreat/articles/vtreatCrossFrames.html" target="_blank">the cross frame vignette</a> for more details.</p>
<p>Thanks to <a rel="nofollow" href="https://www.linkedin.com/in/geoffrey-simmons-bb675242/" target="_blank">Geoffrey Simmons</a>, Principal Data Scientist at Echo Global Logistics, for suggesting partial pooling based level coding (and testing it for us!), introducing us to the references, and reviewing our articles.</p>
<p>In a follow-up article, we will go into partial pooling in more detail, and motivate why you might sometimes prefer it to <code>vtreat</code>‘s default coding.</p>
<h2>References</h2>
<p>Gelman, Andrew and Jennifer Hill. <em>Data Analysis Using Regression and Multilevel/Hierarchical Models</em>. Cambridge University Press, 2007.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2017/09/custom-level-coding-in-vtreat/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/9nkhBMqjLyc" height="1"></a></div>