<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/exploring-embeddings-for-categorical-variables-with-keras/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://flovv.github.io/Embeddings_with_keras/"> Florian Teschner</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In order to stay up to date, I try to follow Jeremy Howard on a regular basis. In one of his recent videos, he shows how to use embeddings for categorical variables (e.g. weekdays).</p>
<p>First off; what are embeddings?<br>
An embedding is a mapping of a categorical vector in a continuous n-dimensional space. The idea is to represent a categorical representation with n-continuous variables. To make it more concrete, let’s say you want to model the effect of day of the week on an outcome. Usually you would try to one-hot encode the variable, which means that you create 6 variables (each for one day of a week minus 1) and set the variable 1 or 0 depending on the value. You end up having a 6-dimensional space to represent a weekday.<br>
So, what is the advantage of mapping the variables in an continuous space?<br>
In a nutshell; with embeddings you can reduce the dimensionality of your feature space which should reduce overfitting in prediction problems.</p>
<p>In order to test the idea on a play example, I downloaded the nyc citi bike count data from Kaggle. It contains daily bicycle counts for major bridges in NYC.</p>
<figure>
<pre><code data-lang="r"><span>#https://www.kaggle.com/new-york-city/nyc-east-river-bicycle-crossings</span><span><br data-jekyll-commonmark-ghpages=""></span><span>df</span><span> </span><span>&lt;-</span><span> </span><span>read.csv</span><span>(</span><span>&quot;data/nyc-east-river-bicycle-counts.csv&quot;</span><span>)</span><span><br data-jekyll-commonmark-ghpages=""> <br data-jekyll-commonmark-ghpages=""></span><span>df</span><span>$</span><span>date</span><span> </span><span>&lt;-</span><span> </span><span>as.Date</span><span>(</span><span>df</span><span>$</span><span>Date</span><span>)</span><span><br data-jekyll-commonmark-ghpages=""></span><span>df</span><span>$</span><span>weekday</span><span> </span><span>&lt;-</span><span> </span><span>lubridate</span><span>::</span><span>wday</span><span>(</span><span>df</span><span>$</span><span>date</span><span>)</span><span><br data-jekyll-commonmark-ghpages=""></span><span>df</span><span>$</span><span>users</span><span> </span><span>&lt;-</span><span> </span><span>df</span><span>$</span><span>Brooklyn.Bridge</span><span><br data-jekyll-commonmark-ghpages=""> <br data-jekyll-commonmark-ghpages=""></span><span>df</span><span> </span><span>&lt;-</span><span> </span><span>df</span><span>[</span><span>df</span><span>$</span><span>users</span><span>></span><span>0</span><span>,]</span><span><br data-jekyll-commonmark-ghpages=""></span><span>df</span><span> </span><span>&lt;-</span><span> </span><span>df</span><span>[</span><span>!</span><span>is.na</span><span>(</span><span>df</span><span>$</span><span>users</span><span>),]</span><span><br data-jekyll-commonmark-ghpages=""></span><span>df</span><span> </span><span>&lt;-</span><span> </span><span>df</span><span>[</span><span>!</span><span>is.na</span><span>(</span><span>df</span><span>$</span><span>weekday</span><span>),]</span><span><br data-jekyll-commonmark-ghpages=""> <br data-jekyll-commonmark-ghpages=""></span><span>df</span><span>$</span><span>ScaledUsers</span><span> </span><span>&lt;-</span><span> </span><span>scale</span><span>(</span><span>df</span><span>$</span><span>users</span><span>)</span></code></pre>
</figure>
<p>Next, we set up a sequentual model with keras. The first layer is the embedding layer with the size of 7 weekdays plus 1 (for the unknowns). The embedding-size defines the dimensionality in which we map the categorical variables.<br>
Jeremy Howard provides the following rule of thumb; embedding size = min(50, number of categories/2).</p>
<figure>
<pre><code data-lang="text">require(keras)<br data-jekyll-commonmark-ghpages="">embedding_size &lt;- 3<br data-jekyll-commonmark-ghpages=""><br data-jekyll-commonmark-ghpages="">model &lt;- keras_model_sequential()<br data-jekyll-commonmark-ghpages=""><br data-jekyll-commonmark-ghpages="">model %>% layer_embedding(input_dim = 7+1, output_dim = embedding_size, input_length = 1, name=&quot;embedding&quot;) %>%<br data-jekyll-commonmark-ghpages="">  layer_flatten()  %>%  <br data-jekyll-commonmark-ghpages="">  layer_dense(units=40, activation = &quot;relu&quot;) %>%  <br data-jekyll-commonmark-ghpages="">  layer_dense(units=10, activation = &quot;relu&quot;) %>%  <br data-jekyll-commonmark-ghpages="">  layer_dense(units=1)<br data-jekyll-commonmark-ghpages=""><br data-jekyll-commonmark-ghpages="">model %>% compile(loss = &quot;mse&quot;, optimizer = &quot;sgd&quot;, metric=&quot;accuracy&quot;)<br data-jekyll-commonmark-ghpages=""><br data-jekyll-commonmark-ghpages="">hist &lt;- model %>% fit(x = as.matrix(df$weekday), y= as.matrix(df$ScaledUsers), epochs = 50, batch_size = 2)</code></pre>
</figure>
<p>It is important to have a rather small batch size and to scale the count data.<br>
After training the model, we can extract individual layers. We named the first layer “embedding”. The weights of the embedding layer define where in the 3-dimensional feature space the network has placed the variables. We can show that by plotting the points in a scatterplot.</p>
<figure>
<pre><code data-lang="r"><span>layer</span><span> </span><span>&lt;-</span><span> </span><span>get_layer</span><span>(</span><span>model</span><span>,</span><span> </span><span>&quot;embedding&quot;</span><span>)</span><span><br data-jekyll-commonmark-ghpages=""> <br data-jekyll-commonmark-ghpages=""></span><span>embeddings</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>layer</span><span>$</span><span>get_weights</span><span>()[[</span><span>1</span><span>]])</span><span><br data-jekyll-commonmark-ghpages=""></span><span>embeddings</span><span>$</span><span>name</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;none&quot;</span><span>,</span><span> </span><span>levels</span><span>(</span><span>wday</span><span>(</span><span>df</span><span>$</span><span>date</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>T</span><span>))</span><span> </span><span>)</span><span><br data-jekyll-commonmark-ghpages=""> <br data-jekyll-commonmark-ghpages=""></span><span>ggplot</span><span>(</span><span>embeddings</span><span>,</span><span> </span><span>aes</span><span>(</span><span>X</span><span>1</span><span>,</span><span> </span><span>X</span><span>2</span><span>,</span><span> </span><span>color</span><span>=</span><span>name</span><span>))</span><span>+</span><span> </span><span>geom_point</span><span>()</span><span> </span><span>+</span><span>geom_text</span><span>(</span><span>aes</span><span>(</span><span>label</span><span>=</span><span>name</span><span>),</span><span>hjust</span><span>=</span><span>0</span><span>,</span><span> </span><span>vjust</span><span>=</span><span>0</span><span>)</span><span> </span><span>+</span><span> </span><span>theme_bw</span><span>()</span><span> </span><span>+</span><span> </span><span>xlab</span><span>(</span><span>&quot;Embedding Dimension 1&quot;</span><span>)</span><span> </span><span>+</span><span>ylab</span><span>(</span><span>&quot;Embedding Dimension 2&quot;</span><span>)</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-46" src="https://i1.wp.com/flovv.github.io/figures/post28/post28unnamed-chunk-46-1.png?w=456" data-recalc-dims="1"></p>
<p>The great thing about the embedding layer weights are, that they act as a lookup table. Merging the variables back to our dataset we can use the dimensions as input (X1, X2, X3) for a simple linear regression replacing the categorical representation of the day of the week variable.</p>
<figure>
<pre><code data-lang="r"><span>df</span><span>$</span><span>weekDayF</span><span> </span><span>&lt;-</span><span> </span><span>wday</span><span>(</span><span>df</span><span>$</span><span>date</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>T</span><span>)</span><span><br data-jekyll-commonmark-ghpages=""></span><span>embeddings</span><span>$</span><span>lookup</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;none&quot;</span><span>,</span><span> </span><span>levels</span><span>(</span><span>df</span><span>$</span><span>weekDayF</span><span>))</span><span><br data-jekyll-commonmark-ghpages=""></span><span>dff</span><span> </span><span>&lt;-</span><span> </span><span>merge</span><span>(</span><span>df</span><span>,</span><span> </span><span>embeddings</span><span>,</span><span> </span><span>by.x</span><span>=</span><span>&quot;weekDayF&quot;</span><span>,</span><span> </span><span>&quot;lookup&quot;</span><span>)</span><span><br data-jekyll-commonmark-ghpages=""> <br data-jekyll-commonmark-ghpages=""></span><span>## we trained the embeddings on the Brooklyn.Bridge variable but test it on another ...</span><span><br data-jekyll-commonmark-ghpages=""></span><span>dff</span><span>$</span><span>users</span><span> </span><span>&lt;-</span><span> </span><span>dff</span><span>$</span><span>Manhattan.Bridge</span><span><br data-jekyll-commonmark-ghpages=""> <br data-jekyll-commonmark-ghpages=""></span><span>testRun</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>x</span><span>){</span><span><br data-jekyll-commonmark-ghpages=""> </span><span>sample</span><span> </span><span>&lt;-</span><span> </span><span>caret</span><span>::</span><span>createDataPartition</span><span>(</span><span>dff</span><span>$</span><span>weekDayF</span><span>,</span><span> </span><span>list</span><span>=</span><span>FALSE</span><span>,</span><span> </span><span>p</span><span> </span><span>=</span><span> </span><span>0.8</span><span>)</span><span><br data-jekyll-commonmark-ghpages=""> </span><span>train</span><span> </span><span>&lt;-</span><span> </span><span>dff</span><span>[</span><span>sample</span><span>,]</span><span><br data-jekyll-commonmark-ghpages=""> </span><span>test</span><span> </span><span>&lt;-</span><span> </span><span>dff</span><span>[</span><span>-</span><span>sample</span><span>,]</span><span><br data-jekyll-commonmark-ghpages=""> <br data-jekyll-commonmark-ghpages=""> </span><span>fit1</span><span> </span><span>&lt;-</span><span> </span><span>lm</span><span>(</span><span>users</span><span> </span><span>~</span><span> </span><span>X</span><span>1</span><span> </span><span>+</span><span> </span><span>X</span><span>2</span><span> </span><span>+</span><span> </span><span>X</span><span>3</span><span>,</span><span> </span><span>data</span><span>=</span><span>train</span><span>)</span><span><br data-jekyll-commonmark-ghpages=""> </span><span>fit2</span><span> </span><span>&lt;-</span><span> </span><span>lm</span><span>(</span><span>users</span><span> </span><span>~</span><span> </span><span>weekDayF</span><span> </span><span>,</span><span> </span><span>data</span><span>=</span><span>train</span><span>)</span><span> </span><span># 6 input variables as lm automatically one-hot encodes categorical variables.</span><span><br data-jekyll-commonmark-ghpages=""> <br data-jekyll-commonmark-ghpages=""> </span><span>data.frame</span><span>(</span><span>run</span><span>=</span><span>x</span><span>,</span><span><br data-jekyll-commonmark-ghpages=""> </span><span>Embedding</span><span>=</span><span>sqrt</span><span>(</span><span>mean</span><span>((</span><span>predict</span><span>(</span><span>fit1</span><span>,</span><span> </span><span>test</span><span>)</span><span> </span><span>-</span><span> </span><span>test</span><span>$</span><span>users</span><span>)</span><span>^</span><span>2</span><span>)),</span><span> <br data-jekyll-commonmark-ghpages=""> </span><span>Categorical</span><span>=</span><span>sqrt</span><span>(</span><span>mean</span><span>((</span><span>predict</span><span>(</span><span>fit2</span><span>,</span><span> </span><span>test</span><span>)</span><span> </span><span>-</span><span> </span><span>test</span><span>$</span><span>users</span><span>)</span><span>^</span><span>2</span><span>)</span><span> </span><span>))</span><span><br data-jekyll-commonmark-ghpages=""></span><span>}</span><span><br data-jekyll-commonmark-ghpages=""> <br data-jekyll-commonmark-ghpages=""></span><span>test</span><span> </span><span>&lt;-</span><span> </span><span>plyr</span><span>::</span><span>ldply</span><span>(</span><span>1</span><span>:</span><span>100</span><span>,</span><span> </span><span>function</span><span>(</span><span>x</span><span>){</span><span>testRun</span><span>(</span><span>x</span><span>)})</span><span><br data-jekyll-commonmark-ghpages=""></span><span>mm</span><span> </span><span>&lt;-</span><span> </span><span>data.table</span><span>::</span><span>melt</span><span>(</span><span>test</span><span>,</span><span> </span><span>id.vars</span><span>=</span><span>&quot;run&quot;</span><span>)</span><span><br data-jekyll-commonmark-ghpages=""></span><span>dd</span><span> </span><span>&lt;-</span><span> </span><span>plyr</span><span>::</span><span>ddply</span><span>(</span><span>mm</span><span>,</span><span>.</span><span>(</span><span>variable</span><span>),</span><span> </span><span>summarise</span><span>,</span><span> </span><span>m</span><span>=</span><span>mean</span><span>(</span><span>value</span><span>))</span><span><br data-jekyll-commonmark-ghpages=""> <br data-jekyll-commonmark-ghpages=""></span><span>df2</span><span> </span><span>&lt;-</span><span> </span><span>Rmisc</span><span>::</span><span>summarySE</span><span>(</span><span>mm</span><span>,</span><span> </span><span>&quot;value&quot;</span><span>,</span><span> </span><span>&quot;variable&quot;</span><span>)</span><span><br data-jekyll-commonmark-ghpages=""></span><span>ggplot</span><span>(</span><span>df2</span><span>,</span><span> </span><span>aes</span><span>(</span><span>x</span><span>=</span><span>variable</span><span>,</span><span> </span><span>y</span><span>=</span><span>value</span><span>))</span><span> </span><span>+</span><span> <br data-jekyll-commonmark-ghpages=""> </span><span>geom_point</span><span>()</span><span>+</span><span><br data-jekyll-commonmark-ghpages=""> </span><span>geom_errorbar</span><span>(</span><span>aes</span><span>(</span><span>ymin</span><span>=</span><span>value</span><span>-</span><span>se</span><span>,</span><span> </span><span>ymax</span><span>=</span><span>value</span><span>+</span><span>se</span><span>),</span><span> </span><span>width</span><span>=</span><span>.2</span><span>,</span><span><br data-jekyll-commonmark-ghpages=""> </span><span>position</span><span>=</span><span>position_dodge</span><span>(</span><span>0.05</span><span>))</span><span> </span><span>+</span><span>ylab</span><span>(</span><span>&quot;Mean Error&quot;</span><span>)</span><span> </span><span>+</span><span> </span><span>xlab</span><span>(</span><span>&quot;Regression Input&quot;</span><span>)</span><span> </span><span>+</span><span> </span><span>theme_bw</span><span>()</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-47" src="https://i0.wp.com/flovv.github.io/figures/post28/post28unnamed-chunk-47-1.png?w=456" data-recalc-dims="1"></p>
<p>Further, we can test if the embedding model outperforms the categorical regression model in an out of sample evaluation. The plot above shows the mean error over 100 test-runs (training on 80%, testing on 20%, metric: RMSE.) The embedding model (with a more compact representation of the day of the week) outperforms the categorical model.</p>
<p>I expect to see more data scientists using embeddings for categorical variables in the upcoming years for prediction problems. While the mapping reduces interpretability, it apparently helps to find a better prediction model.<br>
This has also been shown by <a rel="nofollow" href="https://arxiv.org/abs/1604.06737" target="_blank">Cheng Guo and Felix Berkhahn</a>. They note: “embeddings help to generalize better when the data is sparse and statistics is unknown. Thus, it is especially useful for datasets with lots of high cardinality features, where other methods tend to overfit.”</p>
<p>In a follow up post, I will try to transfer learn such embeddings. The idea is to learn a (weekday) embedding on one problem and transfer it to a different dataset.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://flovv.github.io/Embeddings_with_keras/"> Florian Teschner</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/sSkz1cS4et8" height="1"></a></div>