<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/tfestimators-package-embeddings-for-categorical-variables/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://flovv.github.io/Embeddings_with_tf/"> Florian Teschner</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In my last posts (<a rel="nofollow" href="http://flovv.github.io/Embeddings_with_keras/" target="_blank">here</a> and <a rel="nofollow" href="http://flovv.github.io/Embeddings_with_keras/" target="_blank">here</a>) I explored how to use embeddings to represent categorical variables. Furthermore, I showed how to represent categorical variables with embeddings and add other variable to create a more complex model. Both posts focused on the Keras (R) functionality. I concluded that it feels artificial to represent categorical variables with embeddings in Keras. Especially concatenating multiple input layers is quite cumbersome with the current Keras interface.</p>
<p>This week, I watched the <a rel="nofollow" href="https://www.youtube.com/watch?v=atiYXm7JZv0" target="_blank">official release video</a> by J.J. Allaire and learnt about the tfestimators package. It turns out that the awesome rstudio team build a very handy interface to access tensorflow and train models with multiple parameters and embeddings.</p>
<p>In this post, I will show how to use the package to quickly fit a model in which categorical variables are represented as embeddings.</p>
<p>As in the posts before, I work with the nyc citi bike count data from Kaggle. It contains daily bicycle counts for 4 major bridges in NYC. In order to have a longer dataset, I use the bicycle count for all bridges as the dependent variable.</p>
<figure>
<pre><code data-lang="r"><span>#https://www.kaggle.com/new-york-city/nyc-east-river-bicycle-crossings</span><span>
</span><span>df</span><span> </span><span>&lt;-</span><span> </span><span>read.csv</span><span>(</span><span>&quot;data/nyc-east-river-bicycle-counts.csv&quot;</span><span>)</span><span>
</span><span>dflong</span><span> </span><span>&lt;-</span><span> </span><span>data.table</span><span>::</span><span>melt</span><span>(</span><span>df</span><span>[</span><span>c</span><span>(</span><span>&quot;Date&quot;</span><span>,</span><span> </span><span>&quot;Brooklyn.Bridge&quot;</span><span>,</span><span> </span><span>&quot;Manhattan.Bridge&quot;</span><span>,</span><span> </span><span>&quot;Williamsburg.Bridge&quot;</span><span> </span><span>,</span><span>&quot;Queensboro.Bridge&quot;</span><span>)],</span><span> </span><span>idvars</span><span>=</span><span>&quot;date&quot;</span><span>)</span><span>
</span><span>dflong</span><span>$</span><span>date</span><span> </span><span>&lt;-</span><span> </span><span>as.Date</span><span>(</span><span>dflong</span><span>$</span><span>Date</span><span>)</span><span>
</span><span>dflong</span><span>$</span><span>weekday</span><span> </span><span>&lt;-</span><span> </span><span>wday</span><span>(</span><span>dflong</span><span>$</span><span>date</span><span>,</span><span> </span><span>label</span><span> </span><span>=</span><span> </span><span>T</span><span>)</span><span>
</span><span>dflong</span><span> </span><span>&lt;-</span><span> </span><span>merge</span><span>(</span><span>dflong</span><span>,</span><span> </span><span>df</span><span>[,</span><span> </span><span>c</span><span>(</span><span>&quot;Date&quot;</span><span>,</span><span> </span><span>&quot;Precipitation&quot;</span><span>,</span><span> </span><span>&quot;Low.Temp..Â.F.&quot;</span><span>)],</span><span> </span><span>by</span><span>=</span><span>&quot;Date&quot;</span><span>)</span><span>
</span><span>dflong</span><span>$</span><span>ScaledUsers</span><span> </span><span>&lt;-</span><span> </span><span>scale</span><span>(</span><span>dflong</span><span>$</span><span>value</span><span>)</span><span>
</span><span>dflong</span><span>$</span><span>lowTemp</span><span> </span><span>&lt;-</span><span> </span><span>scale</span><span>(</span><span>dflong</span><span>[,</span><span>&quot;Low.Temp..Â.F.&quot;</span><span>])</span><span>
</span><span>dflong</span><span>$</span><span>rain</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>dflong</span><span>$</span><span>Precipitation</span><span> </span><span>!=</span><span> </span><span>0</span><span>,</span><span> </span><span>0</span><span>,</span><span>1</span><span>)</span><span>
</span><span>dflong</span><span>$</span><span>Bridge</span><span> </span><span>&lt;-</span><span> </span><span>factor</span><span>(</span><span>dflong</span><span>$</span><span>variable</span><span>)</span><span>
</span><span>levels</span><span>(</span><span>dflong</span><span>$</span><span>Bridge</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>1</span><span>:</span><span>length</span><span>(</span><span>levels</span><span>(</span><span>dflong</span><span>$</span><span>Bridge</span><span>))</span><span>
</span><span>levels</span><span>(</span><span>dflong</span><span>$</span><span>weekday</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>1</span><span>:</span><span>length</span><span>(</span><span>levels</span><span>(</span><span>dflong</span><span>$</span><span>weekday</span><span>))</span></code></pre>
</figure>
<p>The goal of our play model is to predict the number of bicycle per day on a certain bridge dependent on the weekday, the bridge (“Brooklyn.Bridge”, “Manhattan.Bridge”, “Williamsburg.Bridge” ,”Queensboro.Bridge”), if it rains and the temperature. So overall we have 2 categorical variables, one binary and one continuous variable.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>tfestimators</span><span>)</span><span>
</span><span>## convert the factor to integer -- tfestimators is strict with input types.</span><span>
</span><span>dflong</span><span>$</span><span>Bridge</span><span> </span><span>&lt;-</span><span> </span><span>as.integer</span><span>(</span><span>dflong</span><span>$</span><span>Bridge</span><span>)</span><span>
</span><span>dflong</span><span>$</span><span>weekday</span><span> </span><span>&lt;-</span><span> </span><span>as.integer</span><span>(</span><span>dflong</span><span>$</span><span>weekday</span><span>)</span><span>
</span><span>embedding_dimension_bridges</span><span> </span><span>=</span><span> </span><span>2</span><span>
</span><span>embedding_dimension_weekdays</span><span> </span><span>=</span><span> </span><span>3</span><span>
</span><span>cols</span><span> </span><span>&lt;-</span><span> </span><span>feature_columns</span><span>(</span><span>
</span><span>column_numeric</span><span>(</span><span>&quot;lowTemp&quot;</span><span>,</span><span>&quot;rain&quot;</span><span>),</span><span>
</span><span>column_embedding</span><span>(</span><span>column_categorical_with_vocabulary_list</span><span>(</span><span>&quot;weekday&quot;</span><span>,</span><span> </span><span>vocabulary_list</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>1</span><span>:</span><span>7</span><span>)),</span><span> </span><span>embedding_dimension_weekdays</span><span>),</span><span>
</span><span>column_embedding</span><span>(</span><span>column_categorical_with_vocabulary_list</span><span>(</span><span>&quot;Bridge&quot;</span><span>,</span><span>vocabulary_list</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>1</span><span>:</span><span>4</span><span>)),</span><span> </span><span>embedding_dimension_bridges</span><span>)</span><span>
</span><span>)</span></code></pre>
</figure>
<p>The first step that we need to do is to define the input variables and their type. Let’s start with the simple numeric variables lowTemp and rain. We just define the input as “column_numeric(“lowTemp”,”rain”)”. Next, the two categorical variables that we want to embed, need a bit more work. a) they need a list of all possible values (defined in vocabulary_list parameter). Additionally, we need to define the embedding_dimension for each categorical variable.</p>
<p>Next, we write a short function that defines the input and output of the the model as well as batch size and number of epochs.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>tfestimators</span><span>)</span><span>
</span><span>bridge_input_fn</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>data</span><span>,</span><span> </span><span>num_epochs</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span> </span><span>{</span><span>
</span><span>tfestimators</span><span>::</span><span>input_fn</span><span>(</span><span>data</span><span>,</span><span> </span><span>features</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;lowTemp&quot;</span><span>,</span><span>&quot;rain&quot;</span><span>,</span><span>&quot;weekday&quot;</span><span>,</span><span> </span><span>&quot;Bridge&quot;</span><span>),</span><span>
</span><span>response</span><span> </span><span>=</span><span> </span><span>&quot;ScaledUsers&quot;</span><span>,</span><span>
</span><span>batch_size</span><span> </span><span>=</span><span> </span><span>2</span><span>,</span><span>
</span><span>num_epochs</span><span> </span><span>=</span><span> </span><span>num_epochs</span><span>)</span><span>
</span><span>}</span><span>
</span><span>############ Train and Test Dataset #############</span><span>
</span><span>indices</span><span> </span><span>&lt;-</span><span> </span><span>sample</span><span>(</span><span>1</span><span>:</span><span>nrow</span><span>(</span><span>dflong</span><span>),</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>0.80</span><span> </span><span>*</span><span> </span><span>nrow</span><span>(</span><span>dflong</span><span>))</span><span>
</span><span>train</span><span> </span><span>&lt;-</span><span> </span><span>dflong</span><span>[</span><span>indices</span><span>,</span><span> </span><span>]</span><span>
</span><span>test</span><span> </span><span>&lt;-</span><span> </span><span>dflong</span><span>[</span><span>-</span><span>indices</span><span>,</span><span> </span><span>]</span><span>
</span><span>############## Define the model ############</span><span>
</span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>dnn_regressor</span><span>(</span><span>feature_columns</span><span> </span><span>=</span><span> </span><span>cols</span><span>,</span><span> </span><span>hidden_units</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>32</span><span>,</span><span> </span><span>10</span><span>),</span><span> </span><span>dropout</span><span> </span><span>=</span><span> </span><span>0.15</span><span>)</span><span>
</span><span># train the model</span><span>
</span><span>history</span><span> </span><span>&lt;-</span><span> </span><span>model</span><span> </span><span>%>%</span><span> </span><span>train</span><span>(</span><span>bridge_input_fn</span><span>(</span><span>train</span><span>[,</span><span>c</span><span>(</span><span>&quot;ScaledUsers&quot;</span><span>,</span><span> </span><span>&quot;lowTemp&quot;</span><span>,</span><span>&quot;rain&quot;</span><span>,</span><span>&quot;weekday&quot;</span><span>,</span><span> </span><span>&quot;Bridge&quot;</span><span> </span><span>)],</span><span> </span><span>num_epochs</span><span> </span><span>=</span><span> </span><span>1</span><span>))</span><span>
</span><span>## eval the output</span><span>
</span><span>model</span><span> </span><span>%>%</span><span> </span><span>evaluate</span><span>(</span><span>bridge_input_fn</span><span>(</span><span>test</span><span>))</span></code></pre>
</figure>
<p>In order to evaluate the model, we split the data in train and test set. We define the model as deep neural network (DNN), regression model with two hidden layers (one with 32 the other with 10 nodes). Compared to the Keras version, in which one needs to concatenate different input layers this interface is straight-forward.<br>
Finally, we check the model’s accuracy on test set and print the learning history.</p>
<figure>
<pre><code data-lang="r"><span>require</span><span>(</span><span>ggplot2</span><span>)</span><span>
</span><span>df</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>losses</span><span> </span><span>=</span><span> </span><span>history</span><span>$</span><span>losses</span><span>$</span><span>mean_losses</span><span>,</span><span> </span><span>steps</span><span>=</span><span>history</span><span>$</span><span>step</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>df</span><span>,</span><span> </span><span>aes</span><span>(</span><span>steps</span><span>,</span><span> </span><span>losses</span><span>))</span><span>+</span><span> </span><span>geom_point</span><span>()</span><span> </span><span>+</span><span>geom_smooth</span><span>()</span><span> </span><span>+</span><span>theme_bw</span><span>()</span><span> </span><span>+</span><span> </span><span>ylab</span><span>(</span><span>&quot;Loss&quot;</span><span>)</span><span> </span><span>+</span><span> </span><span>xlab</span><span>(</span><span>&quot;Training Steps&quot;</span><span>)</span><span> </span><span>+</span><span> </span><span>ggtitle</span><span>(</span><span>&quot;Testing: TF Estimators&quot;</span><span>)</span></code></pre>
</figure>
<p><img alt="plot of chunk unnamed-chunk-5" src="https://i1.wp.com/flovv.github.io/figures/post30/figuresunnamed-chunk-5-1.png?w=456" data-recalc-dims="1"></p>
<p>To conclude, the package is a great step forward to apply deep neural nets to everyday problems and to quickly use embeddings for categorical variables. Big Kudus to the Rstudio team for their efforts. If you have time left and want a quick update on deep learning for the R community check out <a rel="nofollow" href="https://www.youtube.com/watch?v=atiYXm7JZv0" target="_blank">J.J. Allaire’s video</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://flovv.github.io/Embeddings_with_tf/"> Florian Teschner</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/8BUOstMg8Q8" height="1"></a></div>