<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/15-tidyverse-and-data-table-sitting-side-by-side-part-1/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://dirk.eddelbuettel.com/blog/2018/01/21#015_tidyverse_and_datatable_part_1"> Thinking inside the box </a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Welcome to the fifteenth post in the <em>rarely rational R rambling</em> series, or <span><em>R</em><sup>4</sup></span> for short. There are two posts I have been meaning to get out for a bit, and hope to get to shortly—but in the meantime we are going start something else.</p>
<p>Another longer-running idea I had was to present some simple application cases with (one or more) <em>side-by-side</em> code comparisons. Why? Well at times it feels like R, and the R community, are being split. You’re either with one (increasingly &quot;religious&quot; in their defense of their deemed-superior approach) side, or the other. And that is of course utter nonsense. It’s all R after all.</p>
<p>Programming, just like other fields using engineering methods and thinking, is about making choices, and trading off between certain aspects. A simple example is the fairly well-known trade-off between memory use and speed: think <em>e.g.</em> of a hash map allowing for faster lookup at the cost of some more memory. Generally speaking, solutions are rarely limited to just one way, or just one approach. So if pays off to know your tools, and choose wisely among <em>all</em> available options. Having choices is having options, and those tend to have non-negative premiums to take advantage off. Locking yourself into one and just one paradigm can never be better.</p>
<p>In that spirit, I want to (eventually) show a few <em>simple</em> comparisons of code being done two distinct ways.</p>
<p>One obvious first candidate for this is the <a rel="nofollow" href="https://github.com/NYTimes/gunsales" target="_blank">gunsales</a> repository with some R code which backs an earlier NY Times article. I got involved for a similar reason, and updated the code from its initial form. Then again, this project also helped motivate what we did later with the <a rel="nofollow" href="https://github.com/x13org/x13binary" target="_blank">x13binary</a> package which permits automated installation of the X13-ARIMA-SEATS binary to support Christoph’s excellent <a rel="nofollow" href="https://github.com/christophsax/seasonal" target="_blank">seasonal</a> CRAN package (and <a rel="nofollow" href="http://www.seasonal.website/" target="_blank">website</a>) for which we now have a forthcoming JSS paper. But the actual code example is not that interesting / a bit further off the mainstream because of the more specialised seasonal ARIMA modeling.</p>
<p>But then this week I found a much simpler and shorter example, and quickly converted its code. The code comes from the inaugural <a rel="nofollow" href="http://www.thecrosstab.com/datascience/r-1/" target="_blank">datascience 1</a> lesson at the <a rel="nofollow" href="http://www.thecrosstab.com/" target="_blank">Crosstab</a>, a fabulous site by G. Elliot Morris (who may be the highest-energy undergrad I have come across lately) focusssed on political polling, forecasts, and election outcomes. Lesson 1 is a simple introduction, and averages some polls of the 2016 US Presidential Election.</p>
<h4>Complete Code using Approach &quot;TV&quot;</h4>
<p>Elliot does a fine job <a rel="nofollow" href="http://www.thecrosstab.com/datascience/r-1/" target="_blank">walking the reader through his code</a> so I will be brief and simply quote it in one piece:</p>
<div>
<pre><code>
## Getting the polls

<span>library</span>(readr)
polls_<span>2016</span> &lt;-<span> </span><span>read_tsv</span>(<span>url</span>(<span>&quot;http://elections.huffingtonpost.com/pollster/api/v2/questions/16-US-Pres-GE%20TrumpvClinton/poll-responses-clean.tsv&quot;</span>))

## Wrangling the polls

<span>library</span>(dplyr)
polls_<span>2016</span> &lt;-<span> </span>polls_<span>2016</span> <span>%>%</span>
<span> </span><span>filter</span>(sample_subpopulation <span>%in%</span><span> </span><span>c</span>(<span>&quot;Adults&quot;</span>,<span>&quot;Likely Voters&quot;</span>,<span>&quot;Registered Voters&quot;</span>))
<span>library</span>(lubridate)
polls_<span>2016</span> &lt;-<span> </span>polls_<span>2016</span> <span>%>%</span>
<span> </span><span>mutate</span>(<span>end_date =</span> <span>ymd</span>(end_date))
polls_<span>2016</span> &lt;-<span> </span>polls_<span>2016</span> <span>%>%</span>
<span> </span><span>right_join</span>(<span>data.frame</span>(<span>end_date =</span> <span>seq.Date</span>(<span>min</span>(polls_<span>2016</span><span>$</span>end_date),
                                              <span>max</span>(polls_<span>2016</span><span>$</span>end_date), <span>by=</span><span>&quot;days&quot;</span>)))

## Average the polls

polls_<span>2016</span> &lt;-<span> </span>polls_<span>2016</span> <span>%>%</span>
<span> </span><span>group_by</span>(end_date) <span>%>%</span>
<span> </span><span>summarise</span>(<span>Clinton =</span> <span>mean</span>(Clinton),
              <span>Trump =</span> <span>mean</span>(Trump))

<span>library</span>(zoo)
rolling_average &lt;-<span> </span>polls_<span>2016</span> <span>%>%</span>
<span> </span><span>mutate</span>(<span>Clinton.Margin =</span> Clinton<span>-</span>Trump,
           <span>Clinton.Avg =</span>  <span>rollapply</span>(Clinton.Margin,<span>width=</span><span>14</span>,
                                    <span>FUN=</span><span>function</span>(x){<span>mean</span>(x, <span>na.rm=</span><span>TRUE</span>)},
                                    <span>by=</span><span>1</span>, <span>partial=</span><span>TRUE</span>, <span>fill=</span><span>NA</span>, <span>align=</span><span>&quot;right&quot;</span>))

<span>library</span>(ggplot2)
<span>ggplot</span>(rolling_average)<span>+</span>
<span> </span><span>geom_line</span>(<span>aes</span>(<span>x=</span>end_date,<span>y=</span>Clinton.Avg),<span>col=</span><span>&quot;blue&quot;</span>) <span>+</span>
<span> </span><span>geom_point</span>(<span>aes</span>(<span>x=</span>end_date,<span>y=</span>Clinton.Margin))</code></pre>
</div>
<p>It uses five packages to i) read some data off them interwebs, ii) then filters / subsets / modifies it leading to a <a rel="nofollow" href="https://en.wikipedia.org/wiki/Join_(SQL)#Right_outer_join" target="_blank">right (outer) join</a> with itself before iv) averaging per-day polls first and then creates rolling averages over 14 days before v) plotting. Several standard <em>verbs</em> are used: <code>filter()</code>, <code>mutate()</code>, <code>right_join()</code>, <code>group_by()</code>, and <code>summarise()</code>. One non-verse function is <code>rollapply()</code> which comes from <a rel="nofollow" href="https://cran.r-project.org/web/packages/zoo/index.html" target="_blank">zoo</a>, a popular package for time-series data.</p>
<h4>Complete Code using Approach &quot;DT&quot;</h4>
<p>As I will show below, we can do the same with fewer packages as <a rel="nofollow" href="https://github.com/Rdatatable/data.table/wiki" target="_blank">data.table</a> covers the reading, slicing/dicing and time conversion. We still need <a rel="nofollow" href="https://cran.r-project.org/web/packages/zoo/index.html" target="_blank">zoo</a> for its <code>rollapply()</code> and of course the same plotting code:</p>
<div>
<pre><code>
## Getting the polls

<span>library</span>(data.table)
pollsDT &lt;-<span> </span><span>fread</span>(<span>&quot;http://elections.huffingtonpost.com/pollster/api/v2/questions/16-US-Pres-GE%20TrumpvClinton/poll-responses-clean.tsv&quot;</span>)

## Wrangling the polls

pollsDT &lt;-<span> </span>pollsDT[sample_subpopulation <span>%in%</span><span> </span><span>c</span>(<span>&quot;Adults&quot;</span>,<span>&quot;Likely Voters&quot;</span>,<span>&quot;Registered Voters&quot;</span>), ]
pollsDT[, end_date <span>:</span><span>=</span><span> </span><span>as.IDate</span>(end_date)]
pollsDT &lt;-<span> </span>pollsDT[ <span>data.table</span>(<span>end_date =</span> <span>seq</span>(<span>min</span>(pollsDT[,end_date]),
                                              <span>max</span>(pollsDT[,end_date]), <span>by=</span><span>&quot;days&quot;</span>)), on=<span>&quot;end_date&quot;</span>]

## Average the polls

<span>library</span>(zoo)
pollsDT &lt;-<span> </span>pollsDT[, .(<span>Clinton=</span><span>mean</span>(Clinton), <span>Trump=</span><span>mean</span>(Trump)), by=end_date]
pollsDT[, Clinton.Margin <span>:</span><span>=</span><span> </span>Clinton<span>-</span>Trump]
pollsDT[, Clinton.Avg <span>:</span><span>=</span><span> </span><span>rollapply</span>(Clinton.Margin, <span>width=</span><span>14</span>,
                                   <span>FUN=</span><span>function</span>(x){<span>mean</span>(x, <span>na.rm=</span><span>TRUE</span>)},
                                   <span>by=</span><span>1</span>, <span>partial=</span><span>TRUE</span>, <span>fill=</span><span>NA</span>, <span>align=</span><span>&quot;right&quot;</span>)]

<span>library</span>(ggplot2)
<span>ggplot</span>(pollsDT) <span>+</span>
<span> </span><span>geom_line</span>(<span>aes</span>(<span>x=</span>end_date,<span>y=</span>Clinton.Avg),<span>col=</span><span>&quot;blue&quot;</span>) <span>+</span>
<span> </span><span>geom_point</span>(<span>aes</span>(<span>x=</span>end_date,<span>y=</span>Clinton.Margin))</code></pre>
</div>
<p>This uses several of the components of <code>data.table</code> which are often called <code>[i, j, by=...]</code>. Row are selected (<code>i</code>), columns are either modified (via <code>:=</code> assignment) or summarised (via <code>=</code>), and grouping is undertaken by <code>by=...</code>. The outer join is done by having a <code>data.table</code> object indexed by another, and is pretty standard too. That allows us to do all transformations in three lines. We then create per-day average by grouping by day, compute the margin and construct its rolling average as before. The resulting chart is, unsurprisingly, the same.</p>
<h4>Benchmark Reading</h4>
<p>We can looking how the two approaches do on getting data read into our session. For simplicity, we will read a local file to keep the (fixed) download aspect out of it:</p>
<div>
<pre><code>R<span>></span><span> </span>url &lt;-<span> &quot;http://elections.huffingtonpost.com/pollster/api/v2/questions/16-US-Pres-GE%20TrumpvClinton/poll-responses-clean.tsv&quot;</span>
R<span>></span><span> </span><span>download.file</span>(url, <span>destfile=</span>file, <span>quiet=</span><span>TRUE</span>)
R<span>></span><span> </span>file &lt;-<span> &quot;/tmp/poll-responses-clean.tsv&quot;</span>
R<span>></span><span> </span>res &lt;-<span> </span><span>microbenchmark</span>(<span>tidy=</span><span>suppressMessages</span>(readr<span>::</span><span>read_tsv</span>(file)),
<span>+</span><span> </span><span>dt=</span>data.table<span>::</span><span>fread</span>(file, <span>showProgress=</span><span>FALSE</span>))
R<span>></span><span> </span>res
Unit<span>:</span><span> </span>milliseconds
 expr     min      lq    mean  median      uq      max neval
 tidy <span>6.67777</span> <span>6.83458</span> <span>7.13434</span> <span>6.98484</span> <span>7.25831</span>  <span>9.27452</span>   <span>100</span>
   dt <span>1.98890</span> <span>2.04457</span> <span>2.37916</span> <span>2.08261</span> <span>2.14040</span> <span>28.86885</span>   <span>100</span>
R<span>></span><span> </span></code></pre>
</div>
<p>That is a clear relative difference, though the absolute amount of time is not that relevant for such a small (demo) dataset.</p>
<h4>Benchmark Processing</h4>
<p>We can also look at the processing part:</p>
<div>
<pre><code>R<span>></span><span> </span>rdin &lt;-<span> </span><span>suppressMessages</span>(readr<span>::</span><span>read_tsv</span>(file))
R<span>></span><span> </span>dtin &lt;-<span> </span>data.table<span>::</span><span>fread</span>(file, <span>showProgress=</span><span>FALSE</span>)
R<span>></span><span> </span>
R<span>></span><span> </span><span>library</span>(dplyr)
R<span>></span><span> </span><span>library</span>(lubridate)
R<span>></span><span> </span><span>library</span>(zoo)
R<span>></span><span> </span>
R<span>></span><span> </span>transformTV &lt;-<span> </span><span>function</span>(<span>polls_2016=</span>rdin) {
<span>+</span><span> </span>polls_<span>2016</span> &lt;-<span> </span>polls_<span>2016</span> <span>%>%</span>
<span>+</span><span> </span><span>filter</span>(sample_subpopulation <span>%in%</span><span> </span><span>c</span>(<span>&quot;Adults&quot;</span>,<span>&quot;Likely Voters&quot;</span>,<span>&quot;Registered Voters&quot;</span>))
<span>+</span><span> </span>polls_<span>2016</span> &lt;-<span> </span>polls_<span>2016</span> <span>%>%</span>
<span>+</span><span> </span><span>mutate</span>(<span>end_date =</span> <span>ymd</span>(end_date))
<span>+</span><span> </span>polls_<span>2016</span> &lt;-<span> </span>polls_<span>2016</span> <span>%>%</span>
<span>+</span><span> </span><span>right_join</span>(<span>data.frame</span>(<span>end_date =</span> <span>seq.Date</span>(<span>min</span>(polls_<span>2016</span><span>$</span>end_date), 
<span>+</span><span> </span><span>max</span>(polls_<span>2016</span><span>$</span>end_date), <span>by=</span><span>&quot;days&quot;</span>)))
<span>+</span><span> </span>polls_<span>2016</span> &lt;-<span> </span>polls_<span>2016</span> <span>%>%</span>
<span>+</span><span> </span><span>group_by</span>(end_date) <span>%>%</span>
<span>+</span><span> </span><span>summarise</span>(<span>Clinton =</span> <span>mean</span>(Clinton),
<span>+</span><span> </span><span>Trump =</span> <span>mean</span>(Trump))
<span>+</span><span> </span>
<span>+</span><span> </span>rolling_average &lt;-<span> </span>polls_<span>2016</span> <span>%>%</span>
<span>+</span><span> </span><span>mutate</span>(<span>Clinton.Margin =</span> Clinton<span>-</span>Trump,
<span>+</span><span> </span><span>Clinton.Avg =</span>  <span>rollapply</span>(Clinton.Margin,<span>width=</span><span>14</span>,
<span>+</span><span> </span><span>FUN=</span><span>function</span>(x){<span>mean</span>(x, <span>na.rm=</span><span>TRUE</span>)}, 
<span>+</span><span> </span><span>by=</span><span>1</span>, <span>partial=</span><span>TRUE</span>, <span>fill=</span><span>NA</span>, <span>align=</span><span>&quot;right&quot;</span>))
<span>+</span><span> </span>}
R<span>></span><span> </span>
R<span>></span><span> </span>transformDT &lt;-<span> </span><span>function</span>(dtin) {
<span>+</span><span> </span>pollsDT &lt;-<span> </span><span>copy</span>(dtin) ## extra work to protect from reference semantics for benchmark
<span>+</span><span> </span>pollsDT &lt;-<span> </span>pollsDT[sample_subpopulation <span>%in%</span><span> </span><span>c</span>(<span>&quot;Adults&quot;</span>,<span>&quot;Likely Voters&quot;</span>,<span>&quot;Registered Voters&quot;</span>), ]
<span>+</span><span> </span>pollsDT[, end_date <span>:</span><span>=</span><span> </span><span>as.IDate</span>(end_date)]
<span>+</span><span> </span>pollsDT &lt;-<span> </span>pollsDT[ <span>data.table</span>(<span>end_date =</span> <span>seq</span>(<span>min</span>(pollsDT[,end_date]), 
<span>+</span><span> </span><span>max</span>(pollsDT[,end_date]), <span>by=</span><span>&quot;days&quot;</span>)), on=<span>&quot;end_date&quot;</span>]
<span>+</span><span> </span>pollsDT &lt;-<span> </span>pollsDT[, .(<span>Clinton=</span><span>mean</span>(Clinton), <span>Trump=</span><span>mean</span>(Trump)), 
<span>+</span><span> </span>by=end_date][, Clinton.Margin <span>:</span><span>=</span><span> </span>Clinton<span>-</span>Trump]
<span>+</span><span> </span>pollsDT[, Clinton.Avg <span>:</span><span>=</span><span> </span><span>rollapply</span>(Clinton.Margin, <span>width=</span><span>14</span>,
<span>+</span><span> </span><span>FUN=</span><span>function</span>(x){<span>mean</span>(x, <span>na.rm=</span><span>TRUE</span>)}, 
<span>+</span><span> </span><span>by=</span><span>1</span>, <span>partial=</span><span>TRUE</span>, <span>fill=</span><span>NA</span>, <span>align=</span><span>&quot;right&quot;</span>)]
<span>+</span><span> </span>}
R<span>></span><span> </span>
R<span>></span><span> </span>res &lt;-<span> </span><span>microbenchmark</span>(<span>tidy=</span><span>suppressMessages</span>(<span>transformTV</span>(rdin)),
<span>+</span><span> </span><span>dt=</span><span>transformDT</span>(dtin))
R<span>></span><span> </span>res
Unit<span>:</span><span> </span>milliseconds
 expr      min       lq     mean   median       uq      max neval
 tidy <span>12.54723</span> <span>13.18643</span> <span>15.29676</span> <span>13.73418</span> <span>14.71008</span> <span>104.5754</span>   <span>100</span>
   dt  <span>7.66842</span>  <span>8.02404</span>  <span>8.60915</span>  <span>8.29984</span>  <span>8.72071</span>  <span>17.7818</span>   <span>100</span>
R<span>></span><span> </span></code></pre>
</div>
<p>Not quite a factor of two on the small data set, but again a clear advantage. <code>data.table</code> has a reputation for doing really well for large datasets; here we see that it is also faster for small datasets.</p>
<h4>Side-by-side</h4>
<p>Stripping the reading, as well as the plotting both of which are about the same, we can compare the essential data operations.</p>
<div>
<img src="https://i1.wp.com/dirk.eddelbuettel.com/images/crosstab-r1-side-by-side.png?w=456" data-recalc-dims="1">
</div>
<h4>Summary</h4>
<p>We found a simple task solved using code and packages from an increasingly popular sub-culture within R, and contrasted it with a second approach. We find the second approach to i) have fewer dependencies, ii) less code, and iii) running faster.</p>
<p>Now, undoubtedly the former approach will have its staunch defenders (and that is all good and well, after all choice is good and even thirty years later some still debate <code>vi</code> versus <code>emacs</code> endlessly) but I thought it to be instructive to at least to be able to make an informed comparison.</p>
<h4>Acknowledgements</h4>
<p>My thanks to <a rel="nofollow" href="https://twitter.com/gelliottmorris" target="_blank">G. Elliot Morris</a> for a fine example, and of course a fine blog and (if somewhat hyperactive) Twitter account.</p>
<p>
This post by <a rel="nofollow" href="http://dirk.eddelbuettel.com/" target="_blank">Dirk Eddelbuettel</a> originated on his <a rel="nofollow" href="http://dirk.eddelbuettel.com/blog/" target="_blank">Thinking inside the box</a> blog. Please report excessive re-aggregation in third-party for-profit settings.</p>
<p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://dirk.eddelbuettel.com/blog/2018/01/21#015_tidyverse_and_datatable_part_1"> Thinking inside the box </a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/NBSwq0qHrp4" height="1"></a></div>