<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/tidyverse-and-data-table-sitting-side-by-side-and-then-base-r-walks-in/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://www.enchufa2.es/archives/tidyverse-and-data-table-sitting-side-by-side-and-then-base-r-walks-in.html"> R – Enchufa2</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Of course, I’m paraphrasing Dirk’s fifteenth post in the <em>rarely rational R rambling</em> series: <a rel="nofollow" href="http://dirk.eddelbuettel.com/blog/2018/01/21/#015_tidyverse_and_datatable_part_1" target="_blank">#15: Tidyverse and data.table, sitting side by side … (Part 1)</a>. I very much liked it, because, although I’m a happy tidyverse user, I’m always trying not to be tied into that <em>verse</em> too much by replicating certain tasks with other tools (and languages) as an exercise. In this article, I’m going to repeat Dirk’s exercise in base R.</p>
<p>First of all, I would like to clean up the tidyverse version a little, because the original was distributed in chunks and was a little bit too verbose. We can also avoid using <code>lubridate</code>, because <code>readr</code> already parses the <code>end_date</code> column as a date (and that’s why it is significantly slower, among other reasons). This is how I would do it:</p>
<pre><code><span>## Getting the polls</span>

<span>library</span>(tidyverse)
<span>library</span>(zoo)

polls_2016 &lt;- read_tsv(url(<span>&quot;http://elections.huffingtonpost.com/pollster/api/v2/questions/16-US-Pres-GE%20TrumpvClinton/poll-responses-clean.tsv&quot;</span>))

<span>## Wrangling the polls</span>

polls_2016 &lt;- polls_2016 %>%
  filter(sample_subpopulation %<span>in</span>% c(<span>&quot;Adults&quot;</span>,<span>&quot;Likely Voters&quot;</span>,<span>&quot;Registered Voters&quot;</span>)) %>%
  right_join(data.frame(end_date = seq.Date(min(.$end_date), max(.$end_date), by=<span>&quot;days&quot;</span>)), by=<span>&quot;end_date&quot;</span>)

<span>## Average the polls</span>

rolling_average &lt;- polls_2016 %>%
  group_by(end_date) %>%
  summarise(Clinton = mean(Clinton), Trump = mean(Trump)) %>%
  mutate(Clinton.Margin = Clinton-Trump,
         Clinton.Avg =  rollapply(Clinton.Margin,width=<span>14</span>,
                                  FUN=<span>function</span>(x){mean(x, na.rm=<span>TRUE</span>)},
                                  by=<span>1</span>, partial=<span>TRUE</span>, fill=<span>NA</span>, align=<span>&quot;right&quot;</span>))

ggplot(rolling_average) +
  geom_line(aes(x=end_date, y=Clinton.Avg), col=<span>&quot;blue&quot;</span>) +
  geom_point(aes(x=end_date, y=Clinton.Margin))</code></pre>
<p>which, by the way, has exactly the very same number of lines of code than the <code>data.table</code> version:</p>
<pre><code><span>## Getting the polls</span>

<span>library</span>(data.table)
<span>library</span>(zoo)
<span>library</span>(ggplot2)

pollsDT &lt;- fread(<span>&quot;http://elections.huffingtonpost.com/pollster/api/v2/questions/16-US-Pres-GE%20TrumpvClinton/poll-responses-clean.tsv&quot;</span>)

<span>## Wrangling the polls</span>

pollsDT &lt;- pollsDT[sample_subpopulation %<span>in</span>% c(<span>&quot;Adults&quot;</span>,<span>&quot;Likely Voters&quot;</span>,<span>&quot;Registered Voters&quot;</span>), ]
pollsDT[, end_date := as.IDate(end_date)]
pollsDT &lt;- pollsDT[ data.table(end_date = seq(min(pollsDT[,end_date]),
                                              max(pollsDT[,end_date]), by=<span>&quot;days&quot;</span>)), on=<span>&quot;end_date&quot;</span>]

<span>## Average the polls</span>

pollsDT &lt;- pollsDT[, .(Clinton=mean(Clinton), Trump=mean(Trump)), by=end_date]
pollsDT[, Clinton.Margin := Clinton-Trump]
pollsDT[, Clinton.Avg := rollapply(Clinton.Margin, width=<span>14</span>,
                                   FUN=<span>function</span>(x){mean(x, na.rm=<span>TRUE</span>)},
                                   by=<span>1</span>, partial=<span>TRUE</span>, fill=<span>NA</span>, align=<span>&quot;right&quot;</span>)]

ggplot(pollsDT) +
  geom_line(aes(x=end_date, y=Clinton.Avg), col=<span>&quot;blue&quot;</span>) +
  geom_point(aes(x=end_date, y=Clinton.Margin))</code></pre>
<p>Let’s translate this into base R. It is easier to start from the <code>data.table</code> version, mainly because filtering and assigning have a similar look and feel. Unsurprisingly, we have <code>base::merge</code> for the merge operation and <code>stats::aggregate</code> for the aggregation phase. <code>base::as.Date</code> works just fine for these dates and <code>utils::read.csv</code> has the only drawback that you have to specify the separator. Without further ado, this is my version in base R:</p>
<pre><code><span>## Getting the polls</span>

<span>library</span>(zoo)

pollsB &lt;- read.csv(url(<span>&quot;http://elections.huffingtonpost.com/pollster/api/v2/questions/16-US-Pres-GE%20TrumpvClinton/poll-responses-clean.tsv&quot;</span>), sep=<span>&quot;\t&quot;</span>)

<span>## Wrangling the polls</span>

pollsB &lt;- pollsB[pollsB$sample_subpopulation %<span>in</span>% c(<span>&quot;Adults&quot;</span>,<span>&quot;Likely Voters&quot;</span>,<span>&quot;Registered Voters&quot;</span>), ]
pollsB$end_date &lt;- base::as.Date(pollsB$end_date)
endDate &lt;- data.frame(end_date = seq.Date(min(pollsB$end_date), max(pollsB$end_date), by=<span>&quot;days&quot;</span>))
pollsB &lt;- merge(pollsB, endDate, by=<span>&quot;end_date&quot;</span>, all=<span>TRUE</span>)

<span>## Average the polls</span>

pollsB &lt;- aggregate(cbind(Clinton, Trump) ~ end_date, data=pollsB, mean, na.action=na.pass)
pollsB$Clinton.Margin &lt;- pollsB$Clinton - pollsB$Trump
pollsB$Clinton.Avg &lt;- rollapply(pollsB$Clinton.Margin, width=<span>14</span>,
                                FUN=<span>function</span>(x){mean(x, na.rm=<span>TRUE</span>)},
                                by=<span>1</span>, partial=<span>TRUE</span>, fill=<span>NA</span>, align=<span>&quot;right&quot;</span>)

plot(pollsB$end_date, pollsB$Clinton.Margin, pch=<span>16</span>)
lines(pollsB$end_date, pollsB$Clinton.Avg, col=<span>&quot;blue&quot;</span>, lwd=<span>2</span>)</code></pre>
<p>which is the shortest one! Finally, let’s repeat the benchmark too:</p>
<pre><code><span>library</span>(microbenchmark)

url &lt;- <span>&quot;http://elections.huffingtonpost.com/pollster/api/v2/questions/16-US-Pres-GE%20TrumpvClinton/poll-responses-clean.tsv&quot;</span>
file &lt;- <span>&quot;/tmp/poll-responses-clean.tsv&quot;</span>
download.file(url, destfile=file, quiet=<span>TRUE</span>)
res &lt;- microbenchmark(tidy=suppressMessages(readr::read_tsv(file)),
                      dt=data.table::fread(file, showProgress=<span>FALSE</span>),
                      base=read.csv(file, sep=<span>&quot;\t&quot;</span>))
res</code></pre>
<pre><code>## Unit: milliseconds
##  expr       min        lq      mean    median        uq        max neval
##  tidy 13.877036 15.127885 18.549393 15.861311 17.813541 202.389391   100
##    dt  4.084022  4.505943  5.152799  4.845193  5.652579   7.736563   100
##  base 29.029366 30.437742 32.518009 31.449916 33.600937  45.104599   100</code></pre>
<p>Base R is clearly the slowest option for the reading phase. Or, one might say, both <code>readr</code> and <code>data.table</code> have done a great job in improving things! Let’s take a look at the processing part now:</p>
<pre><code>tvin &lt;- suppressMessages(readr::read_tsv(file))
dtin &lt;- data.table::fread(file, showProgress=<span>FALSE</span>)
bsin &lt;- read.csv(file, sep=<span>&quot;\t&quot;</span>)

<span>library</span>(tidyverse)
<span>library</span>(data.table)
<span>library</span>(zoo)

transformTV &lt;- <span>function</span>(polls_2016) {
  polls_2016 &lt;- polls_2016 %>%
    filter(sample_subpopulation %<span>in</span>% c(<span>&quot;Adults&quot;</span>,<span>&quot;Likely Voters&quot;</span>,<span>&quot;Registered Voters&quot;</span>)) %>%
    right_join(data.frame(end_date = seq.Date(min(.$end_date), max(.$end_date), by=<span>&quot;days&quot;</span>)), by=<span>&quot;end_date&quot;</span>)
  
  rolling_average &lt;- polls_2016 %>%
    group_by(end_date) %>%
    summarise(Clinton = mean(Clinton), Trump = mean(Trump)) %>%
    mutate(Clinton.Margin = Clinton-Trump,
           Clinton.Avg =  rollapply(Clinton.Margin,width=<span>14</span>,
                                    FUN=<span>function</span>(x){mean(x, na.rm=<span>TRUE</span>)},
                                    by=<span>1</span>, partial=<span>TRUE</span>, fill=<span>NA</span>, align=<span>&quot;right&quot;</span>))
}

transformDT &lt;- <span>function</span>(dtin) {
  pollsDT &lt;- copy(dtin) <span>## extra work to protect from reference semantics for benchmark</span>
  pollsDT &lt;- pollsDT[sample_subpopulation %<span>in</span>% c(<span>&quot;Adults&quot;</span>,<span>&quot;Likely Voters&quot;</span>,<span>&quot;Registered Voters&quot;</span>), ]
  pollsDT[, end_date := as.IDate(end_date)]
  pollsDT &lt;- pollsDT[ data.table(end_date = seq(min(pollsDT[,end_date]),
                                                max(pollsDT[,end_date]), by=<span>&quot;days&quot;</span>)), on=<span>&quot;end_date&quot;</span>]
  
  pollsDT &lt;- pollsDT[, .(Clinton=mean(Clinton), Trump=mean(Trump)), by=end_date]
  pollsDT[, Clinton.Margin := Clinton-Trump]
  pollsDT[, Clinton.Avg := rollapply(Clinton.Margin, width=<span>14</span>,
                                     FUN=<span>function</span>(x){mean(x, na.rm=<span>TRUE</span>)},
                                     by=<span>1</span>, partial=<span>TRUE</span>, fill=<span>NA</span>, align=<span>&quot;right&quot;</span>)]
}

transformBS &lt;- <span>function</span>(pollsB) {
  pollsB &lt;- pollsB[pollsB$sample_subpopulation %<span>in</span>% c(<span>&quot;Adults&quot;</span>,<span>&quot;Likely Voters&quot;</span>,<span>&quot;Registered Voters&quot;</span>), ]
  pollsB$end_date &lt;- base::as.Date(pollsB$end_date)
  endDate &lt;- data.frame(end_date = seq.Date(min(pollsB$end_date), max(pollsB$end_date), by=<span>&quot;days&quot;</span>))
  pollsB &lt;- merge(pollsB, endDate, by=<span>&quot;end_date&quot;</span>, all=<span>TRUE</span>)
  
  pollsB &lt;- aggregate(cbind(Clinton, Trump) ~ end_date, data=pollsB, mean, na.action=na.pass)
  pollsB$Clinton.Margin &lt;- pollsB$Clinton - pollsB$Trump
  pollsB$Clinton.Avg &lt;- rollapply(pollsB$Clinton.Margin, width=<span>14</span>,
                                  FUN=<span>function</span>(x){mean(x, na.rm=<span>TRUE</span>)},
                                  by=<span>1</span>, partial=<span>TRUE</span>, fill=<span>NA</span>, align=<span>&quot;right&quot;</span>)
}

res &lt;- microbenchmark(tidy=transformTV(tvin),
                      dt=transformDT(dtin),
                      base=transformBS(bsin))
res</code></pre>
<pre><code>## Unit: milliseconds
##  expr      min       lq     mean   median       uq       max neval
##  tidy 20.68435 22.58603 26.67459 24.56170 27.85844  84.55077   100
##    dt 17.25547 18.88340 21.43256 20.24450 22.26448  41.65252   100
##  base 28.39796 30.93722 34.94262 32.97987 34.98222 109.14005   100</code></pre>
<p>I don’t see so much difference between the tidyverse and <code>data.table</code> as Dirk showed, perhaps because I’ve simplified the script a bit, and removed some redundant parts. Again, base R is the slowest option, but don’t set it aside: it is the shortest one, and it is always there, out of the box!</p>
<p><em>Article originally published in Enchufa2.es: <a rel="nofollow" href="https://www.enchufa2.es/archives/tidyverse-and-data-table-sitting-side-by-side-and-then-base-r-walks-in.html" target="_blank">Tidyverse and data.table, sitting side by side… and then base R walks in</a></em>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.enchufa2.es/archives/tidyverse-and-data-table-sitting-side-by-side-and-then-base-r-walks-in.html"> R – Enchufa2</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/QvebHb-EaU0" height="1"></a></div>