<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/replyr-get-a-grip-on-big-data-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2017/03/replyr-get-a-grip-on-big-data-in-r/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a target="_blank" rel="nofollow" href="https://github.com/WinVector/replyr"><code>replyr</code></a> is an <a target="_blank" rel="nofollow" href="https://cran.r-project.org/"><code>R</code></a> package that contains extensions, adaptions, and work-arounds to make remote <code>R</code> <code>dplyr</code> data sources (including big data systems such as <code>Spark</code>) behave more like local data. This allows the analyst to more easily develop and debug procedures that simultaneously work on a variety of data services (in-memory <code>data.frame</code>, <code>SQLite</code>, <code>PostgreSQL</code>, and <code>Spark2</code> currently being the primary supported platforms).</p>
<p><img title="replyrs.png" alt="Replyrs" src="https://i0.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/03/replyrs.png?w=660" border="0" data-recalc-dims="1"><span></span></p>
<p>
<h2>Example</h2>
<p>Suppose we had a large data set hosted on a <code>Spark</code> cluster that we wished to work with using <code>dplyr</code> and <code>sparklyr</code> (for this article we will simulate such using data loaded into <code>Spark</code> from the <code>nycflights13</code> package).</p>
<p>We will work a trivial example: taking a quick peek at your data. The analyst should always be able to and willing to look at the data.</p>
<p>It is easy to look at the top of the data, or any specific set of rows of the data.</p>
<p>Either through <code>print()</code> (which is much safter with <code>tbl_df</code> derived classes, than with base <code>data.frame</code>).</p>
<div>
<pre><code><span>print</span>(flts)</code></pre>
</div>
<pre><code>## Source:   query [3.368e+05 x 19]
## Database: spark connection master=local[4] app=sparklyr local=TRUE
## 
##     year month   day dep_time sched_dep_time dep_delay arr_time
##    &lt;int> &lt;int> &lt;int>    &lt;int>          &lt;int>     &lt;dbl>    &lt;int>
## 1   2013     1     1      517            515         2      830
## 2   2013     1     1      533            529         4      850
## 3   2013     1     1      542            540         2      923
## 4   2013     1     1      544            545        -1     1004
## 5   2013     1     1      554            600        -6      812
## 6   2013     1     1      554            558        -4      740
## 7   2013     1     1      555            600        -5      913
## 8   2013     1     1      557            600        -3      709
## 9   2013     1     1      557            600        -3      838
## 10  2013     1     1      558            600        -2      753
## # ... with 3.368e+05 more rows, and 12 more variables:
## #   sched_arr_time &lt;int>, arr_delay &lt;dbl>, carrier &lt;chr>, flight &lt;int>,
## #   tailnum &lt;chr>, origin &lt;chr>, dest &lt;chr>, air_time &lt;dbl>,
## #   distance &lt;dbl>, hour &lt;dbl>, minute &lt;dbl>, time_hour &lt;dbl></code></pre>
<p>Or with <code>dplyr::glimpse()</code>:</p>
<div>
<pre><code>dplyr::<span>glimpse</span>(flts)</code></pre>
</div>
<pre><code>## Observations: 3.368e+05
## Variables: 19
## $ year           &lt;int> 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2013,...
## $ month          &lt;int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,...
## $ day            &lt;int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,...
## $ dep_time       &lt;int> 517, 533, 542, 544, 554, 554, 555, 557, 557, 55...
## $ sched_dep_time &lt;int> 515, 529, 540, 545, 600, 558, 600, 600, 600, 60...
## $ dep_delay      &lt;dbl> 2, 4, 2, -1, -6, -4, -5, -3, -3, -2, -2, -2, -2...
## $ arr_time       &lt;int> 830, 850, 923, 1004, 812, 740, 913, 709, 838, 7...
## $ sched_arr_time &lt;int> 819, 830, 850, 1022, 837, 728, 854, 723, 846, 7...
## $ arr_delay      &lt;dbl> 11, 20, 33, -18, -25, 12, 19, -14, -8, 8, -2, -...
## $ carrier        &lt;chr> &quot;UA&quot;, &quot;UA&quot;, &quot;AA&quot;, &quot;B6&quot;, &quot;DL&quot;, &quot;UA&quot;, &quot;B6&quot;, &quot;EV&quot;,...
## $ flight         &lt;int> 1545, 1714, 1141, 725, 461, 1696, 507, 5708, 79...
## $ tailnum        &lt;chr> &quot;N14228&quot;, &quot;N24211&quot;, &quot;N619AA&quot;, &quot;N804JB&quot;, &quot;N668DN...
## $ origin         &lt;chr> &quot;EWR&quot;, &quot;LGA&quot;, &quot;JFK&quot;, &quot;JFK&quot;, &quot;LGA&quot;, &quot;EWR&quot;, &quot;EWR&quot;...
## $ dest           &lt;chr> &quot;IAH&quot;, &quot;IAH&quot;, &quot;MIA&quot;, &quot;BQN&quot;, &quot;ATL&quot;, &quot;ORD&quot;, &quot;FLL&quot;...
## $ air_time       &lt;dbl> 227, 227, 160, 183, 116, 150, 158, 53, 140, 138...
## $ distance       &lt;dbl> 1400, 1416, 1089, 1576, 762, 719, 1065, 229, 94...
## $ hour           &lt;dbl> 5, 5, 5, 5, 6, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 5,...
## $ minute         &lt;dbl> 15, 29, 40, 45, 0, 58, 0, 0, 0, 0, 0, 0, 0, 0, ...
## $ time_hour      &lt;dbl> 2013, 2013, 2013, 2013, 2013, 2013, 2013, 2013,...</code></pre>
<p>What <code>replyr</code> adds to the task of &quot;looking at the data&quot; is a rough equivalent to <code>base::summary()</code>: a few per-column statistics.</p>
<div>
<pre><code><span># using dev version of replyr https://github.com/WinVector/replyr</span>
replyr::<span>replyr_summary</span>(flts, 
                       <span>countUniqueNonNum=</span> <span>TRUE</span>)</code></pre>
</div>
<pre><code>##            column index     class  nrows  nna nunique  min  max
## 1            year     1   integer 336776    0      NA 2013 2013
## 2           month     2   integer 336776    0      NA    1   12
## 3             day     3   integer 336776    0      NA    1   31
## 4        dep_time     4   integer 336776 8255      NA    1 2400
## 5  sched_dep_time     5   integer 336776    0      NA  106 2359
## 6       dep_delay     6   numeric 336776 8255      NA  -43 1301
## 7        arr_time     7   integer 336776 8713      NA    1 2400
## 8  sched_arr_time     8   integer 336776    0      NA    1 2359
## 9       arr_delay     9   numeric 336776 9430      NA  -86 1272
## 10        carrier    10 character 336776    0      16   NA   NA
## 11         flight    11   integer 336776    0      NA    1 8500
## 12        tailnum    12 character 336776    0    4044   NA   NA
## 13         origin    13 character 336776    0       3   NA   NA
## 14           dest    14 character 336776    0     105   NA   NA
## 15       air_time    15   numeric 336776 9430      NA   20  695
## 16       distance    16   numeric 336776    0      NA   17 4983
## 17           hour    17   numeric 336776    0      NA    1   23
## 18         minute    18   numeric 336776    0      NA    0   59
## 19      time_hour    19   numeric 336776    0      NA 2013 2013
##           mean          sd lexmin lexmax
## 1  2013.000000    0.000000   &lt;NA>   &lt;NA>
## 2     6.548510    3.414457   &lt;NA>   &lt;NA>
## 3    15.710787    8.768607   &lt;NA>   &lt;NA>
## 4  1349.109947  488.281791   &lt;NA>   &lt;NA>
## 5  1344.254840  467.335756   &lt;NA>   &lt;NA>
## 6    12.639070   40.210061   &lt;NA>   &lt;NA>
## 7  1502.054999  533.264132   &lt;NA>   &lt;NA>
## 8  1536.380220  497.457142   &lt;NA>   &lt;NA>
## 9     6.895377   44.633292   &lt;NA>   &lt;NA>
## 10          NA          NA     9E     YV
## 11 1971.923620 1632.471938   &lt;NA>   &lt;NA>
## 12          NA          NA   &lt;NA>   &lt;NA>
## 13          NA          NA    EWR    LGA
## 14          NA          NA    ABQ    XNA
## 15  150.686460   93.688305   &lt;NA>   &lt;NA>
## 16 1039.912604  733.233033   &lt;NA>   &lt;NA>
## 17   13.180247    4.661316   &lt;NA>   &lt;NA>
## 18   26.230100   19.300846   &lt;NA>   &lt;NA>
## 19 2013.000000    0.000000   &lt;NA>   &lt;NA></code></pre>
<p>As we see, <code>replyr</code> summary returns data in a data frame, and can deal with multiple column types.</p>
<p>Note: the above summary has problems with <code>NA</code> in <code>character</code> columns with <code>Spark</code>, and thus is mis-reporting the <code>NA</code> count in the <code>tailum</code> column. We are working on the issue. That is also one of the advantages of taking your work-arounds from a package: when they do improve you can easily incorporate bring the improvements into your own work by a mere package update.</p>
<p>We could also use <code>dplyr::summarize_each</code> for the task, but it has the minor downside of returning the data in a wide form.</p>
<div>
<pre><code><span># currently throws if tailnum left in column list </span>
vars &lt;-<span> </span><span>setdiff</span>(<span>colnames</span>(flts), <span>'tailnum'</span>)
flts %>%<span> </span><span>summarize_each</span>(<span>funs</span>(min, max, mean, sd), 
                        <span>one_of</span>(vars))</code></pre>
</div>
<pre><code>## Source:   query [1 x 72]
## Database: spark connection master=local[4] app=sparklyr local=TRUE
## 
##   year_min month_min day_min dep_time_min sched_dep_time_min dep_delay_min
##      &lt;int>     &lt;int>   &lt;int>        &lt;int>              &lt;int>         &lt;dbl>
## 1     2013         1       1            1                106           -43
## # ... with 66 more variables: arr_time_min &lt;int>,
## #   sched_arr_time_min &lt;int>, arr_delay_min &lt;dbl>, carrier_min &lt;chr>,
## #   flight_min &lt;int>, origin_min &lt;chr>, dest_min &lt;chr>,
## #   air_time_min &lt;dbl>, distance_min &lt;dbl>, hour_min &lt;dbl>,
## #   minute_min &lt;dbl>, time_hour_min &lt;dbl>, year_max &lt;int>,
## #   month_max &lt;int>, day_max &lt;int>, dep_time_max &lt;int>,
## #   sched_dep_time_max &lt;int>, dep_delay_max &lt;dbl>, arr_time_max &lt;int>,
## #   sched_arr_time_max &lt;int>, arr_delay_max &lt;dbl>, carrier_max &lt;chr>,
## #   flight_max &lt;int>, origin_max &lt;chr>, dest_max &lt;chr>,
## #   air_time_max &lt;dbl>, distance_max &lt;dbl>, hour_max &lt;dbl>,
## #   minute_max &lt;dbl>, time_hour_max &lt;dbl>, year_mean &lt;dbl>,
## #   month_mean &lt;dbl>, day_mean &lt;dbl>, dep_time_mean &lt;dbl>,
## #   sched_dep_time_mean &lt;dbl>, dep_delay_mean &lt;dbl>, arr_time_mean &lt;dbl>,
## #   sched_arr_time_mean &lt;dbl>, arr_delay_mean &lt;dbl>, carrier_mean &lt;dbl>,
## #   flight_mean &lt;dbl>, origin_mean &lt;dbl>, dest_mean &lt;dbl>,
## #   air_time_mean &lt;dbl>, distance_mean &lt;dbl>, hour_mean &lt;dbl>,
## #   minute_mean &lt;dbl>, time_hour_mean &lt;dbl>, year_sd &lt;dbl>,
## #   month_sd &lt;dbl>, day_sd &lt;dbl>, dep_time_sd &lt;dbl>,
## #   sched_dep_time_sd &lt;dbl>, dep_delay_sd &lt;dbl>, arr_time_sd &lt;dbl>,
## #   sched_arr_time_sd &lt;dbl>, arr_delay_sd &lt;dbl>, carrier_sd &lt;dbl>,
## #   flight_sd &lt;dbl>, origin_sd &lt;dbl>, dest_sd &lt;dbl>, air_time_sd &lt;dbl>,
## #   distance_sd &lt;dbl>, hour_sd &lt;dbl>, minute_sd &lt;dbl>, time_hour_sd &lt;dbl></code></pre>
<p>Special code for remote data is needed as none of the obvious &quot;one liner&quot; candidates (<code>base::summary()</code>, or <code>broom:glance()</code>) are not currently (as of March 4, 2017) intended to work with remote data sources.</p>
<div>
<pre><code><span>summary</span>(flts)</code></pre>
</div>
<pre><code>##     Length Class          Mode
## src 1      src_spark      list
## ops 3      op_base_remote list</code></pre>
<div>
<pre><code><span>str</span>(flts)</code></pre>
</div>
<pre><code>## List of 2
##  $ src:List of 1
##   ..$ con:List of 11
##   .. ..$ master       : chr &quot;local[4]&quot;
##   .. ..$ method       : chr &quot;shell&quot;
##   .. ..$ app_name     : chr &quot;sparklyr&quot;
##   ...
</code></pre>
<div>
<pre><code><span>packageVersion</span>(<span>'broom'</span>)</code></pre>
</div>
<pre><code>## [1] '0.4.2'</code></pre>
<div>
<pre><code>broom::<span>glance</span>(flts)</code></pre>
</div>
<pre><code>## Error: glance doesn't know how to deal with data of class tbl_sparktbl_sqltbl_lazytbl</code></pre>
<p>The source for the examples can be found <a target="_blank" rel="nofollow" href="https://github.com/WinVector/Examples/blob/master/replyr/example.Rmd">here</a>.</p>
<h2>Conclusion</h2>
<p><code>replyr_summary</code> is not the only service <code>replyr</code> supplies, <code>replyr</code> includes many more adaptions <a target="_blank" rel="nofollow" href="http://www.win-vector.com/blog/2017/02/the-zero-bug/">including my own version of case-completion</a>.</p>
<p>Roughly <code>replyr</code> is where I collect my adaptions so they don’t infest application code. <code>replyr</code> a way you can use heavy-duty big-data machinery, while keeping you fingers out of the gears.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2017/03/replyr-get-a-grip-on-big-data-in-r/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/KpDLdZIU9QQ"></a></div>