<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/summarizing-big-data-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2017/05/summarizing-big-data-in-r/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Our next <a rel="nofollow" href="http://www.win-vector.com/blog/tag/r-and-big-data/" target="_blank">&quot;R and big data tip&quot;</a> is: summarizing big data.</p>
<p>We always say &quot;if you are not looking at the data, you are not doing science&quot;- and for big data you are very dependent on summaries (as you can’t actually look at everything).</p>
<p>Simple question: is there an easy way to summarize big data in <a rel="nofollow" href="https://www.r-project.org/" target="_blank"><code>R</code></a>?</p>
<p>The answer is: yes, but we suggest you use the <a rel="nofollow" href="https://cran.r-project.org/package=replyr" target="_blank"><code>replyr</code></a> package to do so.</p>
<p><span></span></p>
<p>Let’s set up a trivial example.</p>
<div>
<pre><code><span>suppressPackageStartupMessages</span>(<span>library</span>(<span>&quot;dplyr&quot;</span>))
<span>packageVersion</span>(<span>&quot;dplyr&quot;</span>)</code></pre>
</div>
<pre><code>## [1] '0.5.0'</code></pre>
<div>
<pre><code><span>library</span>(<span>&quot;sparklyr&quot;</span>)
<span>packageVersion</span>(<span>&quot;sparklyr&quot;</span>)</code></pre>
</div>
<pre><code>## [1] '0.5.5'</code></pre>
<div>
<pre><code><span>library</span>(<span>&quot;replyr&quot;</span>)
<span>packageVersion</span>(<span>&quot;replyr&quot;</span>)</code></pre>
</div>
<pre><code>## [1] '0.3.902'</code></pre>
<div>
<pre><code>sc &lt;-<span> </span>sparklyr::<span>spark_connect</span>(<span>version=</span><span>'2.0.2'</span>, 
                              <span>master =</span> <span>&quot;local&quot;</span>)
diris &lt;-<span> </span><span>copy_to</span>(sc, iris, <span>'diris'</span>)</code></pre>
</div>
<p>The usual <code>S3</code>–<code>summary()</code> summarizes the handle, not the data.</p>
<div>
<pre><code><span>summary</span>(diris)</code></pre>
</div>
<pre><code>##     Length Class          Mode
## src 1      src_spark      list
## ops 3      op_base_remote list</code></pre>
<p><code>tibble::glimpse()</code> throws.</p>
<div>
<pre><code><span>packageVersion</span>(<span>&quot;tibble&quot;</span>)</code></pre>
</div>
<pre><code>## [1] '1.3.3'</code></pre>
<div>
<pre><code><span># errors-out</span>
<span>glimpse</span>(diris)</code></pre>
</div>
<pre><code>## Observations: 150
## Variables: 5

## Error in if (width[i] &lt;= max_width[i]) next: missing value where TRUE/FALSE needed</code></pre>
<p><code>broom::glance()</code> throws.</p>
<div>
<pre><code><span>packageVersion</span>(<span>&quot;broom&quot;</span>)</code></pre>
</div>
<pre><code>## [1] '0.4.2'</code></pre>
<div>
<pre><code>broom::<span>glance</span>(diris)</code></pre>
</div>
<pre><code>## Error: glance doesn't know how to deal with data of class tbl_sparktbl_sqltbl_lazytbl</code></pre>
<p><code>replyr_summary()</code> works, and returns results in a <code>data.frame</code>.</p>
<div>
<pre><code><span>replyr_summary</span>(diris) %>%
<span> </span><span>select</span>(-nunique, -index, -nrows)</code></pre>
</div>
<pre><code>##         column     class nna min max     mean        sd lexmin    lexmax
## 1 Sepal_Length   numeric   0 4.3 7.9 5.843333 0.8280661   </code></pre>