<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/why-to-use-the-replyr-r-package/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2017/08/why-to-use-the-replyr-r-package/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Recently <a rel="nofollow" href="https://github.com/rstudio/sparklyr/issues/976" target="_blank">I noticed</a> that the <a rel="nofollow" href="https://cran.r-project.org/" target="_blank"><code>R</code></a> package <a rel="nofollow" href="https://cran.r-project.org/package=sparklyr" target="_blank"><code>sparklyr</code></a> had the following odd behavior:</p>
<pre><code>suppressPackageStartupMessages(library(&quot;dplyr&quot;))
library(&quot;sparklyr&quot;)
packageVersion(&quot;dplyr&quot;)
#> [1] '0.7.2.9000'
packageVersion(&quot;sparklyr&quot;)
#> [1] '0.6.2'
packageVersion(&quot;dbplyr&quot;)
#> [1] '1.1.0.9000'

sc &lt;- spark_connect(master = 'local')
#> * Using Spark: 2.1.0
d &lt;- dplyr::copy_to(sc, data.frame(x = 1:2))

dim(d)
#> [1] NA
ncol(d)
#> [1] NA
nrow(d)
#> [1] NA
</code></pre>
<p>This means user code or user analyses that depend on one of <code>dim()</code>, <code>ncol()</code> or <code>nrow()</code> possibly breaks. <code>nrow()</code> used to return something other than <code>NA</code>, so older work may not be reproducible. </p>
<p>In fact: where I actually noticed this was deep in debugging a client project (not in a trivial example, such as above).</p>
<p><center><br>
<img border="0" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/08/tron.jpg?resize=306%2C205" alt="Tron" title="tron.jpg" data-recalc-dims="1"><br>
Tron: fights for the users.<br>
</center></p>
<p>In my opinion: this choice is going to be a great source of surprises, unexpected behavior, and bugs going forward for both <code>sparklyr</code> and <code>dbplyr</code> users.<span></span></p>
<p>
The explanation is: <a rel="nofollow" href="https://github.com/rstudio/sparklyr/pull/890" target="_blank">“<code>tibble::truncate</code> uses <code>nrow()</code>” and “<code>print.tbl_spark</code> is too slow since <code>dbplyr</code> started using <code>tibble</code> as the default way of printing records”</a>.
<p>A little digging gets us to this:</p>
<p><a rel="nofollow" href="https://github.com/tidyverse/tibble/issues/285#issuecomment-319464898" target="_blank"><img border="0" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/08/nrow.png?w=660" alt="Nrow" title="nrow.png" data-recalc-dims="1"></a></p>
<p>The above might make sense <em>if</em> <code>tibble</code> and <code>dbplyr</code> were the only users of <code>dim()</code>, <code>ncol()</code> or <code>nrow()</code>.</p>
<p>Frankly if I call <code>nrow()</code> I expect to learn the number of rows in a table.</p>
<p>The suggestion is for <em>all</em> user code to adapt to use <code>sdf_dim()</code>, <code>sdf_ncol()</code> and <code>sdf_nrow()</code> (instead of <code>tibble</code> adapting). Even if practical (there are already a lot of existing <code>sparklyr</code> analyses), this prohibits the writing of generic <code>dplyr</code> code that works the same over local data, databases, and <code>Spark</code> (by generic code, we mean code that does not check the data source type and adapt). The situation is possibly even worse for non-<code>sparklyr</code> <code>dbplyr</code> users (i.e., databases such as <code>PostgreSQL</code>), as I don’t see any obvious convenient “no please really calculate the number of rows for me” (other than “<code>d %>% tally %>% pull</code>“).</p>
<p>I admit, calling <code>nrow()</code> against an arbitrary <em>query</em> can be expensive. However, I am usually calling <code>nrow()</code> on physical tables (not on arbitrary <code>dplyr</code> queries or pipelines). Physical tables ofter deliberately carry explicit meta-data to make it possible for <code>nrow()</code> to be a cheap operation.</p>
<p>Allowing the user to write reliable generic code that works against many <code>dplyr</code> data sources is the purpose of our <a rel="nofollow" href="https://winvector.github.io/replyr/" target="_blank"><code>replyr</code></a> package. Being able to use the same code many places increases the value of the code (without user facing complexity) and allows one to rehearse procedures in-memory before trying databases or <code>Spark</code>. Below are the functions <code>replyr</code> supplies for examining the size of tables:</p>
<pre><code>library(&quot;replyr&quot;)
packageVersion(&quot;replyr&quot;)
#> [1] '0.5.4'

replyr_hasrows(d)
#> [1] TRUE
replyr_dim(d)
#> [1] 2 1
replyr_ncol(d)
#> [1] 1
replyr_nrow(d)
#> [1] 2

spark_disconnect(sc)</code></pre>
<p>Note: the above is only working properly in the <a rel="nofollow" href="https://github.com/WinVector/replyr" target="_blank">development version of <code>replyr</code></a>, as I only found out about the issue and made the fix recently.</p>
<p><code>replyr_hasrows()</code> was added as I found in many projects the primary use of <code>nrow()</code> was to determine if there was any data in a table. The idea is: user code uses the <code>replyr</code> functions, and the <code>replyr</code> functions deal with the complexities of dealing with different data sources. This also gives us a central place to collect patches and fixes as we run into future problems. <code>replyr</code> accretes functionality as our group runs into different use cases (and we try to put use cases first, prior to other design considerations).</p>
<p>The point of <code>replyr</code> is to provide re-usable work arounds of design choices far away from our influence.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2017/08/why-to-use-the-replyr-r-package/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/dfUGSSHfnXI" height="1"></a></div>