<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/managing-intermediate-results-when-using-rsparklyr/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2017/06/managing-intermediate-results-when-using-rsparklyr/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In our latest <a rel="nofollow" href="http://www.win-vector.com/blog/tag/r-and-big-data/" target="_blank">“R and big data”</a> article we show how to manage intermediate results in non-trivial <a rel="nofollow" href="https://spark.apache.org/" target="_blank">Apache Spark</a> workflows using <a rel="nofollow" href="https://www.r-project.org/" target="_blank">R</a>, <a rel="nofollow" href="https://cran.r-project.org/package=sparklyr" target="_blank">sparklyr</a>, <a rel="nofollow" href="https://cran.r-project.org/package=dplyr" target="_blank">dplyr</a>, and <a rel="nofollow" href="https://cran.r-project.org/package=replyr" target="_blank">replyr</a>.</p>
<p><center><br>
<a rel="nofollow" href="https://en.wikipedia.org/wiki/Pont_du_Gard" target="_blank"><img border="0" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/06/NewImage.png?resize=598%2C231" alt="NewImage" title="NewImage.png" data-recalc-dims="1"></a></center><span></span></p>
<p>
<h2>Handle management</h2>
<p>Many <a rel="nofollow" href="https://cran.r-project.org/package=sparklyr" target="_blank"><code>Sparklyr</code></a> tasks involve creation of intermediate or temporary tables. This can be through <code>dplyr::copy_to()</code> and through <code>dplyr::compute()</code>. These handles can represent a reference leak and eat up resources.</p>
<p>To help control handle lifetime the <a rel="nofollow" href="https://cran.r-project.org/package=replyr" target="_blank"><code>replyr</code></a> supplies record-retaining temporary name generators (and uses the same internally).</p>
<p>The actual function is pretty simple:</p>
<div>
<pre><code><span>print</span>(replyr::makeTempNameGenerator)</code></pre>
</div>
<pre><code>## function(prefix,
##                                   suffix= NULL) {
##   force(prefix)
##   if((length(prefix)!=1)||(!is.character(prefix))) {
##     stop(&quot;repyr::makeTempNameGenerator prefix must be a string&quot;)
##   }
##   if(is.null(suffix)) {
##     alphabet &lt;- c(letters, toupper(letters), as.character(0:9))
##     suffix &lt;- paste(base::sample(alphabet, size=20, replace= TRUE),
##                     collapse = '')
##   }
##   count &lt;- 0
##   nameList &lt;- c()
##   function(dumpList=FALSE) {
##     if(dumpList) {
##       v &lt;- nameList
##       nameList &lt;- c()
##       return(v)
##     }
##     nm &lt;- paste(prefix, suffix, sprintf('%010d',count), sep='_')
##     nameList &lt;- c(nameList, nm)
##     count &lt;- count + 1
##     nm
##   }
## }
## </code></pre>