<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/managing-spark-data-handles-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2017/05/managing-spark-data-handles-in-r/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>When working with big data with <a rel="nofollow" href="https://cran.r-project.org/" target="_blank"><code>R</code></a> (say, <a rel="nofollow" href="https://github.com/WinVector/BigDataRStrata2017" target="_blank">using <code>Spark</code> and <code>sparklyr</code></a>) we have found it very convenient to keep data handles in a neat list or <code>data_frame</code>.</p>
<p><center><br>
<a rel="nofollow" href="https://www.flickr.com/photos/58915463@N05/5465544053" target="_blank"><img border="0" alt="5465544053 8b626a09c8 b" title="5465544053_8b626a09c8_b.jpg" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/04/5465544053_8b626a09c8_b.jpg?resize=512%2C384" data-recalc-dims="1"></a><br>
</center></p>
<p>Please read on for our handy hints on keeping your data handles neat.<span></span></p>
<p>When using <code>R</code> to work over a big data system (such as <code>Spark</code>) much of your work is over &quot;data handles&quot; and not actual data (data handles are objects that control access to remote data).</p>
<p>Data handles are a lot like sockets or file-handles in that they can not be safely serialized and restored (i.e., you can not save them into a <code>.RDS</code> file and then restore them into another session). This means when you are starting or re-starting a project you must &quot;ready&quot; all of your data references. Your projects will be much easier to manage and document if you load your references using the methods we show below.</p>
<p>Let’s set-up our example <code>Spark</code> cluster:</p>
<div>
<pre><code><span>library</span>(<span>&quot;sparklyr&quot;</span>)
<span>#packageVersion('sparklyr')</span>
<span>suppressPackageStartupMessages</span>(<span>library</span>(<span>&quot;dplyr&quot;</span>))
<span>#packageVersion('dplyr')</span>
<span>suppressPackageStartupMessages</span>(<span>library</span>(<span>&quot;tidyr&quot;</span>))

<span># Please see the following video for installation help</span>
<span># https://youtu.be/qnINvPqcRvE</span>
<span># spark_install(version = &quot;2.0.2&quot;)</span>

<span># set up a local &quot;practice&quot; Spark instance</span>
sc &lt;-<span> </span><span>spark_connect</span>(<span>master =</span> <span>&quot;local&quot;</span>,
                    <span>version =</span> <span>&quot;2.0.2&quot;</span>)
<span>#print(sc)</span></code></pre>
</div>
<p>Data is much easier to manage than code, and much easier to compute over. So the more information you can keep as pure data the better off you will be. In this case we are loading the chosen names and paths of <code>parquet</code> data we wish to work with from an external file that is easy for the user to edit.</p>
<div>
<pre><code><span># Read user's specification of files and paths.</span>
userSpecification &lt;-<span> </span><span>read.csv</span>(<span>'tableCollection.csv'</span>,
                             <span>header =</span> <span>TRUE</span>,
                 <span>strip.white =</span> <span>TRUE</span>,
                 <span>stringsAsFactors =</span> <span>FALSE</span>)
<span>print</span>(userSpecification)</code></pre>
</div>
<pre><code>##   tableName tablePath
## 1   data_01   data_01
## 2   data_02   data_02
## 3   data_03   data_03</code></pre>
<p>We can now read these <code>parquet</code> files (usually stored in <code>Hadoop</code>) into our <code>Spark</code> environment as follows.</p>
<div>
<pre><code>readParquets &lt;-<span> </span>function(userSpecification) {
  userSpecification &lt;-<span> </span><span>as_data_frame</span>(userSpecification)
  userSpecification$handle &lt;-<span> </span><span>lapply</span>(
    <span>seq_len</span>(<span>nrow</span>(userSpecification)),
    function(i) {
      <span>spark_read_parquet</span>(sc, 
                         <span>name =</span> userSpecification$tableName[[i]], 
                         <span>path =</span> userSpecification$tablePath[[i]])
    }
  )
  userSpecification
}

tableCollection &lt;-<span> </span><span>readParquets</span>(userSpecification)
<span>print</span>(tableCollection)</code></pre>
</div>
<pre><code>## # A tibble: 3 x 3
##   tableName tablePath          handle
##       </code></pre>