<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/announcing-dplyrxdf-1-0/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2017/10/announcing-dplyrxdf-10.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>I’m delighted to announce the release of <a rel="nofollow" href="https://github.com/RevolutionAnalytics/dplyrXdf" target="_blank">version 1.0.0 of the dplyrXdf package</a>. dplyrXdf began as a simple (relatively speaking) backend to dplyr for Microsoft Machine Learning Server/Microsoft R Server’s Xdf file format, but has now become a broader suite of tools to ease working with Xdf files.</p>
<p>This update to dplyrXdf brings the following new features:</p>
<ul>
<li>Support for the new <a rel="nofollow" href="http://rlang.tidyverse.org/" target="_blank">tidyeval framework</a> that powers the current release of dplyr</li>
<li>Support for <strong>Spark</strong> and <strong>Hadoop</strong> clusters, including integration with the sparklyr package to process Hive tables in Spark</li>
<li>Integration with dplyr to process <strong>SQL Server tables in-database</strong></li>
<li>Simplified handling of <strong>parallel processing</strong> for grouped data</li>
<li>Several <strong>utility functions</strong> for Xdf and file management</li>
<li>Workarounds for various glitches and unexpected behaviour in MRS and dplyr</li>
</ul>
<div>
<h2>Spark, Hadoop and HDFS</h2>
<p>New in version 1.0.0 of dplyrXdf is support for Xdf files and datasets stored in HDFS in a Hadoop or Spark cluster. Most verbs and pipelines behave the same way, whether the computations are taking place in your R session itself, or in-cluster (except that they should be much more scalable in the latter case). Similarly, dplyrXdf can handle both the scenarios where your R session is taking place on the cluster edge node, or on a remote client.</p>
<p>For example, here is some sample code where we extract a table from Hive, then create a pipeline to process it in the cluster:</p>
<div>
<pre>rxSparkConnect()
sampleHiv <span>&lt;-</span> RxHiveData(table<span>=</span><span>&quot;hivesampletable&quot;</span>)

<span># this will create the composite Xdf 'samplehivetable'</span>
sampleXdf <span>&lt;-</span> as_xdf(sampleHiv)

sampleXdf <span>%>%</span>
    filter(deviceplatform <span>==</span> <span>&quot;Android&quot;</span>) <span>%>%</span>
    group_by(devicemake) <span>%>%</span>
    summarise(n<span>=</span>n()) <span>%>%</span>
    arrange(desc(n)) <span>%>%</span>
    head()
<span>#> devicemake n</span>
<span>#> 1 Samsung 16244</span>
<span>#> 2 LG 7950</span>
<span>#> 3 HTC 2242</span>
<span>#> 4 Unknown 2133</span>
<span>#> 5 Motorola 1524</span>
</pre>
</div>
<p>If you are logged into the edge node, dplyrXdf also has the ability to call sparklyr to process Hive tables in Spark. This can be more efficient than converting the data to Xdf format, since less I/O is involved. To run the above pipeline with sparklyr, we simply omit the step of creating an Xdf file:</p>
<div>
<pre>sampleHiv <span>%>%</span>
    filter(deviceplatform <span>==</span> <span>&quot;Android&quot;</span>) <span>%>%</span>
    group_by(devicemake) <span>%>%</span>
    summarise(n<span>=</span>n()) <span>%>%</span>
    arrange(desc(n))
<span>#> # Source: lazy query [?? x 2]</span>
<span>#> # Database: spark_connection</span>
<span>#> # Ordered by: desc(n)</span>
<span>#> devicemake n</span>
<span>#> </span></pre></div></div></div>