<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/dplyrxdf-0-10-0-beta-prerelease/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2017/08/dplyrxdf-0100-beta-prerelease.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>I’m happy to announce that version 0.10.0 beta of the <a rel="nofollow" href="https://github.com/RevolutionAnalytics/dplyrXdf" target="_blank">dplyrXdf package</a> is now available. You can get it from Github:</p>
<pre>install_github(&quot;RevolutionAnalytics/dplyrXdf&quot;, build_vignettes=FALSE)</pre>
<p>This is a major update to dplyrXdf that adds the following features:</p>
<ul>
<li>Support for the <strong><a rel="nofollow" href="http://dplyr.tidyverse.org/articles/programming.html" target="_blank">tidyeval framework</a></strong> that powers the latest version of dplyr</li>
<li>Works with <strong>Spark and Hadoop</strong> clusters and <strong>files in HDFS</strong></li>
<li>Several utility functions to ease working with files and datasets</li>
<li>Many bugfixes and workarounds for issues with the underlying RevoScaleR functions</li>
</ul>
<p>This (pre-)release of dplyrXdf requires <a rel="nofollow" href="https://docs.microsoft.com/en-us/r-server/what-is-microsoft-r-server" target="_blank">Microsoft R Server</a> or <a rel="nofollow" href="https://docs.microsoft.com/en-us/r-server/r-client/what-is-microsoft-r-client" target="_blank">Client</a> version 8.0 or higher, and dplyr 0.7 or higher. If you’re using R Server, dplyr 0.7 won’t be in the MRAN snapshot that is your default repo, but you can get it from CRAN:</p>
<pre>install.packages(&quot;dplyr&quot;, repos=&quot;https://cloud.r-project.org&quot;)</pre>
<h2>The tidyeval framework</h2>
<p>This completely changes the way in which dplyr handles standard evaluation. Previously, if you wanted to program with dplyr pipelines, you had to use special versions of the verbs ending with &quot;_&quot;: <code>mutate_</code>, <code>select_</code>, and so on. You then provided inputs to these verbs via formulas or strings, in a way that was almost but not quite entirely unlike normal dplyr usage. For example, if you wanted to programmatically carry out a transformation on a given column in a data frame, you did the following:</p>
<div>
<pre>x <span>&lt;-</span> <span>&quot;mpg&quot;</span>
transmute_(mtcars, .dots<span>=</span>list(mpg2<span>=</span>paste0(<span>&quot;2 * &quot;</span>, x)))
<span># mpg2</span>
<span>#1 42.0</span>
<span>#2 42.0</span>
<span>#3 45.6</span>
<span>#4 42.8</span>
<span>#5 37.4</span>
</pre>
</div>
<p>This is prone to errors, since it requires creating a string and then parsing it. Worse, it's also insecure, as you can't always guarantee that the input string won't be malicious.</p>
<p>The tidyeval framework replaces all of that. In dplyr 0.7, you call the same functions for both interactive use and programming. The equivalent of the above in the new framework would be:</p>
<div>
<pre><span># the rlang package implements the tidyeval framework used by dplyr</span>
library(rlang)

x_sym <span>&lt;-</span> sym(x)
transmute(mtcars, mpg2<span>=2</span> <span>*</span> (!!x_sym))
<span># mpg2</span>
<span>#1 42.0</span>
<span>#2 42.0</span>
<span>#3 45.6</span>
<span>#4 42.8</span>
<span>#5 37.4</span>
</pre>
</div>
<p>Here, the <code>!!</code> symbol is a special operator that means to get the column name from the variable to its right. The verbs in dplyr 0.7 understand the special rules for working with quoted symbols introduced in the new framework. The same code also works in dplyrXdf 0.10:</p>
<div>
<pre><span># use the new as_xdf function to import to an Xdf file</span>
mtx <span>&lt;-</span> as_xdf(mtcars)

transmute(mtx, mpg2<span>=2</span> <span>*</span> (<span>!!</span>x_sym)) <span>%>%</span> as.data.frame
<span># mpg2</span>
<span>#1 42.0</span>
<span>#2 42.0</span>
<span>#3 45.6</span>
<span>#4 42.8</span>
<span>#5 37.4</span>
</pre>
</div>
<p>For more information about tidyeval, see the dplyr vignettes on <a rel="nofollow" href="http://dplyr.tidyverse.org/articles/programming.html" target="_blank">programming</a> and <a rel="nofollow" href="http://dplyr.tidyverse.org/articles/compatibility.html" target="_blank">compatibility</a>.</p>
<h2>New features in dplyrXdf</h2>
<h3>Copy, move and delete Xdf files</h3>
<p>The following functions let you manipulate Xdf files <em>as files</em>:</p>
<ul>
<li><code>copy_xdf</code> and <code>move_xdf</code> copy and move an Xdf file, optionally renaming it as well.</li>
<li><code>rename_xdf</code> does a strict rename, ie without changing the file’s location.</li>
<li><code>delete_xdf</code> deletes the Xdf file.</li>
</ul>
<h3>HDFS file transfers</h3>
<p>The following functions let you transfer files and datasets to and from HDFS, for working with a Spark or Hadoop cluster:</p>
<ul>
<li><code>copy_to</code> uploads a dataset (a data frame or data source object) from the native filesystem to HDFS, saving it as an Xdf file.</li>
<li><code>collect</code> and <code>compute</code> do the reverse, downloading an Xdf file from HDFS.</li>
<li><code>hdfs_upload</code> and <code>hdfs_download</code> transfer arbitrary files and directories to and from HDFS.</li>
</ul>
<p>Uploading and downloading works (or should work) both from the edge node and from a remote client. The interface is the same in both cases: no need to remember when to use rxHadoopCopyFromLocal and rxHadoopCopyFromClient. The hdfs_* functions mostly wrap the rxHadoop* functions, but also add extra functionality in some cases (eg vectorised copy/move, test for directory existence, etc).</p>
<h3>HDFS file management</h3>
<p>The following functions are for file management in HDFS, and mirror similar functions in base R for working with the native filesystem:</p>
<ul>
<li><code>hdfs_dir</code> lists files in a HDFS directory, like dir() for the native filesystem.</li>
<li><code>hdfs_dir_exists</code> and <code>hdfs_file_exists</code> test for existence of a directory or file, like dir.exists() and file.exists().</li>
<li><code>hdfs_file_copy</code>, <code>hdfs_file_move</code> and <code>hdfs_file_remove</code> copy, move and delete files in a vectorised fashion, like file.copy(), file.rename() and unlink().</li>
<li><code>hdfs_dir_create</code> and <code>hdfs_dir_remove</code> make and delete directories, like dir.create() and unlink(recursive=TRUE).</li>
<li><code>in_hdfs</code> returns whether a data source is in HDFS or not.</li>
</ul>
<p>As far as possible, the functions avoid reading the data via rxDataStep and so should be more efficient. The only times when rxDataStep is necessary are when importing from a non-Xdf data source, and converting between standard and composite Xdfs.</p>
<h3>Miscellaneous functions</h3>
<ul>
<li><code>as_xdf</code> imports a dataset or data source into an Xdf file, optionally as composite. <code>as_standard_xdf</code> and <code>as_composite_xdf</code> are shortcuts for creating standard and composite Xdfs respectively.</li>
<li><code>is_xdf</code> and <code>is_composite_xdf</code> return whether a data source is a (composite) Xdf.</li>
<li><code>local_exec</code> runs an expression in the local compute context: useful for when you want to work with local Xdf files while connected to a remote cluster.</li>
</ul>
<h2>Wrapup</h2>
<p>For more information, check out the package vignettes in particular &quot;<a rel="nofollow" href="http://htmlpreview.github.io/?https://github.com/RevolutionAnalytics/dplyrXdf/blob/master/inst/doc/dplyrXdf_user.html" target="_blank">Using the dplyrXdf package</a>&quot; and &quot;<a rel="nofollow" href="http://htmlpreview.github.io/?https://github.com/RevolutionAnalytics/dplyrXdf/blob/master/inst/doc/dplyrXdf_hdfs.html" target="_blank">Working with HDFS</a>&quot;.</p>
<p>dplyrXdf 0.10 is tentatively scheduled for a final release at the same time as the next version of Microsoft R Server, or shortly afterwards. In the meantime, please download this and give it a try; if you run into any bugs, or if you have any feedback, you can <a rel="nofollow" href="mailto:hongooi@microsoft.com" target="_blank">email me</a> or <a rel="nofollow" href="https://github.com/RevolutionAnalytics/dplyrXdf/issues" target="_blank">log an issue</a> at the Github repo.</p>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2017/08/dplyrxdf-0100-beta-prerelease.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/ZejevHb-fnk" height="1"></a></div>