<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/dplyrxdf-0-90-now-available/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2016/12/dplyrxdf-090-now-available.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p><em>by Hong Ooi, Sr. Data Scientist, Microsoft</em></p>
<p>Version 0.90 of the <a target="_blank" rel="nofollow" href="https://github.com/RevolutionAnalytics/dplyrXdf">dplyrXdf package</a> has just been released. dplyrXdf is a package that brings <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=dplyr">dplyr</a> pipelines and data transformation verbs to <a target="_blank" rel="nofollow" href="https://msdn.microsoft.com/en-us/microsoft-r/">Microsoft R Server</a>’s xdf files. This version includes several changes, mostly to address performance and efficiency concerns, which I’ll detail these below.</p>
<div>
<h3>The .outFile argument</h3>
<p>All dplyrXdf verbs now support a special argument <code>.outFile</code>, which determines how the output data is handled. If you don’t specify a value for this argument, the data will be saved to a <code>tbl_xdf</code> which will be managed by dplyrXdf. This supports the default behaviour, whereby data files are automatically created and deleted inside a pipeline. There are two other options for <code>.outFile</code>:</p>
<ul>
<li>
<p>If you specify <code>.outFile = NULL</code>, the data will be returned in memory as a data frame.</p>
</li>
<li>
<p>If <code>.outFile</code> is a character string giving a file name, the data will be saved to an xdf file at that location, and a persistent xdf data source will be returned.</p>
</li>
</ul>
<p>This should improve the efficiency of pipelines with large datasets, by reducing the amount of I/O. Previously, to save the output of a pipeline, you had to call the <code>persist</code> verb at the end:</p>
<div>
<pre>xdf %>% filter(<span>...</span>) %>% mutate(<span>...</span>) %>% persist(<span>&quot;final/output.xdf&quot;</span>)
</pre>
</div>
<p>In this example, <code>mutate</code> would save a temporary xdf file in dplyrXdf’s working directory, and <code>persist</code> would then copy that file to the final output location. Now, you can save the output directly to the final location as follows:</p>
<div>
<pre>xdf %>% filter(<span>...</span>) %>% mutate(<span>...</span>, .outFile=<span>&quot;final/output.xdf&quot;</span>)
</pre>
</div>
<p>This omits a redundant file save and copy, thus speeding things up.</p>
<p>The <code>persist</code> verb remains available, for situations where you have already run a pipeline and want to save its output after the fact.</p>
</div>
<div>
<h3>Setting the dplyrXdf working directory</h3>
<p>By default, dplyrXdf will save the data files it creates into the R working directory. On some systems, this may be located on a drive or filesystem that is relatively small; this is rarely an issue with open source R, but can be problematic when working with large xdf files. You can now change the location of the xdf tbl directory with the <code>setXdfTblDir</code> function:</p>
</div>
<div>
<pre><span># set the tbl directory to a network drive (on Windows)</span>
setXdfTblDir(<span>&quot;n:/Rtemp&quot;</span>)
</pre>
</div>
<p>Similarly, you can view the location of the current xdf tbl directory with <code>getXdfTblDir</code>.</p>
<p>For best performance, you should avoid setting the xdf tbl directory to a remote location/network drive unless you have a fast network connection.</p>
<div>
<h3>Extraction operators</h3>
<p>Sometimes it’s useful to be able to extract variables from an Xdf file. With a data frame, you can do this with the <code>$</code> and <code>[[</code> operators: for example <code>iris$Species</code> and <code>iris[[&quot;Species&quot;]]</code> both return the Species column (as a vector) from the iris dataset. This update to dplyrXdf implements the same functionality for Xdf files:</p>
<div>
<pre>sampDir &lt;- system.file(<span>&quot;sampleData&quot;</span>, package=<span>&quot;RevoScaleR&quot;</span>)
airline &lt;- RxXdfData(file.path(sampDir, <span>&quot;AirlineDemoSmall.xdf&quot;</span>))
ArrDelay &lt;- airline$ArrDelay

head(ArrDelay)
<span>## [1] 6 -8 -2 1 -2 -14</span>
</pre>
</div>
<p>By default, the entire column is returned, so you should be careful using these operators when you have very large Xdf files.</p>
</div>
<div>
<h3>The subset verb</h3>
<p>In dplyr, subsetting data is handled by two verbs: <code>filter</code> for subsetting by rows, and <code>select</code> for subsetting by columns. This is fine for data frames, where everything runs in memory; and for SQL databases, where the hard work is done by the database. For Xdf files, however, this is suboptimal, as each verb translates into a separate I/O step where the data is read from disk, subsetted, then written out again. This can waste a lot of time with large datasets.</p>
<p>You can get around this by using the <code>.rxArgs</code> argument in a verb to pass commands directly to the underlying RevoScaleR functions. For example, <code>filter(xdf, .rxArgs=list(varsToKeep=*)))</code> would subset by rows, and simultaneously use the <code>varsToKeep</code> parameter to tell <code>rxDataStep</code> to subset by columns. But this is inelegant. It would be better if there was a verb that could natively subset in both dimensions, without having to rely on workarounds.</p>
<p>As it turns out, base R has a subset generic which (as the name says) performs subsetting on both rows and columns. You’ve probably used it with data frames:</p>
<div>
<pre>subset(iris, Species == <span>&quot;setosa&quot;</span>, c(Sepal.Length, Sepal.Width))
<span>## Source: local data frame [50 x 2]</span>
<span>## </span>
<span>## Sepal.Length Sepal.Width</span>
<span>## (dbl) (dbl)</span>
<span>## 1 5.1 3.5</span>
<span>## 2 4.9 3.0</span>
<span>## 3 4.7 3.2</span>
<span>## 4 4.6 3.1</span>
<span>## 5 5.0 3.6</span>
<span>## 6 5.4 3.9</span>
<span>## .. ... ...</span>
</pre>
</div>
<p>Here, the first argument to subset specifies the rows, and the second argument the columns to return. The subset method for Xdf files works along the same lines:</p>
<pre><code><span>airSubset</span> <span>&lt;-</span> <span>subset</span><span>(</span><span>airline</span>, <span>DayOfWeek</span> <span>==</span> <span>&quot;Monday&quot;</span>, <span>c</span><span>(</span><span>ArrDelay</span>, <span>CRSDepTime</span><span>)</span><span>)</span>
<span>head</span><span>(</span><span>airSubset</span><span>)</span></code></pre>
<div>
<pre>airSubset &lt;- subset(airline, DayOfWeek == <span>&quot;Monday&quot;</span>, c(ArrDelay, CRSDepTime))
head(airSubset)
<span>## Source: local data frame [6 x 2]</span>
<span>## </span>
<span>## ArrDelay CRSDepTime</span>
<span>## (int) (dbl)</span>
<span>## 1 6 9.666666</span>
<span>## 2 -8 19.916666</span>
<span>## 3 -2 13.750000</span>
<span>## 4 1 11.750000</span>
<span>## 5 -2 6.416667</span>
<span>## 6 -14 13.833333</span>
</pre>
</div>
<p>You can also use the same helper functions to choose columns as you would with select:</p>
<div>
<pre>airSubset2 &lt;- subset(airline, , starts_with(<span>&quot;A&quot;</span>))
names(airSubset2)
<span>## [1] &quot;ArrDelay&quot;</span>
</pre>
</div>
</div>
<div>
<h3>Other changes</h3>
<p>In addition to the above, version 0.90 includes the following changes:</p>
<ul>
<li>
<p>The <code>persist</code> verb now uses the base R functions <code>file.copy</code> and <code>file.rename</code> to copy/move a file, which should improve performance considerably on large datasets.</p>
</li>
<li>
<p>The code for two-table verbs has been extensively rewritten, and should be much more reliable than before.</p>
</li>
<li>
<p>The documentation, including the vignettes, has been significantly revised.</p>
</li>
<li>
<p>Unit testing infrastructure has been added, utilising the <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=testthat">testthat</a> package.</p>
</li>
<li>
<p>Several bugs have been fixed, some found with the aid of the aforementioned unit testing.</p>
</li>
</ul>
<p>The latest version of the dplyrXdf package is available on Github at the link below.</p>
<p>Github: <a target="_blank" rel="nofollow" href="https://github.com/RevolutionAnalytics/dplyrXdf">dplyrXdf</a></p>
</div>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2016/12/dplyrxdf-090-now-available.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/ZD61bvpQK5Y"></a></div>