<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/reusable-pipelines-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2018/12/reusable-pipelines-in-r/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Pipelines in <a rel="nofollow" href="https://www.r-project.org/" target="_blank"><code>R</code></a> are popular, the most popular one being <code>magrittr</code> as used by <code>dplyr</code>.</p>
<p>This note will discuss the advanced re-usable piping systems: <a rel="nofollow" href="https://github.com/WinVector/rquery" target="_blank"><code>rquery</code></a>/<a rel="nofollow" href="https://github.com/WinVector/rqdatatable" target="_blank"><code>rqdatatable</code></a> operator trees and <a rel="nofollow" href="https://winvector.github.io/wrapr/articles/Function_Objects.html" target="_blank"><code>wrapr</code> function object pipelines</a>. In each case we have a set of objects designed to extract extra power from the <a rel="nofollow" href="https://journal.r-project.org/archive/2018/RJ-2018-042/index.html" target="_blank"><code>wrapr</code> dot-arrow pipe <code>%.>%</code></a>.</p>
<p><span></span></p>
<p>
<h2>Piping</h2>
<p>Piping is not much more than having a system that lets one treat “<code>x %.>% f(.)</code>” as a near synonym for “<code>f(x)</code>”. For the <code>wrapr</code> dot arrow pipe the semantics are intentionally closer to <code>(x %.>% f(.)) ~ {. &lt;- x; f(.)}</code>.</p>
<p>The pipe notation may be longer, but it avoids nesting and reversed right to left reading for many-stage operations (such as “<code>x %.>% f1(.) %.>% f2(.) %.>% f3(.)</code>” versus “<code>f3(f2(f1(x)))</code>”).</p>
<p>In addition to allowing users to write operations in this notation, most piping systems allow users to save pipelines for later re-use (though some others have issues serializing or saving such pipelines due to entanglement with the defining environment).</p>
<p><code>wrapr</code> and <code>rquery</code>/<code>rqdatatable</code> supply a number of piping tools that are re-usable, serializable, and very powerful (via <code>R</code> <code>S3</code> and <code>S4</code> dispatch features). One of the most compelling features are “function objects” which mans objects can be treated like functions (applied to other objects by pipelines). We will discuss some of these features in the context of <code>rquery</code>/<code>rqdatatable</code> and <code>wrapr</code>.</p>
<h2><code>rquery</code>/<code>rqdatatable</code></h2>
<p>For quite a while the <a rel="nofollow" href="https://github.com/WinVector/rquery" target="_blank"><code>rquery</code></a> and <a rel="nofollow" href="https://github.com/WinVector/rqdatatable" target="_blank"><code>rqdatatable</code></a> packages have supplied a sequence of operators abstraction called an “operator tree” or “operator pipeline”.</p>
<p>These pipelines are (deliberately) fairly strict:</p>
<ul>
<li>They must start with a table description or definition.</li>
<li>Each step must be a table to table transform meeting certain column pre-conditions.</li>
<li>Each step must advertise what columns it makes available or produces, for later condition checking.</li>
</ul>
<p>For a guiding example suppose we want to row-subset some data, get per-group means, and then sort the data by those means.</p>
<div>
<pre><code><span># our example data</span>
d &lt;-<span> </span><span>data.frame</span>(
  <span>group =</span> <span>c</span>(<span>&quot;a&quot;</span>, <span>&quot;a&quot;</span>, <span>&quot;b&quot;</span>, <span>&quot;b&quot;</span>),
  <span>value =</span> <span>c</span>(  <span>1</span>,  <span>2</span>,   <span>2</span>,  <span>-</span><span>10</span>),
  <span>stringsAsFactors =</span> <span>FALSE</span>
)

<span># load our package</span>
<span>library</span>(<span>&quot;rqdatatable&quot;</span>)</code></pre>
</div>
<pre><code>## Loading required package: rquery</code></pre>
<div>
<pre><code><span># build an operator tree</span>
threshold &lt;-<span> </span><span>0.0</span>
ops &lt;-
<span> </span><span># define the data format</span>
<span> </span><span>local_td</span>(d) <span>%.>%</span><span> </span>
<span> </span><span># restrict to rows with value >= threshold</span>
<span> </span><span>select_rows_nse</span>(.,
                  value <span>>=</span><span> </span>threshold) <span>%.>%</span>
<span> </span><span># compute per-group aggegations</span>
<span> </span><span>project_nse</span>(.,
              <span>groupby =</span> <span>&quot;group&quot;</span>,
              <span>mean_value =</span> <span>mean</span>(value)) <span>%.>%</span>
<span> </span><span># sort rows by mean_value decreasing</span>
<span> </span><span>orderby</span>(.,
          <span>cols =</span> <span>&quot;mean_value&quot;</span>,
          <span>reverse =</span> <span>&quot;mean_value&quot;</span>)

<span># show the tree/pipeline</span>
<span>cat</span>(<span>format</span>(ops))</code></pre>
</div>
<pre><code>## table(d; 
##   group,
##   value) %.>%
##  select_rows(.,
##    value >= 0) %.>%
##  project(., mean_value := mean(value),
##   g= group) %.>%
##  orderby(., desc(mean_value))</code></pre>
<p>Of course the purpose of such a pipeline is to be able to apply it to data. This is done simply with the <a rel="nofollow" href="https://journal.r-project.org/archive/2018/RJ-2018-042/index.html" target="_blank"><code>wrapr</code> dot arrow pipe</a>:</p>
<div>
<pre><code>d <span>%.>%</span><span> </span>ops</code></pre>
</div>
<pre><code>##    group mean_value
## 1:     b        2.0
## 2:     a        1.5</code></pre>
<p><code>rquery</code> pipelines are designed to specify and execute data wrangling tasks. An important feature of <code>rquery</code> pipelines is: they are designed for serialization. This means we can save them and also send them to multiple nodes for parallel processing.</p>
<div>
<pre><code><span># save the optree</span>
<span>saveRDS</span>(ops, <span>&quot;rquery_optree.RDS&quot;</span>)

<span># simulate a fresh R session</span>
<span>rm</span>(<span>list=</span><span>setdiff</span>(<span>ls</span>(), <span>&quot;d&quot;</span>))

<span>library</span>(<span>&quot;rqdatatable&quot;</span>)

<span># read the optree back in</span>
ops &lt;-<span> </span><span>readRDS</span>(<span>'rquery_optree.RDS'</span>)

<span># look at it</span>
<span>cat</span>(<span>format</span>(ops))</code></pre>
</div>
<pre><code>## table(d; 
##   group,
##   value) %.>%
##  select_rows(.,
##    value >= 0) %.>%
##  project(., mean_value := mean(value),
##   g= group) %.>%
##  orderby(., desc(mean_value))</code></pre>
<div>
<pre><code><span># use it again</span>
d <span>%.>%</span><span> </span>ops</code></pre>
</div>
<pre><code>##    group mean_value
## 1:     b        2.0
## 2:     a        1.5</code></pre>
<div>
<pre><code><span># clean up</span>
<span>rm</span>(<span>list=</span><span>setdiff</span>(<span>ls</span>(), <span>&quot;d&quot;</span>))</code></pre>
</div>
<p>We can also run <code>rqdatatable</code> operations in “immediate mode”, without pre-defining the pipeline or tables:</p>
<div>
<pre><code>threshold &lt;-<span> </span><span>0.0</span>

d <span>%.>%</span>
<span> </span><span>select_rows_nse</span>(.,
                  value <span>>=</span><span> </span>threshold) <span>%.>%</span>
<span> </span><span>project_nse</span>(.,
              <span>groupby =</span> <span>&quot;group&quot;</span>,
              <span>mean_value =</span> <span>mean</span>(value)) <span>%.>%</span>
<span> </span><span>orderby</span>(.,
          <span>cols =</span> <span>&quot;mean_value&quot;</span>,
          <span>reverse =</span> <span>&quot;mean_value&quot;</span>)</code></pre>
</div>
<pre><code>##    group mean_value
## 1:     b        2.0
## 2:     a        1.5</code></pre>
<h2><code>wrapr</code> function objects</h2>
<p>A natural question is: given we already have <code>rquery</code> pipelines why do we need <a rel="nofollow" href="https://winvector.github.io/wrapr/articles/Function_Objects.html" target="_blank"><code>wrapr</code> function object pipelines</a>? The reason is: <code>rquery</code>/<code>rdatatable</code> pipelines are strict and deliberately restricted to operations that can be hosted both in <code>R</code> (via <code>data.table</code>) or on databases (examples: <code>PostgreSQL</code> and <code>Spark</code>). One might also want a more general pipeline with fewer constraints optimized for working in <code>R</code> directly.</p>
<p>The <code>wrapr</code> “function object” pipelines allow treatment of arbitrary objects as items we can pipe into. Their primary purpose is to <a rel="nofollow" href="https://en.wikipedia.org/wiki/Partial_application" target="_blank">partially apply functions</a> to convert arbitrary objects and functions into single-argument (or unary) functions. This converted form is perfect for pipelining. This, in a sense, lets us treat these objects as functions. The <code>wrapr</code> function object pipeline also has less constraint checking than <code>rquery</code> pipelines, so is more suitable for “black box” steps that do not publish their column use and production details (in fact <code>wrapr</code> function object pipelines work on arbitrary objects, not just <code>data.frame</code>s or tables).</p>
<p>Let’s adapt our above example into a simple <code>wrapr</code> dot arrow pipeline.</p>
<div>
<pre><code><span>library</span>(<span>&quot;wrapr&quot;</span>)

threshold &lt;-<span> </span><span>0</span>

d <span>%.>%</span>
<span> </span>.[.<span>$</span>value <span>>=</span><span> </span>threshold, , drop =<span> </span><span>FALSE</span>] <span>%.>%</span>
<span> </span><span>tapply</span>(.<span>$</span>value, .<span>$</span>group, <span>'mean'</span>) <span>%.>%</span>
<span> </span><span>sort</span>(., <span>decreasing =</span> <span>TRUE</span>)</code></pre>
</div>
<pre><code>##   b   a 
## 2.0 1.5</code></pre>
<p>All we have done is replace the <code>rquery</code> steps with typical base-<code>R</code> commands. As we see the <code>wrapr</code> dot arrow can route data through a sequence of such commands to repeat our example.</p>
<p>Now let’s adapt our above example into a re-usable <code>wrapr</code> function object pipeline.</p>
<div>
<pre><code><span>library</span>(<span>&quot;wrapr&quot;</span>)

threshold &lt;-<span> </span><span>0</span>

pipeline &lt;-
<span> </span><span>srcfn</span>(
    <span>&quot;.[.$value >= threshold, , drop = FALSE]&quot;</span> ) <span>%.>%</span>
<span> </span><span>srcfn</span>(
    <span>&quot;tapply(.$value, .$group, 'mean')&quot;</span> ) <span>%.>%</span>
<span> </span><span>pkgfn</span>(
    <span>&quot;sort&quot;</span>,
    <span>arg_name =</span> <span>&quot;x&quot;</span>,
    <span>args =</span> <span>list</span>(<span>decreasing =</span> <span>TRUE</span>))

<span>cat</span>(<span>format</span>(pipeline))</code></pre>
</div>
<pre><code>## UnaryFnList(
##    SrcFunction{ .[.$value >= threshold, , drop = FALSE] }(.=., ),
##    SrcFunction{ tapply(.$value, .$group, 'mean') }(.=., ),
##    base::sort(x=., decreasing))</code></pre>
<p>We used two <code>wrapr</code> abstractions to capture the steps for re-use (something built in to <code>rquery</code>, and now also supplied by <code>wrapr</code>). The abstractions are:</p>
<ul>
<li><a rel="nofollow" href="https://winvector.github.io/wrapr/reference/srcfn.html" target="_blank"><code>srcfn()</code></a> which wraps arbitrary quoted code as a function object.</li>
<li><a rel="nofollow" href="https://winvector.github.io/wrapr/reference/pkgfn.html" target="_blank"><code>pkgfn()</code></a> which wraps a package qualified function name as a function object (“<code>base</code>” being the default package).</li>
</ul>
<p>This sort of <code>pipeline</code> can be applied to data using pipe notation:</p>
<div>
<pre><code>d <span>%.>%</span><span> </span>pipeline</code></pre>
</div>
<pre><code>##   b   a 
## 2.0 1.5</code></pre>
<p>The above pipeline has one key inconvenience and one key weakness:</p>
<ul>
<li>For the <code>srcfn()</code> steps we had to place the source code in quotes, which defeats any sort of syntax highlighting and auto-completing in our <code>R</code> integrated development environment (IDE).</li>
<li>The above pipeline has a reference to the value of <code>threshold</code> in our current environment, this means the pipeline is not sufficiently self-contained to serialize and share.</li>
</ul>
<p>We can quickly address both of these issues with the <a rel="nofollow" href="https://winvector.github.io/wrapr/reference/qe.html" target="_blank"><code>wrapr::qe()</code></a> (“quote expression”) function. It uses <code>base::substitute()</code> to quote its arguments, and the IDE doesn’t know the contents are quoted and thus can help us with syntax highlighting and auto-completion. Also we are using <a rel="nofollow" href="http://www.win-vector.com/blog/2018/09/parameterizing-with-bquote/" target="_blank"><code>base::bquote()</code> .()-style</a> escaping to bind in the value of <code>threshold</code>.</p>
<div>
<pre><code>pipeline &lt;-
<span> </span><span>srcfn</span>(
    <span>qe</span>( .[.<span>$</span>value <span>>=</span><span> </span>.(threshold), , <span>drop =</span> <span>FALSE</span>] )) <span>%.>%</span>
<span> </span><span>srcfn</span>(
    <span>qe</span>( <span>tapply</span>(.<span>$</span>value, .<span>$</span>group, <span>'mean'</span>) ))  <span>%.>%</span>
<span> </span><span>pkgfn</span>(
    <span>&quot;sort&quot;</span>,
    <span>arg_name =</span> <span>&quot;x&quot;</span>,
    <span>args =</span> <span>list</span>(<span>decreasing =</span> <span>TRUE</span>))

<span>cat</span>(<span>format</span>(pipeline))</code></pre>
</div>
<pre><code>## UnaryFnList(
##    SrcFunction{ .[.$value >= 0, , drop = FALSE] }(.=., ),
##    SrcFunction{ tapply(.$value, .$group, &quot;mean&quot;) }(.=., ),
##    base::sort(x=., decreasing))</code></pre>
<div>
<pre><code>d <span>%.>%</span><span> </span>pipeline</code></pre>
</div>
<pre><code>##   b   a 
## 2.0 1.5</code></pre>
<p>Notice this pipeline works as before, but no longer refers to the external value <code>threshold</code>. This pipeline can be saved and shared.</p>
<p>Another recommended way to bind in values is with the <code>args</code>-argument, which is a named list of values that are expected to be available with a <code>srcfn()</code> is evaluated, or additional named arguments that will be applied to a <code>pkgfn()</code>.</p>
<p>In this notation the pipeline is written as follows.</p>
<div>
<pre><code>pipeline &lt;-
<span> </span><span>srcfn</span>(
    <span>qe</span>( .[.<span>$</span>value <span>>=</span><span> </span>threshold, , <span>drop =</span> <span>FALSE</span>] ),
    <span>args =</span> <span>list</span>(<span>'threshold'</span> =<span> </span>threshold)) <span>%.>%</span>
<span> </span><span>srcfn</span>(
    <span>qe</span>( <span>tapply</span>(.<span>$</span>value, .<span>$</span>group, <span>'mean'</span>) ))  <span>%.>%</span>
<span> </span><span>pkgfn</span>(
    <span>&quot;sort&quot;</span>,
    <span>arg_name =</span> <span>&quot;x&quot;</span>,
    <span>args =</span> <span>list</span>(<span>decreasing =</span> <span>TRUE</span>))

<span>cat</span>(<span>format</span>(pipeline))</code></pre>
</div>
<pre><code>## UnaryFnList(
##    SrcFunction{ .[.$value >= threshold, , drop = FALSE] }(.=., threshold),
##    SrcFunction{ tapply(.$value, .$group, &quot;mean&quot;) }(.=., ),
##    base::sort(x=., decreasing))</code></pre>
<div>
<pre><code>d <span>%.>%</span><span> </span>pipeline</code></pre>
</div>
<pre><code>##   b   a 
## 2.0 1.5</code></pre>
<p>We can save this pipeline.</p>
<div>
<pre><code><span>saveRDS</span>(pipeline, <span>&quot;wrapr_pipeline.RDS&quot;</span>)</code></pre>
</div>
<p>And simulate using it in a fresh environment (i.e. simulate sharing it).</p>
<div>
<pre><code><span># simulate a fresh environment</span>
<span>rm</span>(<span>list =</span> <span>setdiff</span>(<span>ls</span>(), <span>&quot;d&quot;</span>))

<span>library</span>(<span>&quot;wrapr&quot;</span>)

pipeline &lt;-<span> </span><span>readRDS</span>(<span>'wrapr_pipeline.RDS'</span>)

<span>cat</span>(<span>format</span>(pipeline))</code></pre>
</div>
<pre><code>## UnaryFnList(
##    SrcFunction{ .[.$value >= threshold, , drop = FALSE] }(.=., threshold),
##    SrcFunction{ tapply(.$value, .$group, &quot;mean&quot;) }(.=., ),
##    base::sort(x=., decreasing))</code></pre>
<div>
<pre><code>d <span>%.>%</span><span> </span>pipeline</code></pre>
</div>
<pre><code>##   b   a 
## 2.0 1.5</code></pre>
<h1>Conclusion</h1>
<p>And that is some of the power of <code>wrapr</code> piping, <code>rquery</code>/<code>rqdatatable</code>, and <code>wrapr</code> function objects. Essentially <code>wrapr</code> function objects are a reference application of the <code>S3</code>/<code>S4</code> piping abilities discussed in the <a rel="nofollow" href="https://journal.r-project.org/archive/2018/RJ-2018-042/index.html" target="_blank"><code>wrapr</code> pipe formal article</a>.</p>
<p>The technique is very convenient when each of the steps is a substantial (such as non-trivial data preparation and model application steps).</p>
<p>The above techniques can make reproducing and sharing methods much easier.</p>
<p>We have some more examples of the technique <a rel="nofollow" href="http://www.win-vector.com/blog/2018/12/sharing-modeling-pipelines-in-r/" target="_blank">here</a> and <a rel="nofollow" href="https://winvector.github.io/wrapr/articles/Function_Objects.html" target="_blank">here</a>.</p>
<div>
<pre><code><span># clean up after example</span>
<span>unlink</span>(<span>&quot;rquery_optree.RDS&quot;</span>)
<span>unlink</span>(<span>&quot;wrapr_pipeline.RDS&quot;</span>)</code></pre>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2018/12/reusable-pipelines-in-r/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/FZOK9FNUxos" height="1"></a></div>