<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/dplyr-in-context/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2017/05/dplyr-in-context/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h1>Introduction</h1>
<p>Beginning <a target="_blank" rel="nofollow" href="https://cran.r-project.org/"><code>R</code></a> users often come to the false impression that the popular packages <code>dplyr</code> and <code>tidyr</code> are both all of <code>R</code> and <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Sui_generis"><em>sui generis</em></a> inventions (in that they might be unprecedented and there might no other reasonable way to get the same effects in <code>R</code>). These packages and their conventions are high-value, but they are results of evolution and implement a style of programming that has been available in <code>R</code> for some time. They evolved in a context, and did not burst on the scene fully armored with spear in hand.</p>
<p><center>
<div>
<img title="Birth_Athena_Tegner.jpg" alt="Birth Athena Tegner" src="https://i1.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/05/Birth_Athena_Tegner.jpg?w=660" border="0" data-recalc-dims="1">
<p><a target="_blank" rel="nofollow" href="https://theartstack.com/artist/rudolph-tegner/zeus-giving-birth-athen">Zeus giving birth to Athena, Rudolph Tegner</a></p>
</div>
<p><span></span></p>
<p>
<h1><code>dplyr</code> and <code>tidyr</code></h1>
<p>We will start with a (very) brief outline of the primary capabilities of <code>dplyr</code> and <code>tidyr</code>.</p>
<h2><code>dplyr</code></h2>
<p><code>dplyr</code> embodies the idea that data manipulation should be broken down into a sequence of transformations.</p>
<p>For example: in <code>R</code> if one wishes to add a column to a <code>data.frame</code> it is common to perform an &quot;in-place&quot; calculation as shown below:</p>
<div>
<pre><code>d &lt;-<span> </span><span>data.frame</span>(<span>x=</span><span>c</span>(-<span>1</span>,<span>0</span>,<span>1</span>))
<span>print</span>(d)</code></pre>
</div>
<pre><code>##    x
## 1 -1
## 2  0
## 3  1</code></pre>
<div>
<pre><code>d$absx &lt;-<span> </span><span>abs</span>(d$x)
<span>print</span>(d)</code></pre>
</div>
<pre><code>##    x absx
## 1 -1    1
## 2  0    0
## 3  1    1</code></pre>
<p>This has a couple of disadvantages:</p>
<ul>
<li>The original <code>d</code> has been altered, so re-starting calculations (say after we discover a mistake) can be inconvenient.</li>
<li>We have to keep repeating the name of the <code>data.frame</code> which is not only verbose (which is not that important an issue), it is a chance to write the wrong name and introduce an error.</li>
</ul>
<p>The &quot;<code>dplyr</code>-style&quot; is to write the same code as follows:</p>
<div>
<pre><code><span>suppressPackageStartupMessages</span>(<span>library</span>(<span>&quot;dplyr&quot;</span>))
d &lt;-<span> </span><span>data.frame</span>(<span>x=</span><span>c</span>(-<span>1</span>,<span>0</span>,<span>1</span>))

d %>%<span> </span>
<span> </span><span>mutate</span>(<span>absx =</span> <span>abs</span>(x))</code></pre>
</div>
<pre><code>##    x absx
## 1 -1    1
## 2  0    0
## 3  1    1</code></pre>
<div>
<pre><code><span># confirm our original data frame is unaltered</span>
<span>print</span>(d)</code></pre>
</div>
<pre><code>##    x
## 1 -1
## 2  0
## 3  1</code></pre>
<p>The idea is to break your task into the sequential application of a small number of &quot;standard verbs&quot; to produce your result. The verbs are &quot;pipelined&quot; or sequenced using the <code>magrittr</code> pipe &quot;<code>%>%</code>&quot; which can be thought of <a target="_blank" rel="nofollow" href="http://www.win-vector.com/blog/2017/05/why-to-use-wraprlet/"><em>as if</em></a> the following four statements were to be taken as equivalent:</p>
<ul>
<li><code>f(x)</code></li>
<li><code>x %>% f(.)</code></li>
<li><code>x %>% f()</code></li>
<li><code>x %>% f</code></li>
</ul>
<p>This lets one write a sequence of operations as a left to right pipeline (without explicit nesting of functions or use of numerous intermediate variables). Some discussion can be found <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/magrittr/vignettes/magrittr.html">here</a>.</p>
<p>Primary <code>dplyr</code> verbs include the &quot;single table verbs&quot; from the <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/dplyr/vignettes/introduction.html"><code>dplyr 0.5.0</code> introduction vignette</a>:</p>
<ul>
<li><code>filter()</code> (and <code>slice()</code>)</li>
<li><code>arrange()</code></li>
<li><code>select()</code> (and <code>rename()</code>)</li>
<li><code>distinct()</code></li>
<li><code>mutate()</code> (and <code>transmute()</code>)</li>
<li><code>summarise()</code></li>
<li><code>sample_n()</code> (and <code>sample_frac()</code>)</li>
</ul>
<p>These have high-performance implementations (often in <code>C++</code> thanks to <a target="_blank" rel="nofollow" href="http://www.rcpp.org/">Rcpp</a>) and often have defaults that are safer and better for programming (not <a target="_blank" rel="nofollow" href="http://www.win-vector.com/blog/2015/01/r-bracket-is-a-bit-irregular/">changing types on single column data-frames</a>, not promoting strings to factors, and so-on). Not really discussed in the <code>dplyr 0.5.0</code> introduction are the <code>dplyr::*join()</code> operators which are in fact critical components, but easily explained as standard relational joins (i.e., they are very important implementations, but not novel concepts).</p>
<p>Fairly complex data transforms can be broken down in terms of these verbs (plus some verbs from <code>tidyr</code>):</p>
<p>Take for example a slightly extended version of one of the complex work-flows from <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/dplyr/vignettes/introduction.html"><code>dplyr 0.5.0</code> introduction vignette</a>.</p>
<p>The goal is: plot the distribution of average flight arrive delays and flight departure (all averages grouped by date) for dates where either of these averages is at least 30 minutes. The first step is writing down the goal (as we did above). With that clear, someone familiar with <code>dplyr</code> can write a pipeline or work-flow as below (we have added the <code>gather</code> and <code>arrange</code> steps to extend the example a bit):</p>
<div>
<pre><code><span>library</span>(<span>&quot;nycflights13&quot;</span>)
<span>suppressPackageStartupMessages</span>(<span>library</span>(<span>&quot;dplyr&quot;</span>))
<span>library</span>(<span>&quot;tidyr&quot;</span>)
<span>library</span>(<span>&quot;ggplot2&quot;</span>)

summary1 &lt;-<span> </span>flights %>%
<span> </span><span>group_by</span>(year, month, day) %>%
<span> </span><span>select</span>(arr_delay, dep_delay) %>%
<span> </span><span>summarise</span>(
    <span>arr =</span> <span>mean</span>(arr_delay, <span>na.rm =</span> <span>TRUE</span>),
    <span>dep =</span> <span>mean</span>(dep_delay, <span>na.rm =</span> <span>TRUE</span>)
  ) %>%
<span> </span><span>filter</span>(arr ><span> </span><span>30</span> |<span> </span>dep ><span> </span><span>30</span>) %>%
<span> </span><span>gather</span>(<span>key =</span> delayType, 
         <span>value =</span> delayMinutes, 
         arr, dep) %>%
<span> </span><span>arrange</span>(year, month, day, delayType)</code></pre>
</div>
<pre><code>## Adding missing grouping variables: `year`, `month`, `day`</code></pre>
<div>
<pre><code><span>dim</span>(summary1)</code></pre>
</div>
<pre><code>## [1] 98  5</code></pre>
<div>
<pre><code><span>head</span>(summary1)</code></pre>
</div>
<pre><code>## Source: local data frame [6 x 5]
## Groups: year, month [2]
## 
##    year month   day delayType delayMinutes
##   </code></pre></center>