<p>This is a note on debugging <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/magrittr/vignettes/magrittr.html"><code>magrittr</code></a> pipelines in <a target="_blank" rel="nofollow" href="https://cran.r-project.org/"><code>R</code></a> using <a target="_blank" rel="nofollow" href="http://www.win-vector.com/blog/2017/01/using-the-bizarro-pipe-to-debug-magrittr-pipelines-in-r/">Bizarro Pipe</a> and eager assignment.</p>
<p><center><br>
<a target="_blank" rel="nofollow" href="http://sites.harvard.edu/~chsi/markone/language.html"><img title="moth.jpg" alt="Moth" src="https://i0.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/03/moth.jpg?resize=558%2C209" border="0" data-recalc-dims="1"></a><br>
</center><span></span></p>
<p>
<h2>Pipes in R</h2>
<p>The <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/magrittr/vignettes/magrittr.html"><code>magrittr</code></a> <a target="_blank" rel="nofollow" href="https://cran.r-project.org/"><code>R</code></a> package supplies an operator called “pipe” which is written as “<code>%>%</code>“. The pipe operator is partly famous due to its extensive use in <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=dplyr"><code>dplyr</code></a> and use by <code>dplyr</code> users. The pipe operator is roughly described as allowing one to write “<code>sin(5)</code>” as “<code>5 %>% sin</code>“. It is described as being inspired by <code>F#</code>‘s pipe-forward operator “|>” which itself is defined or implemented as:</p>
<pre>
    let (|>) x f = f x
</pre>
<p>The <code>magrittr</code> pipe doesn’t actually perform the above substitution directly. As a consequence “<code>5 %>% sin</code>” is evaluated in a <a target="_blank" rel="nofollow" href="https://github.com/tidyverse/magrittr/issues/141">different environment</a> than “<code>sin(5)</code>” would be (unlike <code>F#</code>‘s “<code>|></code>“), and the actual implementation is <a target="_blank" rel="nofollow" href="https://github.com/tidyverse/magrittr/issues/120">fairly involved</a>.</p>
<p>The environment change is demonstrated below:</p>
<pre>
library(&quot;dplyr&quot;)
f &lt;- function(...) {print(parent.frame())}

f(5)
## &lt;environment: R_GlobalEnv>

5 %>% f
## &lt;environment: 0x1032856a8>
</pre>
<p>Pipes are like any other coding feature: if you code with it you are eventually going to have to debug with it. Exact pipe semantics and implementation details are important when debugging, as one tries to control execution sequence and examine values and environments while debugging.</p>
<h2>A Debugging Example</h2>
<p>Consider the following example taken from the “Chaining” section of <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/dplyr/vignettes/introduction.html">“Introduction to <code>dplyr</code>“</a>.</p>
<pre>
library(&quot;dplyr&quot;)
library(&quot;nycflights13&quot;)

flights %>%
    group_by(year, month, day) %>%
    select(arr_delay, dep_delay) %>%
    summarise(
        arr = mean(arr_delay, na.rm = TRUE),
        dep = mean(dep_delay, na.rm = TRUE)
    ) %>%
    filter(arr > 30 | dep > 30)
<font color="red">## Adding missing grouping variables: `year`, `month`, `day`</font>
## Source: local data frame [49 x 5]
## Groups: year, month [11]
## 
##     year month   day      arr      dep
##    &lt;int> &lt;int> &lt;int>    &lt;dbl>    &lt;dbl>
## 1   2013     1    16 34.24736 24.61287
## 2   2013     1    31 32.60285 28.65836
## ...
</pre>
<p>A beginning <code>dplyr</code> user might wonder at the meaning of the warning “<code><font color="red">Adding missing grouping variables: `year`, `month`, `day`</font></code>“. Similarly, a veteran <code>dplyr</code> user may wonder why we bother with a <code>dplyr::select()</code>, as selection is implied in the following <code>dplyr::summarise()</code>; but this is the example code as we found it.</p>
<h2>Using Bizarro Pipe</h2>
<p>We can run down the cause of the warning quickly by performing the mechanical translation from a <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=magrittr"><code>magrittr</code></a> pipeline to a Bizarro pipeline. This is simply making all the first arguments explicit with “dot” and replacing the operator “<code>%>%</code>” with the Bizarro pipe glyph: “<code>->.;</code>“.</p>
<p>We can re-run the modified code by pasting into <code>R</code>‘s command console and the warning now lands much nearer the cause (even when we paste or execute the entire pipeline at once):</p>
<pre>
flights ->.;
  group_by(., year, month, day) ->.;
  select(., arr_delay, dep_delay) ->.;
<font color="red">## Adding missing grouping variables: `year`, `month`, `day`</font>
  summarise(.,
          arr = mean(arr_delay, na.rm = TRUE),
          dep = mean(dep_delay, na.rm = TRUE)
  ) ->.;
  filter(., arr > 30 | dep > 30)
## Source: local data frame [49 x 5]
## Groups: year, month [11]
## 
##     year month   day      arr      dep
##    &lt;int> &lt;int> &lt;int>    &lt;dbl>    &lt;dbl>
## 1   2013     1    16 34.24736 24.61287
## 2   2013     1    31 32.60285 28.65836
## ...
</pre>
<p>We can now clearly see the warning was issued by <code>dplyr::select()</code> (even though we just pasted in the whole block of commands at once). This means despite <code>help(select)</code> saying “<code>select()</code> keeps only the variables you mention” this example is depending on the (useful) accommodation that <code>dplyr::select()</code> preserves grouping columns in addition to user specified columns (though this accommodation is not made for columns specified in <code>dplyr::arrange()</code>).</p>
<h3>A Caveat</h3>
<p>To capture a value from a Bizarro pipe we must make an assignment at the end of the pipe, not the beginning. The following will not work as it would capture only the value after the first line (“<code>flights ->.;</code>“) and <em>not</em> the value at the end of the pipeline.</p>
<p>One must not write:</p>
<pre><font color="brown">
VARIABLE &lt;- flights ->.;
group_by(., year, month, day)
</font></pre>
<p>To capture pipeline results we must write:</p>
<pre><font color="green">
flights ->.;
group_by(., year, month, day) -> VARIABLE
</font></pre>
<p>I think the right assignment is very readable if you have the discipline to only use pipe operators as line-enders, making assignments the unique lines without pipes. Also, leaving an extra line break after assignments helps with readability.</p>
<h2>Making Things More Eager</h2>
<p>A remaining issue is: Bizarro pipe only made the composition eager. For a data structure with additional lazy semantics (such as <code>dplyr</code>‘s view of a remote <code>SQL</code> system) we would still not have the warning near the cause. </p>
<p>Unfortunately different <code>dplyr</code> backends give different warnings, so we can’t demonstrate the same warning here. We can, however, deliberately introduce an error and show how to localize errors in the presence of lazy eval data structures. In the example below I have misspelled “month” as “moth”. Notice the error is again not seen until printing, long after we finished composing the pipeline.</p>
<pre>
s &lt;- dplyr::src_sqlite(&quot;:memory:&quot;, create = TRUE)                                 
flts &lt;- dplyr::copy_to(s, flights)

flts ->.;
  group_by(., year, moth, day) ->.;
  select(., arr_delay, dep_delay) ->.;
  summarise(.,
          arr = mean(arr_delay, na.rm = TRUE),
          dep = mean(dep_delay, na.rm = TRUE)
          ) ->.;
  filter(., arr > 30 | dep > 30)

## Source:   query [?? x 5]
## Database: sqlite 3.11.1 [:memory:]
## Groups: year, moth

<font color="red">## na.rm not needed in SQL: NULL are always droppedFALSE</font>
<font color="red">## na.rm not needed in SQL: NULL are always droppedFALSE</font>
<font color="red">## Error in rsqlite_send_query(conn@ptr, statement) : no such column: moth</font>
</pre>
<p>We can try to force <code>dplyr</code> into eager evaluation using the eager value landing operator “<code>replyr::`%->%`</code>” (from <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=replyr"><code>replyr</code></a> package) to form the “extra eager” Bizarro glyph: “<code>%->%.;</code>“. </p>
<p><center><br>
<a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=replyr"><img title="replyrs.png" alt="Replyrs" src="https://i1.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/03/replyrs-2.png?w=660" border="0" data-recalc-dims="1"></a><br>
</center></p>
<p>When we re-write the code in terms of the extra eager Bizarro glyph we get the following.</p>
<pre>
install.packages(&quot;replyr&quot;)
library(&quot;replyr&quot;)

flts %->%.;
  group_by(., year, moth, day) %->%.;
<font color="red">## Error in rsqlite_send_query(conn@ptr, statement) : no such column: moth</font>
  select(., arr_delay, dep_delay) %->%.;
  summarise(.,
          arr = mean(arr_delay, na.rm = TRUE),
          dep = mean(dep_delay, na.rm = TRUE)
          ) %->%.;
<font color="red">## na.rm not needed in SQL: NULL are always droppedFALSE</font>
<font color="red">## na.rm not needed in SQL: NULL are always droppedFALSE</font>
  filter(., arr > 30 | dep > 30)
## Source:   query [?? x 5]
## Database: sqlite 3.11.1 [:memory:]
</pre>
<p>Notice we have successfully localized the error.</p>
<h3>Nota Bene</h3>
<p>One thing to be careful with in “dot debugging” is: when a statement such as <code>dplyr::select()</code> errors-out this means<br>
the Bizarro assignment on that line does not occur (normal <code>R</code> exception semantics). Thus “dot” will be still carrying the value from the previous line, and the pasted block of code will continue after the failing line using this older data state found in “dot.” So you may see strange results and additional errors indicated in the pipeline. The debugging advice is: at most the first error message is trustworthy.</p>
<h3>The Trick</h3>
<p>The trick is to train your eyes to to read “<code>->.;</code>” or “<code>%->%.;</code>” as a single atomic or indivisible glyph, and not as a sequence of operators, variables, and separators. I see Bizarro pipe as a kind of strange superhero.</p>
<p><center><br>
<img src="https://i0.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/01/BizarroPipe.png?w=400" data-recalc-dims="1"><br>
</center></p>
<h2>Conclusion</h2>
<p>Pipes are a fun notation, and even the original <code>magrittr</code> package experiments with a number of interesting variations of them. I hope you add Bizarro pipe (which turns out has been available in <code>R</code> all along, without requiring any packages!) and extra eager Bizarro pipe to your debugging workflow.</p>
<p><img title="Pipe2.png" alt="Pipe2" src="https://i1.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/03/Pipe2.png?w=660" border="0" data-recalc-dims="1"></p>
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/78KwbvXdHr8">