<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/a-subtle-flaw-in-some-popular-r-nse-interfaces/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2018/09/a-subtle-flaw-in-some-popular-r-nse-interfaces/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>It is no great secret: I like value oriented interfaces that preserve <a rel="nofollow" href="https://en.wikipedia.org/wiki/Referential_transparency" target="_blank">referential transparency</a>. It is the side of the public debate I take in <a rel="nofollow" href="https://www.r-project.org/" target="_blank"><code>R</code></a> programming.</p>
<blockquote>
<p>
&quot;One of the most useful properties of expressions is that called by Quine referential transparency. In essence this means that if we wish to find the value of an expression which contains a sub-expression, the only thing we need to know about the sub-expression is its value.&quot;</p>
<p><a rel="nofollow" href="https://www.itu.dk/courses/BPRD/E2009/fundamental-1967.pdf" target="_blank">Christopher Strachey, &quot;Fundamental Concepts in Programming Languages&quot;, Higher-Order and Symbolic Computation, 13, 1149, 2000, Kluwer Academic Publishers (lecture notes written by Christopher Strachey for the International Summer School in Computer Programming at Copenhagen in August, 1967).</a>
</p>
</blockquote>
<p>Please read on for discussion of a subtle bug shared by a few popular non-standard evaluation interfaces.</p>
<p><span></span></p>
<p>Most of my work on <a rel="nofollow" href="http://developer.r-project.org/nonstandard-eval.pdf" target="_blank">non-standard-evaluation</a> (NSE or code capturing interfaces, alternatives to value oriented or referentially transparent interfaces) is to help eliminate and contain use of NSE. For instance <a rel="nofollow" href="https://winvector.github.io/wrapr/reference/let.html" target="_blank"><code>wrapr::let()</code></a> is designed to adapt existing NSE interfaces into standard value oriented interfaces, not to create new NSE interfaces. I pretty much feel composing one NSE interface into another NSE interface is cleaning up one mess by adding another mess. If <code>wrapr::let()</code> fails to clean up a mess: it can be evidence of a limitation of <code>wrapr::let()</code>, evidence of user error, or evidence the initial mess was already a problem.</p>
<p>Let’s take a look at a standard interface. For example we can use base-<code>R</code> to select a column from a <code>data.frame</code> by name (technically we are building a new <code>data.frame</code> with the selected column(s)). The clarity of the value oriented notation is most valuable when exposed to potentially unclear code (we most value help when we most need help). For example: suppose the user is selecting the column named <code>x</code>, and their selection is in a poorly-named variable called <code>y</code> (technically values are not &quot;in&quot; variables, but there are environments associating values with names).</p>
<div>
<pre><code>y &lt;-<span> &quot;x&quot;</span>
d &lt;-<span> </span><span>data.frame</span>(<span>x =</span> <span>1</span>, <span>y =</span> <span>2</span>)

d[, y, drop=<span> </span><span>FALSE</span>] <span># returns a data.frame</span></code></pre>
</div>
<pre><code>##   x
## 1 1</code></pre>
<div>
<pre><code>d[[y]]  <span># returns a column</span></code></pre>
</div>
<pre><code>## [1] 1</code></pre>
<p>And if the column we are looking for (&quot;<code>x</code>&quot;) is not present we still get reliable and easily predictable results.</p>
<div>
<pre><code>y &lt;-<span> &quot;x&quot;</span>
d &lt;-<span> </span><span>data.frame</span>(<span>y =</span> <span>2</span>)

d[, y, drop =<span> </span><span>FALSE</span>] <span># exception, signalling no such column</span></code></pre>
</div>
<pre><code>## Error in `[.data.frame`(d, , y, drop = FALSE): undefined columns selected</code></pre>
<div>
<pre><code>d[[y]]  <span># NULL, signalling no such column</span></code></pre>
</div>
<pre><code>## NULL</code></pre>
<p>Beyond the uninformative choice of variable name there was nothing confusing or dangerous about the above code.</p>
<p>Now let’s look at a non-standard or code capturing interface: &quot;<code>$</code>&quot;.</p>
<div>
<pre><code>y &lt;-<span> &quot;x&quot;</span>
d &lt;-<span> </span><span>data.frame</span>(<span>x =</span> <span>1</span>, <span>y =</span> <span>2</span>)

d<span>$</span>y</code></pre>
</div>
<pre><code>## [1] 2</code></pre>
<div>
<pre><code>d &lt;-<span> </span><span>data.frame</span>(<span>x =</span> <span>1</span>)
d<span>$</span>y</code></pre>
</div>
<pre><code>## NULL</code></pre>
<p>Notice both of these operations used the name &quot;<code>y</code>&quot; (captured from user code) to try and retrieve a column named &quot;<code>y</code>&quot;, regardless of what the value referred to by the variable &quot;<code>y</code>&quot; is. In these cases the interface is unambiguously taking the name from the user code. This non-standard interface is convenient and reliable, and has an obvious standard analogue in &quot;<code>[[]]</code>&quot;, so it does not present problems.</p>
<p>Now let’s look at a more problematic non-standard interface <code>base::subset()</code></p>
<div>
<pre><code>y &lt;-<span> &quot;x&quot;</span>
d &lt;-<span> </span><span>data.frame</span>(<span>x =</span> <span>1</span>, <span>y =</span> <span>2</span>)

<span>subset</span>(d, <span>select =</span> y)</code></pre>
</div>
<pre><code>##   y
## 1 2</code></pre>
<p><code>subset()</code> appears to be using the name found in the code to look-up columns. This is not quite the case: from <code>help(subset)</code> we see the <code>select</code> argument is defined as &quot;expression, indicating columns to select from a data frame.&quot; What may not be obvious from a cursory read of the documentation is these expressions are evaluated in a new environment that maps the <code>data.frame</code> column names to integers (a clever way to replace column names with column indices) but in an enclosure by the current execution environment (via a call to <code>parent.frame()</code>). This subtlety means that any names that are not column names of the <code>data.frame</code> are evaluated as expressions in the caller’s environment. This in turn means they are replaced by values from this environment, which leads to very confusing outcomes such as the following.</p>
<div>
<pre><code>y &lt;-<span> &quot;x&quot;</span>
d &lt;-<span> </span><span>data.frame</span>(<span>x =</span> <span>1</span>)

<span>subset</span>(d, <span>select =</span> y)</code></pre>
</div>
<pre><code>##   x
## 1 1</code></pre>
<p>Notice the column &quot;<code>x</code>&quot; was returned, (not a <code>NULL</code>). Essentially each specified column name gets two chances to resolve to a <code>data.frame</code> column name or index: first from an environment mapping <code>data.frame</code> names to indices, and second from the calling environment.</p>
<p>The above issue would be fixed if the expression evaluation occurred in a limited environment where only a few symbols (such as &quot;<code>-</code>&quot; and &quot;<code>c</code>&quot; were defined). Such a function is given below:</p>
<div>
<pre><code>subset_strict &lt;-<span> </span><span>function</span>(x, subset, select, <span>drop =</span> <span>FALSE</span>) {
  r &lt;-<span> </span><span>if</span> (<span>missing</span>(subset))
    <span>rep_len</span>(<span>TRUE</span>, <span>nrow</span>(x))
  <span>else</span> {
    e &lt;-<span> </span><span>substitute</span>(subset)
    r &lt;-<span> </span><span>eval</span>(e, x, <span>parent.frame</span>())
    <span>if</span> (<span>!</span><span>is.logical</span>(r))
      <span>stop</span>(<span>&quot;'subset' must be logical&quot;</span>)
    r <span>&amp;</span><span> </span><span>!</span><span>is.na</span>(r)
  }
  vars &lt;-<span> </span><span>if</span> (<span>missing</span>(select))
    <span>TRUE</span>
  <span>else</span> {
    nl &lt;-<span> </span><span>as.list</span>(<span>seq_along</span>(x))
    <span>names</span>(nl) &lt;-<span> </span><span>names</span>(x)
    env &lt;-<span> </span><span>new.env</span>(<span>parent =</span> <span>emptyenv</span>())
    <span>for</span>(sym <span>in</span> <span>c</span>(<span>&quot;-&quot;</span>, <span>&quot;c&quot;</span>, <span>&quot;:&quot;</span>)) {
      <span>assign</span>(sym, <span>get</span>(sym, <span>envir =</span> <span>parent.frame</span>()), <span>envir =</span> env)
    }
    <span>eval</span>(<span>substitute</span>(select), nl, env)
  }
  x[r, vars, drop =<span> </span>drop]
}</code></pre>
</div>
<p>This function is stricter than <code>base::subset()</code>, reliably signalling errors when non-columns are requested.</p>
<div>
<pre><code>y &lt;-<span> &quot;x&quot;</span>
d &lt;-<span> </span><span>data.frame</span>(<span>x =</span> <span>1</span>)

<span>subset_strict</span>(d, <span>select =</span> y)</code></pre>
</div>
<pre><code>## Error in eval(substitute(select), nl, env): object 'y' not found</code></pre>
<div>
<pre><code>d &lt;-<span> </span><span>data.frame</span>(<span>x =</span> <span>1</span>, <span>y =</span> <span>2</span>, <span>z =</span> <span>3</span>)
<span>subset_strict</span>(d, <span>select =</span> y)</code></pre>
</div>
<pre><code>##   y
## 1 2</code></pre>
<div>
<pre><code><span>subset_strict</span>(d, <span>select =</span> <span>-</span>y)</code></pre>
</div>
<pre><code>##   x z
## 1 1 3</code></pre>
<div>
<pre><code><span>subset_strict</span>(d, <span>select =</span> <span>c</span>(x, y))</code></pre>
</div>
<pre><code>##   x y
## 1 1 2</code></pre>
<div>
<pre><code><span>subset_strict</span>(d, <span>select =</span> <span>c</span>(x<span>:</span>z))</code></pre>
</div>
<pre><code>##   x y z
## 1 1 2 3</code></pre>
<p>Roughly I would say the above issue is a reason to not use <code>base::subset()</code> when you are not around to check if it is correct (i.e. in scripts or packages). And <code>help(subset)</code> has some text about this:</p>
<blockquote>
<p>
<strong>Warning</strong></p>
<p>This is a convenience function intended for use interactively. For programming it is better to use the standard subsetting functions like [, and in particular the non-standard evaluation of argument subset can have unanticipated consequences.
</p>
</blockquote>
<p>If you don’t use <code>base::subset()</code> you would think this is a non-issue. Except a similar mal-feature seems to be also in <code>dplyr::select()</code>, and <code>dplyr</code> documentation doesn’t currently seem to advise not using <code>dplyr</code> in non-interactive environments.</p>
<p><code>help(select, package = &quot;dplyr&quot;)</code> tells us that <code>select()</code> takes positional argument from &quot;<code>...</code>&quot; and they are to be &quot;One or more unquoted expressions separated by commas. You can treat variable names like they are positions.&quot; What is unclear is: what environment &quot;<code>...</code>&quot; is actually evaluated in.</p>
<div>
<pre><code>R.version.string</code></pre>
</div>
<pre><code>## [1] &quot;R version 3.5.0 (2018-04-23)&quot;</code></pre>
<div>
<pre><code><span>library</span>(<span>&quot;dplyr&quot;</span>)
<span>packageVersion</span>(<span>&quot;dplyr&quot;</span>)</code></pre>
</div>
<pre><code>## [1] '0.7.6'</code></pre>
<div>
<pre><code><span>packageVersion</span>(<span>&quot;rlang&quot;</span>)</code></pre>
</div>
<pre><code>## [1] '0.2.2'</code></pre>
<div>
<pre><code><span>packageVersion</span>(<span>&quot;tidyselect&quot;</span>)</code></pre>
</div>
<pre><code>## [1] '0.2.4'</code></pre>
<p>Now consider the following code:</p>
<div>
<pre><code>y &lt;-<span> &quot;x&quot;</span>

<span>data.frame</span>(<span>x =</span> <span>1</span>) <span>%>%</span><span> </span>
<span> </span><span>select</span>(y)</code></pre>
</div>
<p>My theory is the typical <code>dplyr</code> user expectation is that the above <em>should</em> throw an exception (as that is what <code>select(data.frame(x = 1), z)</code> does in our clean environment). My opinion is: the above expresses that we are asking for a column named &quot;<code>y</code>&quot;, and there is no such column in the data. The unfortunate coincidence that &quot;<code>y</code>&quot; has a value in our environment should be irrelevant to a <code>select()</code>. &quot;<code>y</code>&quot;’s value might (or might not) be relevant in a complex expression, such as a <code>mutate()</code>– but not to a <code>select()</code>.</p>
<p>We the result actually is as before: the value &quot;<code>y</code>&quot; refers to in the executing environment can unfortunately alter the result (depending on the columns present in the <code>data.frame</code>).</p>
<div>
<pre><code>y &lt;-<span> &quot;x&quot;</span>

<span>data.frame</span>(<span>x =</span> <span>1</span>) <span>%>%</span><span> </span>
<span> </span><span>select</span>(y)</code></pre>
</div>
<pre><code>##   x
## 1 1</code></pre>
<p>Notice we got back a <code>data.frame</code> with a column named &quot;<code>x</code>&quot;. There was no warning or error indicated. This wrong value could poison later results (especially if we had used <code>pull()</code> to convert the data into an unnamed vector). Maybe your code has never run into this, or maybe it already quietly has (as it is not a signalling error).</p>
<p>Notice how the above result differs from the following:</p>
<div>
<pre><code>y &lt;-<span> &quot;x&quot;</span>

<span>data.frame</span>(<span>x =</span> <span>1</span>, <span>y =</span> <span>2</span>) <span>%>%</span><span> </span>
<span> </span><span>select</span>(y)</code></pre>
</div>
<pre><code>##   y
## 1 2</code></pre>
<p>The name/string version of this issue possibly goes back to <code>dplyr 0.7.0</code> (the introduction of <code>rlang</code>). A numeric-index version of the issue can be exhibited for <code>dplyr 0.5.0</code> and <code>dplyr 0.4.0</code> (and possibly earlier, <code>dplyr 0.4.0</code> is the earliest version of <code>dplyr</code> that is convenient to install in <code>R</code> <code>3.5.0</code>). Notice how results vary with data changes below.</p>
<div>
<pre><code><span># restart R</span>
<span>install.packages</span>(<span>&quot;dplyr_0.4.0.tar.gz&quot;</span>, <span>repos =</span> <span>NULL</span>)
<span># restart R</span>
<span>packageVersion</span>(<span>&quot;dplyr&quot;</span>)
<span># [1] ‘0.4.0’</span>
<span>library</span>(<span>&quot;dplyr&quot;</span>)


y &lt;-<span> &quot;x&quot;</span>

<span>data.frame</span>(<span>x =</span> <span>1</span>, <span>y =</span> <span>2</span>) <span>%>%</span><span> </span>
<span> </span><span>select</span>(y)
<span># y</span>
<span># 1 2</span>

<span>data.frame</span>(<span>x =</span> <span>1</span>) <span>%>%</span><span> </span>
<span> </span><span>select</span>(y)
<span># Error: All select() inputs must resolve to integer column positions.</span>
<span># The following do not:</span>
<span># * y</span>

y &lt;-<span> </span>1L

<span>data.frame</span>(<span>x =</span> <span>1</span>, <span>y =</span> <span>2</span>) <span>%>%</span><span> </span>
<span> </span><span>select</span>(y)
<span># y</span>
<span># 1 2</span>

<span>data.frame</span>(<span>x =</span> <span>1</span>) <span>%>%</span><span> </span>
<span> </span><span>select</span>(y)
<span># x</span>
<span># 1 1</span>
  

<span># restart R</span>
<span>install.packages</span>(<span>&quot;dplyr&quot;</span>)
<span># restart R</span></code></pre>
</div>
<p>It is possible this is in fact designed or intentional behavior. However, if that is the case it still violates some principles of safety through isolation and <a rel="nofollow" href="https://en.wikipedia.org/wiki/Principle_of_least_astonishment" target="_blank">least astonishment</a>.</p>
<p>In <code>dplyr 0.7.6</code>: <code>select(.data[[y]])</code> shows the same issue, while <code>select(.data[[!!y]])</code> does not (and <code>select(.data$y)</code> appears to always use the name &quot;<code>y</code>&quot;, which seems right). I suspect &quot;<code>.data[y]</code>&quot;, while <a rel="nofollow" href="https://blog.rstudio.com/2017/06/13/dplyr-0-7-0/" target="_blank">commonly taught</a>, may not be well-formed <code>dplyr</code> code (i.e. the user should not type that so the results are possibly user error, and not package error).</p>
<p>In conclusion:</p>
<ul>
<li><code>base::subset()</code> in addition to having known issues with the non-standard evaluation of its <code>subset</code> argument has a dangerous double look-up mal-feature in its <code>select</code> argument.</li>
<li><code>dplyr::select()</code> has variations of a similar mal-feature. In the presence of this issue the only safe way to use name-capturing code with <code>dplyr::select()</code> appears to be <code>select(&quot;y&quot;)</code> or <code>select(.data$y)</code>, which are both inconvenient enough to defeat the purpose of non-standard evaluation (the eliding of quotation marks in this case).</li>
<li>It is possible <code>select(.data[[y]])</code> is not well-formed <code>dplyr</code> code, or is ambiguous (is it meant to operate on names, on values, or both?), or has a similar bug.</li>
</ul>
<p>I think the above shows a small bit of how NSE interfaces <em>can</em> be confusing and hard to get right both for users and package developers. This is why I advise avoiding introducing NSE interfaces unless they are buying you something much larger than leaving out a few quote marks.</p>
<blockquote>
<p>
&quot;Those who would give up essential Referential Transparency, to purchase a little temporary Notational Conveneince, deserve neither Referential Transparency nor Notational Conveneince.&quot;</p>
<p>(not) Ben Franklin
</p>
</blockquote>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2018/09/a-subtle-flaw-in-some-popular-r-nse-interfaces/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/oexjjd93hDg" height="1"></a></div>