<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/parametric-variable-names-and-dplyr/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2016/12/parametric-variable-names-and-dplyr/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>When writing reusable code or packages you often do not know the names of the columns or variables you need to work over. This is what I call “parametric treatment of variables.” This can be a problem when using <code>R</code> libraries that assume you know the variable names. The <a target="_blank" rel="nofollow" href="https://cran.r-project.org/"><code>R</code></a> data manipulation library <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=dplyr"><code>dplyr</code></a> currently supports parametric treatment of variables through “underbar forms” (methods of the form <code>dplyr::*_</code>), but their use can get tricky.</p>
<p><center><br>
<img title="NewImage.png" alt="NewImage" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2016/12/NewImage-1.png?w=660" border="0" data-recalc-dims="1"><br>
<a target="_blank" rel="nofollow" href="https://commons.wikimedia.org/wiki/File:Rube_Goldberg%27s_%22Self-Operating_Napkin%22_(cropped).gif">Rube Goldberg machine 1931 (public domain).</a><br>
</center></p>
<p>Better support for parametric treatment of variable names would be a boon to <code>dplyr</code> users. To this end the <a target="_blank" rel="nofollow" href="https://github.com/WinVector/replyr"><code>replyr</code></a> package now has a method designed to re-map parametric variable names to known concrete variable names. This allows concrete <code>dplyr</code> code to be used as if it was parametric.<span></span></p>
<h2>The Problem</h2>
<p><code>dplyr</code> is a library that prefers you know the name of the column you want to work with. This is great when performing a specific analysis, but somewhat painful when supplying re-usable functions or packages. <code>dplyr</code> has a complete parametric interface with the “underbar forms” (for example: using <code>dplyr::filter_</code> instead of <code>dplyr::filter</code>). However, the underbar notation (and the related necessary details around specifying lazy evaluation of formulas) rapidly becomes difficult.</p>
<p>As an attempted work-around <code>replyr</code> now supplies an adapter that applies a mapping from column names you have (which can be supplied parametrically) to concrete column names you wish you had (which would allow you to write <code>dplyr</code> code simply in terms of known or assumed column names).</p>
<p>It is easier to show than explain.</p>
<h2>An Example</h2>
<p>First we set up our libraries and type in some notional data as our example:</p>
<p><code>
<pre>
# install.packages('devtools')                 # Run this if you don't already have devtools
# devtools::install_github('WinVector/replyr') # Run this if you don't already have replyr
library('dplyr')
library('replyr')

d &lt;- data.frame(Sepal_Length=c(5.8,5.7),Sepal_Width=c(4.0,4.4),
                Species='setosa',rank=c(1,2))
print(d)

 #   Sepal_Length Sepal_Width Species rank
 # 1          5.8         4.0  setosa    1
 # 2          5.7         4.4  setosa    2
</pre>
<p></p>
<p>Then we rename the columns to standard values while restricting to only the named columns (this is the magic step):</p>
<p><code>
<pre>
nmap &lt;- c('GroupColumn','ValueColumn','RankColumn')
names(nmap) &lt;-  c('Species','Sepal_Length','rank')
print(nmap)

 #       Species  Sepal_Length          rank 
 # &quot;GroupColumn&quot; &quot;ValueColumn&quot;  &quot;RankColumn&quot; 

dtmp &lt;- replyr_renameRestrictCols(d,nmap)
print(dtmp)

 #   GroupColumn ValueColumn RankColumn
 # 1      setosa         5.8          1
 # 2      setosa         5.7          2
</pre>
<p></p>
<p>At this point you do know the column names (they are the ones you picked) and can write nice neat <code>dplyr</code>. You can then do your work:</p>
<p><code>
<pre>
# pretend this block is a huge sequence of complicated and expensive operations.
dtmp %>% mutate(RankColumn=RankColumn-1) -> dtmp # start ranks at zero
</pre>
<p></p>
<p>Notice we were able to use <code>dplyr::mutate</code> without needing to use <code>dplyr::mutate_</code> (and without needing to go to Stack Overflow to lookup the lazy-eval notation yet again).</p>
<p>Once you have your desired result you restore the original names of our restricted column set:</p>
<p><code>
<pre>
invmap &lt;- names(nmap)
names(invmap) &lt;- as.character(nmap)
print(invmap)

 #    GroupColumn    ValueColumn     RankColumn 
 #      &quot;Species&quot; &quot;Sepal_Length&quot;         &quot;rank&quot; 

replyr_renameRestrictCols(dtmp,invmap)

 #   Species Sepal_Length rank
 # 1  setosa          5.8    0
 # 2  setosa          5.7    1
</pre>
<p></p>
<p>If you haven’t worked a lot with <code>dplyr</code> this won’t look that interesting. If you do work a lot with <code>dplyr</code> you may have been asking for something like this for quite a while. If you use dplyr::*_ you will love <code>replyr::replyr_renameRestrictCols</code>. Be aware: <code>replyr::replyr_renameRestrictCols</code> is a bit of a hack; it mutates all of the columns it is working with, which is unlikely to be a cheap operation. </p>
<h2>A Proposal</h2>
<p>I feel the <code>replyr::replyr_renameRestrictCols</code> interface represents the correct design for a better <code>dplyr</code> based adapter. </p>
<p>I’ll call this the “column view stack proposal.” I would suggest the addition of two operators to <code>dplyr</code>:</p>
<ol>
<li><code>view_as(df,columnNameMap)</code> takes a data item and returns a data item reference that behaves as if the column names have been re-mapped.</li>
<li><code>unview()</code> removes the <code>view_as</code> annotation.</li>
</ol>
<p>Obviously there is an issue of nested views, I would suggest maintaining the views as a stack while using the composite transformation implied by the stack of mapping specifications. I am assuming <code>dplyr</code> does not currently have such a facility. Another possibility is a term-rewriting engine to re-map formulas from standard names to target names, but this is what the lazy-eval notations are already attempting (and frankly it isn’t convenient or pretty). </p>
<p>I would also suggest that <code>dplyr::arrange</code> be enhanced to have a visible annotation (just the column names it has arranged by) that allows the user to check if the data is believed to be ordered (crucial for window-function applications). With these two suggestions <code>dplyr</code> data sources would support three primary annotations:</p>
<ol>
<li><code>Groups</code>: placed by <code>dplyr::group_by</code>, removed by <code>dplyr::ungroup</code>, and viewed by <code>dplyr::groups</code>.</li>
<li><code>Orders</code>: placed by <code>dplyr::arrange</code>, removed by <code>Xdplyr::unarrange</code> (just removes annotation, does not undo arrangement; annotation also removed by any operation that re-orders the data, such as <code>join</code>), and viewed by <code>Xdplyr::arrangement</code>.</li>
<li><code>Column Views</code>: placed by <code>Xdplyr::view_as</code>, removed by <code>Xdplyr::unview</code>, and viewed by <code>Xdplyr::views</code>.</li>
</ol>
<p>The “<code>Xdplyr::</code>” items are the extensions that are being proposed.</p>
<h2>An Alternative Proposal</h2>
<p>Another possibility would be some sort of “<code>let</code>” statement that controls name bindings for the duration of a block of code. Using such a statement we would write our above example calculation as:</p>
<p><code>
<pre>
let(
 list(RankColumn='rank',GroupColumn='Species'),
 {
  # pretend this block is a huge sequence of complicated and expensive operations.
  d %>% mutate(RankColumn=RankColumn-1) -> dtmp # start ranks at zero
 }
)
</pre>
<p></p>
<p>The idea is the items <code>'rank'</code> and <code>'Species'</code> could be passed in parametrically (notice the <code>let</code> specification is essentially <code>invmap</code>, so we could just pass that in). This isn’t quite <code>R</code>‘s “<code>with</code>” statement as we are not binding names to values, but names to names. Essentially we are asking for macro facility that is compatible with <code>dplyr</code> remote data sources (and the non-standard evaluation methods used to capture variables names). I haven’t experimented with <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=gtools">gtools</a> yet, so I don’t know if this is feasible (some <code>R</code> macro examples can be found <a target="_blank" rel="nofollow" href="https://www.r-bloggers.com/macros-in-r/">here</a>).</p>
<p>If somebody has this working I would love to hear about it. The reason I worry is the following. Clever meta-programming depends on all of the rest of the code not being clever and not breaking referential transparency. And in this case <code>dplyr</code> is using non-standard evaluation to capture variable names, which is typically not referentially transparent. Also <code>R</code> functions are roughly <code>Lisp</code> <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Fexpr"><code>fexpr</code></a>s, which signals to me <code>R</code>‘s design may not have a fully clean distinctions between functions and macros.</p>
<h2>Conclusion</h2>
<p>I feel the above methods will make working with parameterized variables in <code>dplyr</code> much easier.</p>

<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2016/12/parametric-variable-names-and-dplyr/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/mQcOHDqOsRo"></a></div></code></code></code></code></code>