<div> 

<div>   
<div>
<ul>
<li><a rel="nofollow" href="https://r-some-blog.blogspot.com/2018/12/smartly-select-and-mutate-data-frame_9.html#motivation" target="_blank">Motivation</a></li>
<li><a rel="nofollow" href="https://r-some-blog.blogspot.com/2018/12/smartly-select-and-mutate-data-frame_9.html#column-operations" target="_blank">Column operations</a>
<ul>
<li><a rel="nofollow" href="https://r-some-blog.blogspot.com/2018/12/smartly-select-and-mutate-data-frame_9.html#add" target="_blank">Add</a></li>
<li><a rel="nofollow" href="https://r-some-blog.blogspot.com/2018/12/smartly-select-and-mutate-data-frame_9.html#modify" target="_blank">Modify</a></li>
<li><a rel="nofollow" href="https://r-some-blog.blogspot.com/2018/12/smartly-select-and-mutate-data-frame_9.html#remove" target="_blank">Remove</a></li>
</ul>
</li>
<li><a rel="nofollow" href="https://r-some-blog.blogspot.com/2018/12/smartly-select-and-mutate-data-frame_9.html#benchmark" target="_blank">Benchmark</a></li>
<li><a rel="nofollow" href="https://r-some-blog.blogspot.com/2018/12/smartly-select-and-mutate-data-frame_9.html#summary" target="_blank">Summary</a></li>
</ul>
</div>
<div>
<h2>Motivation</h2>
<p>The <a rel="nofollow" href="https://cran.r-project.org/package=dplyr" target="_blank">dplyr</a> functions <code>select</code> and <code>mutate</code> nowadays are commonly applied to perform <code>data.frame</code> column operations, frequently combined with <a rel="nofollow" href="https://cran.r-project.org/package=magrittr" target="_blank">magrittr</a>s forward <code>%>%</code> pipe. While working well interactively, however, these methods often would require additional checking if used in “serious” code, for example, to catch column name clashes.</p>
<p>In principle, the <a rel="nofollow" href="https://cran.r-project.org/web/packages/container/index.html" target="_blank">container</a> package provides a <code>dict</code>-class (resembling <a rel="nofollow" href="https://www.python.org/" target="_blank">Python</a>s dict type), which allows to cover these issues more easily. In its very recent update, the <a rel="nofollow" href="https://cran.r-project.org/web/packages/container/index.html" target="_blank">container</a> package for this reason gained an S3 method interface plus functions to convert back and forth between <code>dict</code> and <code>data.frame</code>. This can be used to extend the set of <code>data.frame</code> column operations and in this post I will show how and when they can serve as a useful alternative to <code>mutate</code> and <code>select</code>.</p>
<p>To keep matters simple, we use a tiny data table.</p>
<pre><code>data &lt;- data.frame(x = c(0.2, 0.5), y = letters[1:2])<br>data<br>##     x y<br>## 1 0.2 a<br>## 2 0.5 b</code></pre>
</div>
<div>
<h2>Column operations</h2>
<div>
<h3>Add</h3>
<p>Let’s add a column using <code>mutate</code>.</p>
<pre><code>library(dplyr)<br>n &lt;- nrow(data)<br><br>data %>% <br>    mutate(ID = 1:n)<br>##     x y ID<br>## 1 0.2 a  1<br>## 2 0.5 b  2</code></pre>
<p>For someone not familar with the <a rel="nofollow" href="https://www.tidyverse.org/" target="_blank">tidyverse</a>, this code block might read somewhat odd as the column is added and <em>not</em> mutated. To add a column using <code>dict</code> simply use <code>add</code>.</p>
<pre><code>library(container)<br>data %>% <br>    as.dict() %>% <br>    add(&quot;ID&quot;, 1:n) %>%<br>    as.data.frame()<br>##     x y ID<br>## 1 0.2 a  1<br>## 2 0.5 b  2</code></pre>
<p>The intended add-operation is stated more clearly, but on the downside we also had to add some overhead. Of course, since this has to be done only at the beginning and at the end of the pipe, it will be less of an issue if multiple dict-operations are performed in between. Next, instead of ID, let’s add another numeric column <code>y</code>, which happens to “name-clash” with the already existing column.</p>
<pre><code>data %>% <br>    mutate(y = rnorm(n))<br>##     x         y<br>## 1 0.2 1.5295729<br>## 2 0.5 0.4821652</code></pre>
<p>Ooops – we have accidently overwritten the initial y-column. While this was easy to see here, it may not if the <code>data.frame</code> has a lot of columns or if column names are created automatically as part of some script. To catch this, usually some overhead is required, too.</p>
<pre><code>if (&quot;y&quot; %in% colnames(data)) {<br>    stop(&quot;column y already exists&quot;)<br>} else {<br>    data %>% <br>        mutate(y = rnorm(n))<br>}<br>## Error in eval(expr, envir, enclos): column y already exists</code></pre>
<p>Let’s see the dict-operation in comparison.</p>
<pre><code>data %>% <br>    as.dict() %>% <br>    add(&quot;y&quot;, rnorm(n)) %>%<br>    as.data.frame()<br>## Error in x$add(key, value): key 'y' already in Dict</code></pre>
<p>The name clash is catched by default and the overhead does not look so silly anymore. As a bonus, the error message still provides information about the originally <em>intended</em> add-operation.</p>
</div>
<div>
<h3>Modify</h3>
<p>If the intend was indeed to overwrite the value, the dict-function <code>setval</code> can be used.</p>
<pre><code>data %>% <br>    as.dict() %>% <br>    setval(&quot;y&quot;, rnorm(n)) %>%<br>    as.data.frame()<br>##     x          y<br>## 1 0.2  0.5315501<br>## 2 0.5 -0.1573612</code></pre>
<p>As we saw above, if a column does not exist, <code>mutate</code> silently creates it for you. If this is not what you want, which means, you want to make sure something is overwritten, again, a workaround is needed.</p>
<pre><code>if (&quot;ID&quot; %in% colnames(data)) {<br>    data %>% <br>        mutate(ID = 1:n)    <br>} else {<br>    stop(&quot;column ID not in data.frame&quot;)<br>}<br>## Error in eval(expr, envir, enclos): column ID not in data.frame</code></pre>
<p>Once again, the workaround is already “built-in” in the dict-framework.</p>
<pre><code>data %>% <br>    as.dict() %>% <br>    setval(&quot;ID&quot;, 1:n) %>%<br>    as.data.frame()<br>## Error in x$set(key, value, add): key 'ID' not in Dict</code></pre>
<p>After all, the intend of the <code>mutate</code> function actually would be something like: <em>overwrite a column, or, create it if it does not exist</em>. If desired, this behaviour can be expressed within the dict-framework as well.</p>
<pre><code>data %>% <br>    as.dict() %>% <br>    setval(&quot;ID&quot;, 1:n, add=TRUE) %>%<br>    as.data.frame()<br>##     x y ID<br>## 1 0.2 a  1<br>## 2 0.5 b  2</code></pre>
</div>
<div>
<h3>Remove</h3>
<p>A common <a rel="nofollow" href="https://www.tidyverse.org/" target="_blank">tidyverse</a> approach to remove a column is based on the <code>select</code> function. One corresponding dict-function is <code>remove</code>.</p>
<pre><code>data %>% <br>        select(-&quot;y&quot;)<br>##     x<br>## 1 0.2<br>## 2 0.5<br><br>data %>% <br>    as.dict() %>% <br>    remove(&quot;y&quot;) %>% <br>    as.data.frame()<br>##     x<br>## 1 0.2<br>## 2 0.5</code></pre>
<p>Let’s see what happens if the column does not exist in the first place.</p>
<pre><code>data %>%<br>    select(-&quot;ID&quot;)<br>## Error: Unknown column `ID`<br><br>data %>% <br>    as.dict() %>% <br>    remove(&quot;ID&quot;) %>% <br>    as.data.frame()<br>## Error in x$remove(key): key 'ID' not in Dict</code></pre>
<p>Again, we obtain a slightly more informative error message with dict. Assume we want the column to be removed if it exist but otherwise silently ignore the command, for example:</p>
<pre><code>if (&quot;ID&quot; %in% colnames(data)) {<br>    data %>%<br>        select(-&quot;ID&quot;)    <br>}</code></pre>
<p>You may have expected this by now – the dict-framework provides a straigh-forward solution, namely, the <code>discard</code> function:</p>
<pre><code>data %>% <br>    as.dict() %>% <br>    discard(&quot;ID&quot;) %>% <br>    discard(&quot;y&quot;) %>%<br>    add(&quot;z&quot;, c(&quot;Hello&quot;, &quot;World&quot;)) %>%<br>    as.data.frame()<br>##     x     z<br>## 1 0.2 Hello<br>## 2 0.5 World</code></pre>
</div>
</div>
<div>
<h2>Benchmark</h2>
<p>The required additional code lines are limited but what about the computational overhead? To examine this, we benchmark some column operations using the famous ‘iris’ data set. As a hallmark reference we will also bring the <a rel="nofollow" href="https://cran.r-project.org/package=data.table" target="_blank">data.table</a> framework to the competition.</p>
<pre><code>set.seed(123)<br>library(microbenchmark)<br>library(ggplot2)<br>library(data.table)<br>data &lt;- iris<br>n &lt;- nrow(data)<br>head(iris)<br>##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species<br>## 1          5.1         3.5          1.4         0.2  setosa<br>## 2          4.9         3.0          1.4         0.2  setosa<br>## 3          4.7         3.2          1.3         0.2  setosa<br>## 4          4.6         3.1          1.5         0.2  setosa<br>## 5          5.0         3.6          1.4         0.2  setosa<br>## 6          5.4         3.9          1.7         0.4  setosa</code></pre>
<p>For the benchmark, we add one, transform one and finally delete one column.</p>
<pre><code>bm &lt;- microbenchmark(control = list(order=&quot;inorder&quot;), times = 100,<br>    dplyr = data %>% <br>        mutate(ID = 1:n) %>%<br>        mutate(Species = as.character(Species)) %>%<br>        select(-Sepal.Width),<br>    <br>    dict = data %>%<br>        as.dict() %>%<br>        add(&quot;ID&quot;, 1:n) %>%<br>        setval(., &quot;Species&quot;, as.character(.$get(&quot;Species&quot;))) %>%<br>        remove(&quot;Sepal.Width&quot;) %>%<br>        as.data.frame(),<br>    <br>    `[.data.table` = data %>% <br>        as.data.table() %>%<br>        .[, ID := 1:n] %>%<br>        .[, Species := as.character(Species)] %>%<br>        .[, Sepal.Width := NULL]<br>)<br>autoplot(bm) + theme_bw()</code></pre>
<p><a rel="nofollow" href="https://i0.wp.com/1.bp.blogspot.com/-54WhxkOCNSg/XAlvgZwUcwI/AAAAAAAAAtI/QsqeLbzxut49sjuAljo_NKnnCwGm1oijgCPcBGAYYCw/s1600/benchmark1-1.png?ssl=1" target="_blank"><img border="0" data-original-height="961" src="https://i0.wp.com/1.bp.blogspot.com/-54WhxkOCNSg/XAlvgZwUcwI/AAAAAAAAAtI/QsqeLbzxut49sjuAljo_NKnnCwGm1oijgCPcBGAYYCw/s1600/benchmark1-1.png?resize=450%2C961&ssl=1" data-recalc-dims="1" data-original-width="450"></a></p>
<p>Somewhat surprisingly maybe, the dict-implementation is closer to the <a rel="nofollow" href="https://cran.r-project.org/package=data.table" target="_blank">data.table</a> than to the <a rel="nofollow" href="https://cran.r-project.org/package=dplyr" target="_blank">dplyr</a> performance. Let’s examine each operation in more detail.</p>
<pre><code>bm &lt;- microbenchmark(control = list(order=&quot;inorder&quot;), times = 100,<br>    tbl &lt;- mutate(data, ID = 1:n),<br>    tbl &lt;- mutate(tbl, Species = as.character(Species)),<br>    tbl &lt;- select(tbl, -Sepal.Width),<br>    <br>    dic &lt;- as.dict(data),<br>    add(dic, &quot;ID&quot;, 1:n),<br>    setval(dic, &quot;Species&quot;, as.character(dic$get(&quot;Species&quot;))),<br>    remove(dic, &quot;Sepal.Width&quot;),<br>    dic &lt;- as.data.frame(dic),<br>    <br>    dt &lt;- as.data.table(data),<br>    dt[, ID := 1:n],<br>    dt[, Species := as.character(Species)],<br>    dt[, Sepal.Width := NULL]<br>)<br>autoplot(bm) + theme_bw()</code></pre>
<p><a rel="nofollow" href="https://i2.wp.com/2.bp.blogspot.com/-c2nnWFqsAlo/XAlusRLz8bI/AAAAAAAAAs4/tholDFE7pb0YyZzUknnT0d-a8tKmrpQGACPcBGAYYCw/s1600/benchmark2-1.png?ssl=1" target="_blank"><img border="0" data-original-height="961" src="https://i2.wp.com/2.bp.blogspot.com/-c2nnWFqsAlo/XAlusRLz8bI/AAAAAAAAAs4/tholDFE7pb0YyZzUknnT0d-a8tKmrpQGACPcBGAYYCw/s1600/benchmark2-1.png?resize=450%2C961&ssl=1" data-recalc-dims="1" data-original-width="450"></a></p>
<p>Apparently, the mutate and select operations are the slowest in comparison, I think, because both the dict and <a rel="nofollow" href="https://cran.r-project.org/package=data.table" target="_blank">data.table</a> approach work by reference while probably some copying is done in the <a rel="nofollow" href="https://cran.r-project.org/package=dplyr" target="_blank">dplyr</a> pipe. We also see that the dict-approach spends most of the computation time for the transformation back and forth between a <code>dict</code> and a <code>data.frame</code> while the actual column operations seem very efficient, even more efficient than that of <a rel="nofollow" href="https://cran.r-project.org/package=data.table" target="_blank">data.table</a>. This certainly came as a surprise to me, as the focus when developing the <a rel="nofollow" href="https://cran.r-project.org/web/packages/container/index.html" target="_blank">container</a> package has never been on speed but rather on providing a concise data structure. Internally the dict simply consists of a named list, so I guess this speaks for the efficiency of base R list operations. Having said that, I found that the <a rel="nofollow" href="https://cran.r-project.org/package=data.table" target="_blank">data.table</a> code can be further improved by avoiding the overhead of the <code>[.data.table</code> operator and instead use the built-in <code>set</code> function:</p>
<pre><code>bm &lt;- microbenchmark(control = list(order=&quot;inorder&quot;), times = 100,<br>    dplyr = data %>% <br>        mutate(ID = 1:n) %>%<br>        mutate(Species = as.character(Species)) %>%<br>        select(-Sepal.Width),<br>    <br>    dict = data %>%<br>        as.dict() %>%<br>        add(&quot;ID&quot;, 1:n) %>%<br>        setval(., &quot;Species&quot;, as.character(.$get(&quot;Species&quot;))) %>%<br>        remove(&quot;Sepal.Width&quot;) %>%<br>        as.data.frame(),<br>    <br>    `[.data.table` = data %>% <br>        as.data.table() %>%<br>        .[, ID := 1:n] %>%<br>        .[, Species := as.character(Species)] %>%<br>        .[, Sepal.Width := NULL],<br>    <br>    data.table.set = data %>% <br>        as.data.table() %>%<br>        set(., j= &quot;ID&quot;, value = 1:n) %>%<br>        set(., j = &quot;Species&quot;, value = as.character(.[[&quot;Species&quot;]])) %>%<br>        set(., j = &quot;Sepal.Width&quot;, value = NULL)<br>)<br>autoplot(bm) + theme_bw()</code></pre>
<p><a rel="nofollow" href="https://i2.wp.com/2.bp.blogspot.com/-0hHhodE858c/XAluc7ZoctI/AAAAAAAAAsw/tkYk09xyTNYYsr8bd6ey9H434sYpNPIswCPcBGAYYCw/s1600/benchmark3-1.png?ssl=1" target="_blank"><img border="0" data-original-height="961" src="https://i2.wp.com/2.bp.blogspot.com/-0hHhodE858c/XAluc7ZoctI/AAAAAAAAAsw/tkYk09xyTNYYsr8bd6ey9H434sYpNPIswCPcBGAYYCw/s1600/benchmark3-1.png?resize=450%2C961&ssl=1" data-recalc-dims="1" data-original-width="450"></a></p>
<p>This puts things back into perspective, I guess <img alt="🙂" src="https://s.w.org/images/core/emoji/2.4/72x72/1f642.png" class="wp-smiley"> It might also be interesting to know, how much of the computation time is spent on the non-standard evaluation part of the <code>dplyr</code> and <code>[.data.table</code> implementation, but that’s probably a topic on its own.</p>
</div>
<div>
<h2>Summary</h2>
<p>Accidently overwriting existing data columns leads to nasty bugs. The presented workflow allows to increase both reliability and precision of standard data frame column manipulation at very little cost. The intended column operations can be expressed more clearly and, in case of failures, informative error messages are provided by default. As a result, the dict-framework may serve as a useful supplement to “interactive piping”.</p>
</div></div>
<p>   </p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://r-some-blog.blogspot.com/2018/12/smartly-select-and-mutate-data-frame_9.html"> R some blog </a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/tyKASIGEVzQ" height="1"></a></div></div>