<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/track-changes-in-data-with-the-lumberjack/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.markvanderloo.eu/yaRb/2017/06/23/track-changes-in-data-with-the-lumberjack/"> R – Mark van der Loo</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>So you are using this pipeline to have data treated by different functions in R. For example, you may be imputing some missing values using the <a rel="nofollow" href="https://cran.r-project.org/package=simputation" target="_blank">simputation</a> package. Let us first load the only realistic dataset in R</p>
<pre><code>> data(retailers, package=&quot;validate&quot;)
> head(retailers, 3)
  size incl.prob staff turnover other.rev total.rev staff.costs total.costs profit vat
1  sc0      0.02    75       NA        NA      1130          NA       18915  20045  NA
2  sc3      0.14     9     1607        NA      1607         131        1544     63  NA
3  sc3      0.14    NA     6886       -33      6919         324        6493    426  NA
</code></pre>
<p>This data is dirty with missings and full of errors. Let us do some imputations with simputation.</p>
<pre><code>> out &lt;- retailers %>% 
+   impute_lm(other.rev ~ turnover) %>%
+   impute_median(other.rev ~ size)
> 
> head(out,3)
  size incl.prob staff turnover other.rev total.rev staff.costs total.costs profit vat
1  sc0      0.02    75       NA  6114.775      1130          NA       18915  20045  NA
2  sc3      0.14     9     1607  5427.113      1607         131        1544     63  NA
3  sc3      0.14    NA     6886   -33.000      6919         324        6493    426  NA
> 
</code></pre>
<p>Ok, cool, we know all that. But what if you’d like to know what value was imputed with which method? That’s where the <a rel="nofollow" href="https://cran.r-project.org/package=lumberjack" target="_blank">lumberjack</a> comes in.</p>
<p>The lumberjack operator is a `pipe'[1] operator that allows you to track changes in data.</p>
<pre><code>> library(lumberjack)
> retailers$id &lt;- seq_len(nrow(retailers))
> out &lt;- retailers %>>% 
+   start_log(log=cellwise$new(key=&quot;id&quot;)) %>>%
+   impute_lm(other.rev ~ turnover) %>>%
+   impute_median(other.rev ~ size) %>>%
+   dump_log(stop=TRUE)
Dumped a log at cellwise.csv
> 
> read.csv(&quot;cellwise.csv&quot;) %>>% dplyr::arrange(key) %>>% head(3)
  step                     time                      expression key  variable old      new
1    2 2017-06-23 21:11:05 CEST impute_median(other.rev ~ size)   1 other.rev  NA 6114.775
2    1 2017-06-23 21:11:05 CEST impute_lm(other.rev ~ turnover)   2 other.rev  NA 5427.113
3    1 2017-06-23 21:11:05 CEST impute_lm(other.rev ~ turnover)   6 other.rev  NA 6341.683
> 
</code></pre>
<p>So, to track changes we only need to switch from <code>%>%</code> to <code>%>>%</code> and add the <code>start_log()</code> and <code>dump_log()</code> function calls in the data pipeline. (to be sure: it works with any function, not only with simputation). The package is on CRAN now, and please see the <a rel="nofollow" href="https://cran.r-project.org/web/packages/lumberjack/vignettes/intro.html" target="_blank">introductory vignette</a> for more examples and ways to customize it.</p>
<p>There are many ways to track changes in data. That is why the lumberjack is completely extensible. The package comes with a few loggers, but users or package authors are invited to write their own. Please see the <a rel="nofollow" href="https://cran.r-project.org/web/packages/lumberjack/vignettes/extending.html" target="_blank">extending lumberjack</a> vignette for instructions.</p>
<p>If this post got you interested, please install the package using</p>
<pre><code>install.packages('lumberjack')
</code></pre>
<p>You can get started with the <a rel="nofollow" href="https://cran.r-project.org/web/packages/lumberjack/vignettes/intro.html" target="_blank">introductory vignette</a> or even just use the lumberjack operator <code>%>>%</code> as a (close) replacement of the <code>%>%</code> operator.</p>
<p>As always, I am open to suggestions and comments. Either through the packages <a rel="nofollow" href="https://github.com/markvanderloo/lumberjack" target="_blank">github page</a>.</p>
<p>Also, I will be <a rel="nofollow" href="http://sched.co/AxsM" target="_blank">talking</a> at <a rel="nofollow" href="https://user2017.brussels/" target="_blank">useR2017</a> about the simputation package, but I will sneak in a bit of lumberjack as well :p.</p>
<p>And finally, here’s a picture of a lumberjack smoking a pipe.<br>
<img alt="" src="https://i1.wp.com/s-media-cache-ak0.pinimg.com/736x/40/71/85/4071850941f4b7a09b9a3a74b2cc14e1.jpg?w=456&ssl=1" data-recalc-dims="1"></p>
<p>[1] It really should be called a function composition operator, but potetoes/potatoes.</p>
</div>
<div class="wp-gfm-footer">
<div class="wp-gfm-ad"><span class="wp-gfm-powered-by">Markdown with by <img alt="❤" src="https://i0.wp.com/s.w.org/images/core/emoji/72x72/2764.png?resize=10%2C10&ssl=1" data-recalc-dims="1"> <a rel="nofollow" href="https://github.com/makotokw/wp-gfm" target="_blank">wp-gfm</a></span></div>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.markvanderloo.eu/yaRb/2017/06/23/track-changes-in-data-with-the-lumberjack/"> R – Mark van der Loo</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/4_hrSPXg96c" height="1"></a></div>