<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/the-prequel-to-the-drake-r-package/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ropensci.org/blog/2018/02/06/drake/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>The <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a> R package is a <a rel="nofollow" href="https://github.com/pditommaso/awesome-pipeline" target="_blank">pipeline</a> toolkit. It manages data science workflows, <a rel="nofollow" href="https://ropensci.github.io/drake/#what-gets-done-stays-done-" target="_blank">saves time</a>, and <a rel="nofollow" href="https://ropensci.github.io/drake/#reproducibility-with-confidence" target="_blank">adds more confidence to reproducibility</a>. I hope it will impact the landscapes of reproducible research and high-performance computing, but I originally created it for different reasons. This post is the prequel to <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a>’s inception. There was struggle, and <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a> was the answer.</p>
<h3>Dissertation frustration</h3>
<p><center><br>
<img alt="Sisyphus. https://sites.google.com/site/sisyphusa/" src="https://i2.wp.com/ropensci.org/img/blog-images/2018-02-06-drake/uphill.png?w=456&ssl=1" data-recalc-dims="1"><br>

<div><em>Sisyphus. <a rel="nofollow" href="https://sites.google.com/site/sisyphusa/" target="_blank">https://sites.google.com/site/sisyphusa/</a></em></div>
<p></p>
<p>My dissertation project was intense. The final computational challenge was to analyze multiple genomics datasets using an <a rel="nofollow" href="https://github.com/wlandau/fbseq" target="_blank">emerging method</a> and its competitors. Even with <a rel="nofollow" href="http://blog.revolutionanalytics.com/2015/06/computing-with-gpus-in-r.html" target="_blank">GPU computing</a>, which shrank days of runtime down to hours, the full battery of <a rel="nofollow" href="https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo" target="_blank">Markov chain Monte Carlo</a> runs took several weeks from start to finish. I organized my <a rel="nofollow" href="https://github.com/wlandau/fbseqStudies" target="_blank">workflow as an R package</a>, and I worked in a loop:</p>
<ol>
<li>Deploy the computations.</li>
<li>Wait a few weeks.</li>
<li>Discover an issue.</li>
<li>Restart from scratch.</li>
</ol>
<p>At the time, the dominant R-focused workflow managers could not break the cycle. <a rel="nofollow" href="https://github.com/yihui/knitr" target="_blank">Knitr</a> was designed to weave together code and prose, and its paradigm deliberately lacked enough modularity to properly scale. <a rel="nofollow" href="https://github.com/johnmyleswhite/ProjectTemplate" target="_blank">ProjectTemplate</a> was mostly concerned with organization and readability. These and similar tools had only traces of the functionality I sought. For the right solution, I needed to step off the beaten path.</p>
<h3>GNU Make</h3>
<pre><code>paper.pdf: paper.tex figure.png 
    pdflatex paper.tex
    
figure.png: figure.R results.csv
    Rscript figure.R
    
results.csv: long-computation.R
    Rscript long-computation.R
</code></pre>
<p><a rel="nofollow" href="https://www.gnu.org/software/make/" target="_blank">GNU Make</a> is a dependency watcher first and foremost. Its top priority is to bring results up to date with as little work as possible, and it gives you parallel computing for free. In fact, according to <a rel="nofollow" href="http://kbroman.org/minimal_make/" target="_blank">Karl Broman</a> and others, <a rel="nofollow" href="https://www.gnu.org/software/make/" target="_blank">Make</a> surpasses even <a rel="nofollow" href="https://github.com/yihui/knitr" target="_blank">knitr</a> as a helpful reproducible research tool.</p>
<p>My advisor, <a rel="nofollow" href="https://github.com/jarad/" target="_blank">Jarad Niemi</a>, repeatedly urged me to use <a rel="nofollow" href="https://www.gnu.org/software/make/" target="_blank">Make</a>. At the time, I was too entrenched in half-written code to transition, so I finished my graduate school work with brute force. I defended my thesis, moved out of Iowa, and because I had goofed up the paperwork, remained a graduate student for one last summer. <a rel="nofollow" href="https://github.com/jarad/" target="_blank">Jarad</a> and I used most of our remaining time to find and create better tools for future students. <a rel="nofollow" href="https://github.com/jarad/" target="_blank">Jarad</a> started a <a rel="nofollow" href="https://github.com/jarad/project_template" target="_blank">Make-based project template</a>, and I looked for existing solutions. I liked the idea of <a rel="nofollow" href="https://www.gnu.org/software/make/" target="_blank">Make</a>, but I hoped to find something more scalable and friendlier for R-based projects.</p>
<h3>Remake</h3>
<p><center><br>
<img alt="Person pushing a boulder. https://sites.google.com/site/sisyphusa/" src="https://i0.wp.com/ropensci.org/img/blog-images/2018-02-06-drake/flat.png?w=456&ssl=1" data-recalc-dims="1"><br>

<div><em>Person pushing a boulder. <a rel="nofollow" href="https://sites.google.com/site/sisyphusa/" target="_blank">https://sites.google.com/site/sisyphusa/</a></em></div>
<p></p>
<p><a rel="nofollow" href="https://github.com/richfitz" target="_blank">Rich FitzJohn</a>’s <a rel="nofollow" href="https://github.com/richfitz/remake" target="_blank">remake package</a> package was nearly ideal. Almost totally R-focused, <a rel="nofollow" href="https://github.com/richfitz/remake" target="_blank">remake</a> tracked changes more discerningly than <a rel="nofollow" href="https://www.gnu.org/software/make/" target="_blank">Make</a>. However, it lacked high-performance computing support, and it required a cumbersome <a rel="nofollow" href="http://yaml.org/" target="_blank">YAML</a> configuration file to list all the steps of the analysis. So I wrote two sidekick packages: <a rel="nofollow" href="https://github.com/wlandau/parallelRemake" target="_blank">one to deploy jobs in parallel</a> and <a rel="nofollow" href="https://github.com/wlandau/remakeGenerator" target="_blank">another to generate large remake-style YAML files</a>. With <a rel="nofollow" href="https://github.com/richfitz/remake" target="_blank">remake</a> and its add-ons, my post-thesis wrap-up work was steady and smooth, a breeze compared to the thesis itself. Data science projects suddenly became much more fun.</p>
<h3>Drake</h3>
<p><center><br>
<img alt="Hexagon logo. https://github.com/ropensci/drake" src="https://ropensci.org/img/blog-images/2018-02-06-drake/logo.svg"><br>

<div><em>Hexagon logo. <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank">https://github.com/ropensci/drake</a></em></div>
<p></p>
<p>I originally intended to contribute to <a rel="nofollow" href="https://github.com/richfitz/remake" target="_blank">remake</a>. I wanted to inject life back into development, and I wanted to see it on <a rel="nofollow" href="https://cran.r-project.org/" target="_blank">CRAN</a>. However, I was not experienced enough with the problem, and I did not understand <a rel="nofollow" href="https://github.com/richfitz/remake" target="_blank">remake</a>’s internals. I began <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a> as a learning exercise, and it quickly morphed into a beast of its own, friendly and fully scalable. <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>Drake</code></a> is by far my most gratifying project from 2017, and it is still a joy to maintain and proselytize.</p>
<h3>A taste</h3>
<pre><code># install.packages(&quot;drake&quot;)                  # Latest CRAN release, or
# devtools::install_github(&quot;ropensci/drake&quot;) # The development version
library(drake)

# The basic example explores a trend in the mtcars dataset.
load_basic_example()   # Get the code with drake_example(&quot;basic&quot;).
config &lt;- drake_config(my_plan)
outdated(config)        # Which targets need to be (re)built?
make(my_plan)          # Build the right things.
outdated(config)        # Everything is up to date.
reg2 &lt;- function(d){    # Change a dependency.
  d$x3 &lt;- d$x ^ 3
  lm(y ~ x3, data = d)
}
outdated(config)        # Some targets are now out of date.
vis_drake_graph(config) # Interact with the graph. Hover, click, drag...
</code></pre>
<p><iframe width="100%" src="https://cdn.rawgit.com/ropensci/drake/0b76e536/images/reg2.html" height="600px"><br>
</iframe></p>
<h3>Acknowledgements</h3>
<p>I started collaborating with <a rel="nofollow" href="https://github.com/krlmlr" target="_blank">Kirill Müller</a> on <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a> at <a rel="nofollow" href="https://www.rstudio.com/conference/" target="_blank">RStudio::conf(2018)</a>, and we are working to take it to an entirely new level of performance and ease of use. <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>Drake</code></a>’s future is brighter and I am a better software developer because of his coaching sessions over those four days alone.</p>
<p>I would also like to thank Kirill for his <a rel="nofollow" href="https://krlmlr.github.io/drake-pitch/" target="_blank"><code>drake</code> pitch</a> during <a rel="nofollow" href="https://www.rstudio.com/conference/" target="_blank">RStudio::conf(2018)</a> and <a rel="nofollow" href="https://github.com/jennybc" target="_blank">Jenny Bryan</a> for including it in her workshop on <a rel="nofollow" href="https://github.com/jennybc/what-they-forgot" target="_blank">“What They Forgot to Teach You About R”</a>. Their time and generosity boosted <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a>’s presence and popularity overnight.</p>
<p>Many thanks to <a rel="nofollow" href="https://github.com/benmarwick" target="_blank">Ben Marwick</a>, <a rel="nofollow" href="https://github.com/jules32" target="_blank">Julia Lowndes</a>, and <a rel="nofollow" href="https://github.com/gothub" target="_blank">Peter Slaughter</a> for <a rel="nofollow" href="https://github.com/ropensci/onboarding/issues/156" target="_blank">reviewing drake for rOpenSci</a>, and to <a rel="nofollow" href="https://github.com/maelle" target="_blank">Maëlle Salmon</a> for such active involvement and encouragement as the editor.</p>
<p>Thanks also to the following people for contributing early in development.</p>
<ul>
<li><a rel="nofollow" href="https://github.com/AlexAxthelm" target="_blank">Alex Axthelm</a></li>
<li><a rel="nofollow" href="https://github.com/mrchypark" target="_blank">Chan-Yub Park</a></li>
<li><a rel="nofollow" href="https://github.com/dfalster" target="_blank">Daniel Falster</a></li>
<li><a rel="nofollow" href="https://github.com/thercast" target="_blank">Eric Nantz</a></li>
<li><a rel="nofollow" href="https://github.com/HenrikBengtsson" target="_blank">Henrik Bengtsson</a></li>
<li><a rel="nofollow" href="https://github.com/IanAWatson" target="_blank">Ian Watson</a></li>
<li><a rel="nofollow" href="https://github.com/dapperjapper" target="_blank">Jasper Clarkberg</a></li>
<li><a rel="nofollow" href="https://github.com/kendonB" target="_blank">Kendon Bell</a></li>
</ul>
<p>And of course, special thanks to <a rel="nofollow" href="https://github.com/jarad/" target="_blank">Jarad</a>, who originally set me on this path.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://ropensci.org/blog/2018/02/06/drake/"> rOpenSci - open tools for open science</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/Irh-WSxRIsY" height="1"></a></div></center></center></center>