The drake R package is a pipeline toolkit. It manages data science workflows, saves time, and adds more confidence to reproducibility. I hope it will impact the landscapes of reproducible research and high-performance computing, but I originally created it for different reasons. This post is the prequel to drake’s inception. There was struggle, and drake was the answer.
Dissertation frustration
Sisyphus. https://sites.google.com/site/sisyphusa/
My dissertation project was intense. The final computational challenge was to analyze multiple genomics datasets using an emerging method and its competitors. Even with GPU computing, which shrank days of runtime down to hours, the full battery of Markov chain Monte Carlo runs took several weeks from start to finish. I organized my workflow as an R package, and I worked in a loop:
Deploy the computations.
Wait a few weeks.
Discover an issue.
Restart from scratch.
At the time, the dominant R-focused workflow managers could not break the cycle. Knitr was designed to weave together code and prose, and its paradigm deliberately lacked enough modularity to properly scale. ProjectTemplate was mostly concerned with organization and readability. These and similar tools had only traces of the functionality I sought. For the right solution, I needed to step off the beaten path.
GNU Make
paper.pdf: paper.tex figure.png pdflatex paper.tex
figure.png: figure.R results.csv
Rscript figure.R
results.csv: long-computation.R
Rscript long-computation.R
GNU Make is a dependency watcher first and foremost. Its top priority is to bring results up to date with as little work as possible, and it gives you parallel computing for free. In fact, according to Karl Broman and others, Make surpasses even knitr as a helpful reproducible research tool.
My advisor, Jarad Niemi, repeatedly urged me to use Make. At the time, I was too entrenched in half-written code to transition, so I finished my graduate school work with brute force. I defended my thesis, moved out of Iowa, and because I had goofed up the paperwork, remained a graduate student for one last summer. Jarad and I used most of our remaining time to find and create better tools for future students. Jarad started a Make-based project template, and I looked for existing solutions. I liked the idea of Make, but I hoped to find something more scalable and friendlier for R-based projects.
Remake
Person pushing a boulder. https://sites.google.com/site/sisyphusa/
Rich FitzJohn’s remake package package was nearly ideal. Almost totally R-focused, remake tracked changes more discerningly than Make. However, it lacked high-performance computing support, and it required a cumbersome YAML configuration file to list all the steps of the analysis. So I wrote two sidekick packages: one to deploy jobs in parallel and another to generate large remake-style YAML files. With remake and its add-ons, my post-thesis wrap-up work was steady and smooth, a breeze compared to the thesis itself. Data science projects suddenly became much more fun.
Drake
Hexagon logo. https://github.com/ropensci/drake
I originally intended to contribute to remake. I wanted to inject life back into development, and I wanted to see it on CRAN. However, I was not experienced enough with the problem, and I did not understand remake’s internals. I began drake as a learning exercise, and it quickly morphed into a beast of its own, friendly and fully scalable. Drake is by far my most gratifying project from 2017, and it is still a joy to maintain and proselytize.
A taste
# install.packages(&quot;drake&quot;) # Latest CRAN release, or
# devtools::install_github(&quot;ropensci/drake&quot;) # The development version
library(drake)
# The basic example explores a trend in the mtcars dataset.
load_basic_example() # Get the code with drake_example(&quot;basic&quot;).
config