<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/drakes-improved-high-performance-computing-power/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ropensci.org/technotes/2018/05/18/drake-hpc/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>The <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code> R package</a> is not only a reproducible research solution, but also a serious high-performance computing engine. The <a rel="nofollow" href="https://ropensci.github.io/drake/articles/drake.html" target="_blank">Get Started page</a> introduces <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a>, and this technical note draws from the guides on <a rel="nofollow" href="https://ropensci.github.io/drake/articles/parallelism.html" target="_blank">high-performance computing</a> and <a rel="nofollow" href="https://ropensci.github.io/drake/articles/timing.html" target="_blank">timing</a>.</p>
<h3>You can help!</h3>
<p>Some of these features are brand new, and others are newly refactored. The <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank">GitHub version</a> has all the advertised functionality, but it needs more testing and development before I can submit it to CRAN in good conscience. New issues such as <a rel="nofollow" href="http://github.com/r-lib/processx/issues/113" target="_blank">r-lib/processx#113</a> and <a rel="nofollow" href="https://github.com/HenrikBengtsson/future/issues/226" target="_blank">HenrikBengtsson/future#226</a> seem to affect <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a>, and more may emerge. If you use <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a> for your own work, please consider supporting the project by field-testing the claims below and posting feedback <a rel="nofollow" href="https://github.com/ropensci/drake/issues/369" target="_blank">here</a>.</p>
<h3>Let drake schedule your targets.</h3>
<p>A typical workflow is a sequence of interdependent data transformations. Consider the example from the <a rel="nofollow" href="https://ropensci.github.io/drake/articles/drake.html" target="_blank">Get Started page</a>.</p>
<p><iframe width="100%" src="https://cdn.rawgit.com/ropensci/drake/8685d94d/docs/images/pitch1.html" height="600px"><br>
</iframe></p>
<p>When you call <code>make()</code> on this project, <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a> takes care of <code>&quot;raw_data.xlsx&quot;</code>, then <code>raw_data</code>, and then <code>data</code> in sequence. Once <code>data</code> completes, <code>fit</code> and <code>hist</code> can launch in parallel, and then <code>&quot;report.md&quot;</code> begins once everything else is done. It is <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a>’s responsibility to deduce this order of execution, hunt for ways to parallelize your work, and free you up to focus on the substance of your research.</p>
<h3>Activate parallel processing.</h3>
<p>Simply set the <code>jobs</code> argument to an integer greater than 1. The following <code>make()</code> recruits multiple processes on your local machine.</p>
<pre><code>make(plan, jobs = 2)
</code></pre>
<p>For parallel deployment to a computing cluster (<a rel="nofollow" href="https://slurm.schedmd.com/" target="_blank">SLURM</a>, <a rel="nofollow" href="http://www.adaptivecomputing.com/products/open-source/torque/" target="_blank">TORQUE</a>, <a rel="nofollow" href="http://www.univa.com/products/" target="_blank">SGE</a>, etc.) <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a> calls on packages <a rel="nofollow" href="https://github.com/HenrikBengtsson/future" target="_blank"><code>future</code></a>, <a rel="nofollow" href="https://github.com/mllg/batchtools" target="_blank"><code>batchtools</code></a>, and <a rel="nofollow" href="https://github.com/HenrikBengtsson/future.batchtools" target="_blank"><code>future.batchtools</code></a>. First, create a <a rel="nofollow" href="https://github.com/mllg/batchtools/tree/master/inst/templates" target="_blank"><code>batchtools</code> template file</a> to declare your resource requirements and <a rel="nofollow" href="http://modules.sourceforge.net/" target="_blank">environment modules</a>. There are built-in example files in <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a>, but you will likely need to tweak your own by hand.</p>
<pre><code>drake_batchtools_tmpl_file(&quot;slurm&quot;) # Writes batchtools.slurm.tmpl.
</code></pre>
<p>Next, tell <a rel="nofollow" href="https://github.com/HenrikBengtsson/future.batchtools" target="_blank"><code>future.batchtools</code></a> to talk to the cluster.</p>
<pre><code>library(future.batchtools)
future::plan(batchtools_slurm, template = &quot;batchtools.slurm.tmpl&quot;)
</code></pre>
<p>Finally, set <code>make()</code>’s <code>parallelism</code> argument equal to <code>&quot;future&quot;</code> or <code>&quot;future_lapply&quot;</code>.</p>
<pre><code>make(plan, parallelism = &quot;future&quot;, jobs = 8)
</code></pre>
<h3>Choose a scheduling algorithm.</h3>
<p>The <code>parallelism</code> argument of <code>make()</code> controls not only where to deploy the workers, but also how to schedule them. The following table categorizes the 7 options.</p>
<table>
<thead>
<tr>
<th></th>
<th align="center">Deploy: local</th>
<th align="right">Deploy: remote</th>
</tr>
</thead>
<tbody>
<tr>
<td><b>Schedule: persistent</b></td>
<td align="center">“mclapply”, “parLapply”</td>
<td align="right">“future_lapply”</td>
</tr>
<tr>
<td><b>Schedule: transient</b></td>
<td align="center"></td>
<td align="right">“future”, “Makefile”</td>
</tr>
<tr>
<td><b>Schedule: staged</b></td>
<td align="center">“mclapply_staged”, “parLapply_staged”</td>
<td align="right"></td>
</tr>
</tbody>
</table>
<p></p>
<h4>Staged scheduling</h4>
<p><a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a>’s first custom parallel algorithm was staged scheduling. It was easier to implement than the other two, but the workers run in lockstep. In other words, all the workers pick up their targets at the same time, and each worker has to finish its target before any worker can move on. The following animation illustrates the concept.</p>
<div>
<br>
<iframe width="450" src="https://www.powtoon.com/embed/dQKbGttIYud/"></iframe><br>
</div>
<p>But despite weak parallel efficiency, staged scheduling remains useful because of its low overhead. Without the bottleneck of a formal master process, staged scheduling blasts through armies of tiny conditionally independent targets (<a rel="nofollow" href="https://ropensci.github.io/drake/articles/parallelism.html#staged-scheduling" target="_blank">example here</a>). Consider it if the bulk of your work is finely diced and <a rel="nofollow" href="https://en.wikipedia.org/wiki/Embarrassingly_parallel" target="_blank">perfectly parallel</a>, maybe if your dependency graph is tall and thin.</p>
<div>
<iframe width="100%" src="https://cdn.rawgit.com/ropensci/drake/cdacd23e/docs/images/staged.html" height="600px"><br>
</iframe>
</div>
<h4>Persistent scheduling</h4>
<p>Persistent scheduling is brand new to <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a>. Here, <code>make(jobs = 2)</code> deploys three processes: two workers and one master. Whenever a worker is idle, the master assigns it the next target whose dependencies are fully ready. The workers keep running until no more targets remain. See the animation below.</p>
<p><center><br>
<br>
<iframe width="450" src="https://www.powtoon.com/embed/bUfSIaXjrw5/"></iframe><br>
<br>
</center></p>
<h4>Transient scheduling</h4>
<p>If the time limits of your cluster are too strict for persistent workers, consider transient scheduling, another new arrival. Here, <code>make(jobs = 2)</code> starts a brand new worker for each individual target. See the following video.</p>
<div>
<br>
<iframe width="450" src="https://www.powtoon.com/embed/cHIdOqudELB/"></iframe><br>
</div>
<h3>How many jobs should you choose?</h3>
<p>The <code>predict_runtime()</code> function can help. Let’s revisit the <a rel="nofollow" href="https://ropensci.github.io/drake/articles/example-mtcars.html" target="_blank"><code>mtcars</code> example</a>.</p>
<div>
<iframe width="100%" src="https://cdn.rawgit.com/ropensci/drake/cdacd23e/docs/images/outdated.html" height="600px"><br>
</iframe>
</div>
<p>Let’s also</p>
<ol>
<li>Plan for non-staged scheduling,</li>
<li>Assume each non-file target (black circle) takes 2 hours to build, and</li>
<li>Rest assured that everything else is super quick.</li>
</ol>
<p>When we declare the runtime assumptions with the <code>known_times</code> argument and cycle over a reasonable range of <code>jobs</code>, <code>predict_runtime()</code> paints a clear picture.</p>
<div>
<img alt="times" src="https://i0.wp.com/cdn.rawgit.com/ropensci/drake/21c41083/images/times.png?w=456&ssl=1" data-recalc-dims="1">
</div>
<p><code>jobs = 4</code> is a solid choice. Any fewer would slow us down, and the next 2-hour speedup would take double the <code>jobs</code> and the hardware to back it up. Your choice of <code>jobs</code> for <code>make()</code> ultimately depends on the runtime you can tolerate and the computing resources at your disposal.</p>
<h3>Thanks!</h3>
<p>When I attended <a rel="nofollow" href="https://www.rstudio.com/conference/" target="_blank"><code>RStudio::conf(2018)</code></a>, <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a> relied almost exclusively on staged scheduling. <a rel="nofollow" href="https://github.com/krlmlr" target="_blank">Kirill Müller</a> spent hours on site and hours afterwards helping me approach the problem and educating me on <a rel="nofollow" href="https://en.wikipedia.org/wiki/Priority_queue" target="_blank">priority queues</a>, <a rel="nofollow" href="https://en.wikipedia.org/wiki/Message_queue" target="_blank">message queues</a>, and the <a rel="nofollow" href="https://en.wikipedia.org/wiki/Knapsack_problem" target="_blank">knapsack problem</a>. His generous help paved the way for <a rel="nofollow" href="https://github.com/ropensci/drake" target="_blank"><code>drake</code></a>’s latest enhancements.</p>
<h3>Disclaimer</h3>
<p>This post is a product of my own personal experiences and opinions and does not necessarily represent the official views of my employer. I created and embedded the <a rel="nofollow" href="https://www.powtoon.com/" target="_blank">Powtoon</a> videos only as explicitly permitted in the <a rel="nofollow" href="https://www.powtoon.com/terms-and-conditions/" target="_blank">Terms and Conditions of Use</a>, and I make no copyright claim to any of the constituent graphics.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://ropensci.org/technotes/2018/05/18/drake-hpc/"> rOpenSci - open tools for open science</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/h1_TZNUV62w" height="1"></a></div>