<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/future-apply-parallelize-any-base-r-apply-function/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://www.jottr.org/2018/06/23/future.apply_1.0.0/"> JottR on R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img alt="0% to 100% utilization" src="https://i1.wp.com/www.jottr.org/post/future.apply_1.0.0-htop_32cores.png?w=456&ssl=1" data-recalc-dims="1"><br>
<em>Got compute?</em></p>
<p><a rel="nofollow" href="https://cran.r-project.org/package=future.apply" target="_blank">future.apply</a> 1.0.0 – <em>Apply Function to Elements in Parallel using Futures</em> – is on CRAN. With this milestone release, all<sup>*</sup> base R apply functions now have corresponding futurized implementations. This makes it easier than ever before to parallelize your existing <code>apply()</code>, <code>lapply()</code>, <code>mapply()</code>, … code – just prepend <code>future_</code> to an apply call that takes a long time to complete. That’s it! The default is sequential processing but by using <code>plan(multiprocess)</code> it’ll run in parallel.</p>
<p>
<em>Table: All future_nnn() functions in the <strong>future.apply</strong> package. Each function takes the same arguments as the corresponding <strong>base</strong> function does.</em></p>
<table>
<thead>
<tr>
<th>Function</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>future_<strong>apply()</strong></code></td>
<td>Apply Functions Over Array Margins</td>
</tr>
<tr>
<td><code>future_<strong>lapply()</strong></code></td>
<td>Apply a Function over a List or Vector</td>
</tr>
<tr>
<td><code>future_<strong>sapply()</strong></code></td>
<td>– “ –</td>
</tr>
<tr>
<td><code>future_<strong>vapply()</strong></code></td>
<td>– “ –</td>
</tr>
<tr>
<td><code>future_<strong>replicate()</strong></code></td>
<td>– “ –</td>
</tr>
<tr>
<td><code>future_<strong>mapply()</strong></code></td>
<td>Apply a Function to Multiple List or Vector Arguments</td>
</tr>
<tr>
<td><code>future_<strong>Map()</strong></code></td>
<td>– “ –</td>
</tr>
<tr>
<td><code>future_<strong>eapply()</strong></code></td>
<td>Apply a Function Over Values in an Environment</td>
</tr>
<tr>
<td><code>future_<strong>tapply()</strong></code></td>
<td>Apply a Function Over a Ragged Array</td>
</tr>
</tbody>
</table>
<p><sup>*</sup> <code>future_<strong>rapply()</strong></code> – Recursively Apply a Function to a List – is yet to be implemented.</p>
<h2>A Motivating Example</h2>
<p>In the <strong>parallel</strong> package there is an example – in <code>?clusterApply</code> – showing how to perform bootstrap simulations in parallel. After some small modifications to clarify the steps, it looks like the following:</p>
<pre><code>library(parallel)
library(boot)

run1 &lt;- function(...) {
   library(boot)
   cd4.rg &lt;- function(data, mle) MASS::mvrnorm(nrow(data), mle$m, mle$v)
   cd4.mle &lt;- list(m = colMeans(cd4), v = var(cd4))
   boot(cd4, corr, R = 500, sim = &quot;parametric&quot;,
        ran.gen = cd4.rg, mle = cd4.mle)
}

cl &lt;- makeCluster(4) ## Parallelize using four cores
clusterSetRNGStream(cl, 123)
cd4.boot &lt;- do.call(c, parLapply(cl, 1:4, run1))
boot.ci(cd4.boot, type = c(&quot;norm&quot;, &quot;basic&quot;, &quot;perc&quot;),
                  conf = 0.9, h = atanh, hinv = tanh)
stopCluster(cl)
</code></pre>
<p>The script defines a function <code>run1()</code> that produces 500 bootstrap samples, and then it calls this function four times, combines the four replicated samples into one <code>cd4.boot</code>, and at the end it uses <code>boot.ci()</code> to summarize the results.</p>
<p>The corresponding sequential implementation would look something like:</p>
<pre><code>library(boot)

run1 &lt;- function(...) {
   cd4.rg &lt;- function(data, mle) MASS::mvrnorm(nrow(data), mle$m, mle$v)
   cd4.mle &lt;- list(m = colMeans(cd4), v = var(cd4))
   boot(cd4, corr, R = 500, sim = &quot;parametric&quot;,
        ran.gen = cd4.rg, mle = cd4.mle)
}

set.seed(123)
cd4.boot &lt;- do.call(c, lapply(1:4, run1))
boot.ci(cd4.boot, type = c(&quot;norm&quot;, &quot;basic&quot;, &quot;perc&quot;),
                  conf = 0.9, h = atanh, hinv = tanh)
</code></pre>
<p>We notice a few things about these two code snippets. First of all, in the parallel code, there are two <code>library(boot)</code> calls; one in the main code and one inside the <code>run1()</code> function. The reason for this is to make sure that the <strong>boot</strong> package is also attached in the parallel, background R session when <code>run1()</code> is called there. The <strong>boot</strong> package defines the <code>boot.ci()</code> function, as well as the <code>boot()</code> function and the <code>cd4</code> data.frame – both used inside <code>run1()</code>. If <strong>boot</strong> is not attached inside the function, we would get an error on <code>&quot;object 'cd4' not found&quot;</code> when running the parallel code. In contrast, we do not need to do this in the sequential code. Also, if we later would turn our parallel script into a package, then <code>R CMD check</code> would complain if we kept the <code>library(boot)</code> call inside the <code>run1()</code> function.</p>
<p>Second, the example uses <code>MASS::mvrnorm()</code> in <code>run1()</code>. The reason for this is related to the above – if we use only <code>mvrnorm()</code>, we need to attach the <strong>MASS</strong> package using <code>library(MASS)</code> and also do so inside <code>run1()</code>. Since there is only one <strong>MASS</strong> function called, it’s easier and neater to use the form <code>MASS::mvrnorm()</code>.</p>
<p>Third, the random-seed setup differs between the sequential and the parallel approach.</p>
<p>In summary, in order to turn the sequential script into a script that parallelizes using the <strong>parallel</strong> package, we would have to not only rewrite parts of the code but also be aware of important differences in order to avoid getting run-time errors due to missing packages or global variables.</p>
<p>One of the objectives of the <strong>future.apply</strong> package, and the <strong>future</strong> ecosystem in general, is to make transitions from writing sequential code to writing parallel code as simple and frictionless as possible.</p>
<p>Here is the same example parallelized using the <strong>future.apply</strong> package:</p>
<pre><code>library(future.apply)
plan(multiprocess, workers = 4) ## Parallelize using four cores
library(boot)

run1 &lt;- function(...) {
   cd4.rg &lt;- function(data, mle) MASS::mvrnorm(nrow(data), mle$m, mle$v)
   cd4.mle &lt;- list(m = colMeans(cd4), v = var(cd4))
   boot(cd4, corr, R = 500, sim = &quot;parametric&quot;,
        ran.gen = cd4.rg, mle = cd4.mle)
}

set.seed(123)
cd4.boot &lt;- do.call(c, future_lapply(1:4, run1, future.seed = TRUE))
boot.ci(cd4.boot, type = c(&quot;norm&quot;, &quot;basic&quot;, &quot;perc&quot;),
                  conf = 0.9, h = atanh, hinv = tanh)
</code></pre>
<p>The difference between the sequential base-R implementation and the <strong>future.apply</strong> implementation is minimal. The <strong>future.apply</strong> package is attached, the parallel plan of four workers is set up, and the <code>apply()</code> function is replaced by <code>future_apply()</code>, where we specify <code>future.seed = TRUE</code> to get statistical sound and numerically reproducible parallel random number generation (RNG).<br>
More importantly, notice how there is no need to worry about which packages need to be attached on the workers and which global variables need to be exported. That is all taken care of automatically by the <strong>future</strong> framework.</p>
<h2>Q&amp;A</h2>
<p>Q. <em>What are my options for parallelization?</em><br>
A. Everything in <strong>future.apply</strong> is processed through the <a rel="nofollow" href="https://cran.r-project.org/package=future" target="_blank">future</a> framework. This means that all parallelization backends supported by the <strong>parallel</strong> package are supported out of the box, e.g. on your <strong>local machine</strong>, and on <strong>local</strong> or <strong>remote</strong> ad-hoc <strong>compute clusters</strong> (also in the <strong>cloud</strong>). Additional parallelization and distribution schemas are provided by backends such as <strong><a rel="nofollow" href="https://cran.r-project.org/package=future.callr" target="_blank">future.callr</a></strong> (parallelization on your local machine) and <strong><a rel="nofollow" href="https://cran.r-project.org/package=future.batchtools" target="_blank">future.batchtools</a></strong> (large-scale parallelization via <strong>HPC job schedulers</strong>). For other alternatives, see the CRAN Page for the <strong><a rel="nofollow" href="https://cran.r-project.org/package=future" target="_blank">future</a></strong> package and the <a rel="nofollow" href="https://cran.r-project.org/web/views/HighPerformanceComputing.html" target="_blank">High-Performance and Parallel Computing with R</a> CRAN Task View.</p>
<p>Q. <em>Righty-oh, so how do I specify which parallelization backend to use?</em><br>
A. A fundamental design pattern of the future framework is that <em>the end user decides <strong>how and where</strong> to parallelize</em> while <em>the developer decides <strong>what</strong> to parallelize</em>. This means that you do <em>not</em> specify the backend via some argument to the <code>future_nnn()</code> functions. Instead, the backend is specified by the <code>plan()</code> function – you can almost think of it as a global option that the end user controls. For example, <code>plan(multiprocess)</code> will parallelize on the local machine, so will <code>plan(future.callr::callr)</code>, whereas <code>plan(cluster, workers = c(&quot;n1&quot;, &quot;n2&quot;, &quot;remote.server.org&quot;))</code> will parallelize on two local machines and one remote machine. Using <code>plan(future.batchtools::batchtools_sge)</code> will distribute the processing on your SGE-supported compute cluster. BTW, you can also have <a rel="nofollow" href="https://cran.r-project.org/web/packages/future/vignettes/future-3-topologies.html" target="_blank">nested parallelization strategies</a>, e.g. <code>plan(list(tweak(cluster, workers = nodes), multiprocess))</code> where <code>nodes = c(&quot;n1&quot;, &quot;n2&quot;, &quot;remote.server.org&quot;)</code>.</p>
<p>Q. <em>What about load balancing?</em><br>
A. The default behavior of all functions is to distribute <strong>equally-sized chunks</strong> of elements to each available background worker – such that each worker process exactly one chunk (= one future). If the processing times vary significantly across chunks, you can increase the average number of chunks processed by each worker, e.g. to have them process two chunks on average, specify <code>future.scheduling = 2.0</code>. Alternatively, you can specify the number of elements processed per chunk, e.g. <code>future.chunk.size = 10L</code> (an analog to the <code>chunk.size</code> argument added to the <strong>parallel</strong> package in R 3.5.0).</p>
<p>Q. <em>What about random number generation (RNG)? I’ve heard it’s tricky to get right when running in parallel.</em><br>
A. Just add <code>future.seed = TRUE</code> and you’re good. This will use <strong>parallel safe</strong> and <strong>statistical sound</strong> <strong>L’Ecuyer-CMRG RNG</strong>, which is a well-established parallel RNG algorithm and used by the <strong>parallel</strong> package. The <strong>future.apply</strong> functions use this in a way that is also <strong>invariant to</strong> the future backend and the amount of “chunking” used. To produce numerically reproducible results, set <code>set.seed(123)</code> before (as in the above example), or simply use <code>future.seed = 123</code>.</p>
<p>Q. <em>What about global variables? Whenever I’ve tried to parallelize code before, I often ran into errors on “this or that variable is not found”.</em><br>
A. This is very rarely a problem when using the <a rel="nofollow" href="https://cran.r-project.org/package=future" target="_blank">future</a> framework – things work out of the box. <strong>Global variables and packages</strong> needed are <strong>automatically identified</strong> from static code inspection and passed on to the workers – even when the workers run on remote computers or in the cloud.</p>
<p><em>Happy futuring!</em></p>
<h2>Links</h2>
<ul>
<li>future.apply package:
<ul>
<li>CRAN page: <a rel="nofollow" href="https://cran.r-project.org/package=future.apply" target="_blank">https://cran.r-project.org/package=future.apply</a></li>
<li>GitHub page: <a rel="nofollow" href="https://github.com/HenrikBengtsson/future.apply" target="_blank">https://github.com/HenrikBengtsson/future.apply</a></li>
</ul>
</li>
<li>future package:
<ul>
<li>CRAN page: <a rel="nofollow" href="https://cran.r-project.org/package=future" target="_blank">https://cran.r-project.org/package=future</a></li>
<li>GitHub page: <a rel="nofollow" href="https://github.com/HenrikBengtsson/future" target="_blank">https://github.com/HenrikBengtsson/future</a></li>
</ul>
</li>
<li>future.batchtools package:
<ul>
<li>CRAN page: <a rel="nofollow" href="https://cran.r-project.org/package=future.batchtools" target="_blank">https://cran.r-project.org/package=future.batchtools</a></li>
<li>GitHub page: <a rel="nofollow" href="https://github.com/HenrikBengtsson/future.batchtools" target="_blank">https://github.com/HenrikBengtsson/future.batchtools</a></li>
</ul>
</li>
<li>doFuture package:
<ul>
<li>CRAN page: <a rel="nofollow" href="https://cran.r-project.org/package=doFuture" target="_blank">https://cran.r-project.org/package=doFuture</a></li>
<li>GitHub page: <a rel="nofollow" href="https://github.com/HenrikBengtsson/doFuture" target="_blank">https://github.com/HenrikBengtsson/doFuture</a></li>
</ul>
</li>
</ul>
<h2>See also</h2>
<ul>
<li><a rel="nofollow" href="https://www.jottr.org/2018/06/23/future.apply_1.0.0/2018/06/18/future-erum2018-slides/" target="_blank">Delayed Future(Slides from eRum 2018)</a>, 2018-06-19</li>
<li><a rel="nofollow" href="https://www.jottr.org/2018/04/12/future-results/" target="_blank">future 1.8.0: Preparing for a Shiny Future</a>, 2018-04-12</li>
<li><a rel="nofollow" href="https://www.jottr.org/2017/06/05/many-faced-future/" target="_blank">The Many-Faced Future</a>, 2017-06-05</li>
<li><a rel="nofollow" href="https://www.jottr.org/2017/02/19/future-rng/" target="_blank">future 1.3.0 Reproducible RNGs, future_lapply() and More</a>, 2017-02-19</li>
<li><a rel="nofollow" href="https://www.jottr.org/2016/10/22/future-hpc/" target="_blank">High-Performance Compute in R Using Futures</a>, 2016-10-22</li>
<li><a rel="nofollow" href="https://www.jottr.org/2016/10/11/future-remotes/" target="_blank">Remote Processing Using Futures</a>, 2016-10-11</li>
</ul>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.jottr.org/2018/06/23/future.apply_1.0.0/"> JottR on R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/ZtAURVnibu8" height="1"></a></div>