<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/the-many-faced-future/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.jottr.org/2017/06/the-many-faced-future.html"> jottR</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>The <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=future">future</a> package defines the Future API, which is a unified, generic, friendly API for parallel processing. The Future API follows the principle of <strong>write code once and run anywhere</strong> – the developer chooses what to parallelize and the user how and where. </p>
<p>The nature of a future is such that it lends itself to be used with several of the existing map-reduce frameworks already available in R. In this post, I’ll give an example of how to apply a function over a set of elements concurrently using plain sequential R, the parallel package, the <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=future">future</a> package alone, as well as future in combination of the <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=foreach">foreach</a>, the <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=plyr">plyr</a>, and the <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=purrr">purrr</a> packages. </p>
<p> <br>
<table align="center" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td><img src="https://i1.wp.com/4.bp.blogspot.com/-O6a-wA8-H00/WTZGh7s2pkI/AAAAAAABYEU/oTJ5JGB6TTQdR8-GStVt3sCB7zBFPrWFwCLcB/s320/julia_sets.gif?w=320&ssl=1" border="0" data-recalc-dims="1"></td>
</tr>
<tr>
<td>You can choose your own future and what you want to do with it.</td>
</tr>
</tbody>
</table>
<h2>Example: Multiple Mandelbrot sets</h2>
<p>The <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=Julia">Julia</a> package provides the <code>JuliaImage()</code> function for generating a <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Julia_set">Julia set</a> for a given set of start parameters <code>(centre, L, C)</code> where <code>centre</code> specify the center point in the complex plane, <code>L</code> specify the width and height of the square region around this location, and <code>C</code> is a complex coefficient controlling the “shape” of the generated Julia set. For example, to generate one of the above Julia set images (1000-by-1000 pixels), you can use:
<pre><code>library(&quot;Julia&quot;)<br>set &lt;- JuliaImage(1000, centre = 0 + 0i, L = 3.5, C = -0.4 + 0.6i)<br>plot_julia(set)<br></code></pre>
<p>with
<pre><code>plot_julia &lt;- function(img, col = topo.colors(16)) {<br>  par(mar = c(0, 0, 0, 0))<br>  image(img, col = col, axes = FALSE)<br>}<br></code></pre>
<p>For the purpose of illustrating how to calculate different Julia sets in parallel, I will use the same <code>(centre, L) = (0 + 0i, 3.5)</code> region as above with the following ten complex coeffients (from <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Julia_set">Julia set</a>):
<pre><code>Cs &lt;- list(<br>  a = -0.618   + 0i,<br>  b = -0.4     + 0.6i,<br>  c =  0.285   + 0i,<br>  d =  0.285   + 0.01i,<br>  e =  0.45    + 0.1428i,<br>  f = -0.70176 - 0.3842i,<br>  g =  0.835   - 0.2321i,<br>  h = -0.8     + 0.156i,<br>  i = -0.7269  + 0.1889i,<br>  j =  0       - 0.8i<br>)<br></code></pre>
<p>Now we’re ready to see how we can use futures in combination of different map-reduce implementations in R for generating these ten sets in parallel. Note that all approaches will generate the exact same ten Julia sets. So, feel free to pick your favorite approach.<br>
<h2>Sequential</h2>
<p>To process the above ten regions sequentially, we can use the <code>lapply()</code> function:
<pre><code>library(&quot;Julia&quot;)<br>sets &lt;- lapply(Cs, function(C) {<br>  JuliaImage(1000, centre = 0 + 0i, L = 3.5, C = C)<br>})<br></code></pre>
<h2>Parallel</h2>
<pre><code>library(&quot;parallel&quot;)<br>ncores &lt;- future::availableCores() ## a friendly version of detectCores()<br>cl &lt;- makeCluster(ncores)<br><br>clusterEvalQ(cl, library(&quot;Julia&quot;))<br>sets &lt;- parLapply(cl, Cs, function(C) {<br>  JuliaImage(1000, centre = 0 + 0i, L = 3.5, C = C)<br>})<br></code></pre>
<h2>Futures (in parallel)</h2>
<pre><code>library(&quot;future&quot;)<br>plan(multisession)  ## defaults to availableCores() workers<br><br>library(&quot;Julia&quot;)<br>sets &lt;- future_lapply(Cs, function(C) {<br>  JuliaImage(1000, centre = 0 + 0i, L = 3.5, C = C)<br>})<br></code></pre>
<p>We could also have used the more explicit setup <code>plan(cluster, workers = makeCluster(availableCores()))</code>, which is identical to <code>plan(multisession)</code>.<br>
<h2>Futures with foreach</h2>
<pre><code>library(&quot;doFuture&quot;)<br>registerDoFuture()  ## tells foreach futures should be used<br>plan(multisession)  ## specifies what type of futures<br><br>library(&quot;Julia&quot;)<br>sets &lt;- foreach(C = Cs) %dopar% {<br>  JuliaImage(1000, centre = 0 + 0i, L = 3.5, C = C)<br>}<br></code></pre>
<p>Note that I didn’t pass <code>.packages = &quot;Julia&quot;</code> to <code>foreach()</code> because the doFuture backend will do that automatically for us – that’s one of the treats of using futures. If we would have used <code>doParallel::registerDoParallel(cl)</code> or similar, we would have had to worry about that.<br>
<h2>Futures with plyr</h2>
<p>The plyr package will utilize foreach internally if we pass <code>.parallel = TRUE</code>. Because of this, we can use <code>plyr::llply()</code> to parallelize via futures as follows:
<pre><code>library(&quot;plyr&quot;)<br>library(&quot;doFuture&quot;)<br>registerDoFuture()  ## tells foreach futures should be used<br>plan(multisession)  ## specifies what type of futures<br><br>library(&quot;Julia&quot;)<br>sets &lt;- llply(Cs, function(C) {<br>  JuliaImage(1000, centre = 0 + 0i, L = 3.5, C = C)<br>}, .parallel = TRUE)<br></code></pre>
<p>For the same reason as above, also here we don’t have to worry about global variables and making sure needed packages are attached; that’s all handles by the future packages.<br>
<h2>Futures with purrr (= furrr)</h2>
<p>As a final example, here is how you can use futures to parallelize your <code>purrr::map()</code> calls:
<pre><code>library(&quot;purrr&quot;)<br>library(&quot;future&quot;)<br>plan(multisession)<br><br>library(&quot;Julia&quot;)<br>sets &lt;- Cs %>%<br>        map(~ future(JuliaImage(1000, centre = 0 + 0i, L = 3.5, C = .x))) %>%<br>        values<br></code></pre>
<p> <i>Comment:</i> This latter approach will not perform load balance (“scheduling”) across backend workers; that’s a feature that ideally would be taken care of by purrr itself. However, I have some ideas for future versions of future (pun…) that may achieve this without having to modify the purrr package.<br>
<h1>Got compute?</h1>
<p>If you have access to one or more machines with R installed (e.g. a local or remote cluster, or a <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=googleComputeEngineR">Google Compute Engine cluster</a>), and you’ve got direct SSH access to those machines, you can have those machines calculate the above Julia sets for you; just change future plan, e.g.
<pre><code>plan(cluster, workers = c(&quot;machine1&quot;, &quot;machine2&quot;, &quot;machine3.remote.org&quot;))<br></code></pre>
<p>If you have access to a high-performance compute (HPC) cluster with a HPC scheduler (e.g. Slurm, TORQUE / PBS, LSF, and SGE), you can harness its power by switching to:
<pre><code>library(&quot;future.batchtools&quot;)<br>plan(batchtools_sge)<br></code></pre>
<p>For more details, see the vignettes of the <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=future.batchtools">future.batchtools</a> and <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=batchtools">batchtools</a> packages.<br> <p>Happy futuring! </p>
<p> <br>
<h2>Links</h2>
<ul>
<li>future package:
<ul>
<li>CRAN page: <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=future">https://cran.r-project.org/package=future</a></li>
<li>GitHub page: <a target="_blank" rel="nofollow" href="https://github.com/HenrikBengtsson/future">https://github.com/HenrikBengtsson/future</a></li>
</ul>
</li>
<li>future.batchtools package:
<ul>
<li>CRAN page: <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=future.batchtools">https://cran.r-project.org/package=future.batchtools</a></li>
<li>GitHub page: <a target="_blank" rel="nofollow" href="https://github.com/HenrikBengtsson/future.batchtools">https://github.com/HenrikBengtsson/future.batchtools</a></li>
</ul>
</li>
<li>doFuture package (an <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=foreach">foreach</a> adaptor):
<ul>
<li>CRAN page: <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=doFuture">https://cran.r-project.org/package=doFuture</a></li>
<li>GitHub page: <a target="_blank" rel="nofollow" href="https://github.com/HenrikBengtsson/doFuture">https://github.com/HenrikBengtsson/doFuture</a></li>
</ul>
</li>
</ul>
<h2>See also</h2>
<ul>
<li><a target="_blank" rel="nofollow" href="http://www.jottr.org/2016/07/a-future-for-r-slides-from-user-2016.html">A Future for R: Slides from useR 2016</a>, 2016-07-02</li>
<li><a target="_blank" rel="nofollow" href="http://www.jottr.org/2016/10/remote-processing-using-futures.html">Remote Processing Using Futures</a>, 2016-10-21</li>
<li><a target="_blank" rel="nofollow" href="http://www.jottr.org/2017/02/future-reproducible-rngs-futurelapply.html">future: Reproducible RNGs, future_lapply() and more</a>, 2017-02-19</li>
<li><a target="_blank" rel="nofollow" href="http://www.jottr.org/2017/03/dofuture-universal-foreach-adapator.html">doFuture: A universal foreach adaptor ready to be used by 1,000+ packages</a>, 2017-03-18</li>
</ul>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.jottr.org/2017/06/the-many-faced-future.html"> jottR</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...

	<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/E52vVdQo99M"></a></div>