<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/future-reproducible-rngs-future_lapply-and-more/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.jottr.org/2017/02/future-reproducible-rngs-futurelapply.html"> jottR</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=future">future</a> 1.3.0 is available on CRAN. With futures, it is easy to <strong>write R code once</strong>, which the user can choose to evaluate in parallel using whatever resources s/he has available, e.g. a local machine, a set of local machines, a set of remote machines, a high-end compute cluster (via <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=future.BatchJobs">future.BatchJobs</a> and soon also <a target="_blank" rel="nofollow" href="https://github.com/HenrikBengtsson/future.batchtools">future.batchtools</a>), or in the cloud (e.g. via <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=googleComputeEngineR">googleComputeEngineR</a>).</p>
<table align="center" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td><img src="https://i2.wp.com/4.bp.blogspot.com/-oyDTAnMGW8c/WKpLCWos65I/AAAAAAABVEo/OiWVqgYF4Uk-88ldy9VAX6YP93GnLZxaACLcB/s320/funny_car_magnet_animated.gif?w=456&ssl=1" border="0" data-recalc-dims="1"></td>
</tr>
<tr>
<td>Futures makes it easy to harness any resources at hand.</td>
</tr>
</tbody>
</table>
<p> Thanks to great feedback from the community, this new version provides:<br> <ul>
<li><strong>A convenient lapply() function</strong><br> <ul>
<li>Added <code>future_lapply()</code> that works like <code>lapply()</code> and gives identical results with the difference that futures are used internally. Depending on user’s choice of <code>plan()</code>, these calculations may be processed sequential, in parallel, or distributed on multiple machines.</li>
<li>Perfect reproducible random number generation (RNG) is guaranteed given the same initial seed, regardless of the type of futures used and choice of load balancing. Argument <code>future.seed = TRUE</code>(default) will use a random initial seed, which may also be specified as <code>future.seed = &lt;integer></code>. L’Ecuyer-CMRG RNG streams are used internally.</li>
<li>Load balancing can be controlled by argument <code>future.scheduling</code>, which is a scalar adjusting how many futures each worker should process.</li>
</ul>
</li>
<li><strong>Clarifies distinction between developer and end user</strong><br> <ul>
<li>The end user controls what future strategy to use by default, e.g. <code>plan(multiprocess)</code> or <code>plan(cluster, workers = c(&quot;machine1&quot;, &quot;machine2&quot;, &quot;remote.server.org&quot;))</code>.</li>
<li>The developer controls whether futures should be resolved eagerly (default) or lazily, e.g. <code>f &lt;- future(..., lazy = TRUE)</code>. Because of this, <code>plan(lazy)</code> is now deprecated.</li>
</ul>
</li>
<li><strong>Is even more friendly to multi-tenant compute environments</strong><br> <ul>
<li><code>availableCores()</code> returns the number of cores available to the current R process. On a regular machine, this typically corresponds to the number of cores on the machine (<code>parallel::detectCores()</code>). If option <code>mc.cores</code> or environment variable <code>MC_CORES</code> is set, then that will be returned. However, on compute clusters using schedulers such as SGE, Slurm, and TORQUE / PBS, the function detects the number of cores allotted to the job by the scheduler and returns that instead. <strong>This way developers don’t have to adjust their code to match a certain compute environment; the default works everywhere</strong>.</li>
<li>With the new version, it is possible to override the fallback value used when nothing else is specified to not be the number of cores on the machine but to option <code>future.availableCores.fallback</code> or environment variable <code>R_FUTURE_AVAILABLE_FALLBACK</code>. For instance, by using <code>R_FUTURE_AVAILABLE_FALLBACK=1</code>system-wide in HPC environments, any user running outside of the scheduler will automatically use single-core processing unless explicitly requesting more cores. This lowers the risk of overloading the CPU by mistake.</li>
<li>Analogously to how <code>availableCores()</code> returns the number of cores, the new function <code>availableWorkers()</code> returns the host names available to the R process. The default is <code>rep(&quot;localhost&quot;, times = availableCores())</code>, but when using HPC schedulers it may be the host names of other compute notes allocated to the job. </li>
</ul>
</li>
</ul>
<p>For full details on updates, please see the <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=future">NEWS</a> file. <strong>The future package installs out-of-the-box on all operating systems</strong>.<br> <br>
<h2>A quick example</h2>
<p>The bootstrap example of <code>help(&quot;clusterApply&quot;, package = &quot;parallel&quot;)</code> adapted to make use of futures.<br> <pre><code>library(&quot;future&quot;)<br>library(&quot;boot&quot;)<br><br>run &lt;- function(...) {<br>  cd4.rg &lt;- function(data, mle) MASS::mvrnorm(nrow(data), mle$m, mle$v)<br>  cd4.mle &lt;- list(m = colMeans(cd4), v = var(cd4))<br>  boot(cd4, corr, R = 5000, sim = &quot;parametric&quot;, ran.gen = cd4.rg, mle = cd4.mle)<br>}<br><br># base::lapply()<br>system.time(boot &lt;- lapply(1:100, FUN = run))<br>###    user  system elapsed <br>### 133.637   0.000 133.744<br><br># Sequentially on the local machine<br>plan(sequential)<br>system.time(boot0 &lt;- future_lapply(1:100, FUN = run, future.seed = 0xBEEF))<br>###    user  system elapsed <br>### 134.916   0.003 135.039 <br><br># In parallel on the local machine (with 8 cores)<br>plan(multisession)<br>system.time(boot1 &lt;- future_lapply(1:100, FUN = run, future.seed = 0xBEEF))<br>###    user  system elapsed<br>###   0.960   0.041  29.527 <br>stopifnot(all.equal(boot1, boot0))<br></code></pre>
<h2>What’s next?</h2>
<p>The <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=future.BatchJobs">future.BatchJobs</a> package, which builds on top of <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=BatchJobs">BatchJobs</a>, provides future strategies for various HPC schedulers, e.g. SGE, Slurm, and TORQUE / PBS. For example, by using <code>plan(batchjobs_torque)</code> instead of <code>plan(multiprocess)</code> your futures will be resolved distributed on a compute cluster instead of parallel on your local machine. That’s it! However, since last year, the BatchJobs package has been decommissioned and the authors recommend everyone to use their new <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=batchtools">batchtools</a> package instead. Just like BatchJobs, it is a very well written package, but at the same time it is more robust against cluster problems and it also supports more types of HPC schedulers. Because of this, I’ve been working on <a target="_blank" rel="nofollow" href="https://github.com/HenrikBengtsson/future.batchtools">future.batchtools</a> which I hope to be able to release soon.</p>
<p>Finally, I’m really keen on looking into how futures can be used with Shaun Jackman’s <a target="_blank" rel="nofollow" href="https://github.com/sjackman/lambdar">lambdar</a>, which is a proof-of-concept that allows you to execute R code on Amazon’s “serverless” <a target="_blank" rel="nofollow" href="https://aws.amazon.com/lambda/">AWS Lambda</a> framework. My hope is that, in a not too far future (pun not intended*), we’ll be able to resolve our futures on AWS Lambda using <code>plan(aws_lambda)</code>.</p>
<p>Happy futuring!</p>
<p> (*) Alright, I admit, it was intended.<br> <br>
<h2>Links</h2>
<ul>
<li>future package:
<ul>
<li>CRAN page: <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=future">https://cran.r-project.org/package=future</a></li>
<li>GitHub page: <a target="_blank" rel="nofollow" href="https://github.com/HenrikBengtsson/future">https://github.com/HenrikBengtsson/future</a></li>
</ul>
</li>
<li>future.BatchJobs package:
<ul>
<li>CRAN page: <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=future.BatchJobs">https://cran.r-project.org/package=future.BatchJobs</a></li>
<li>GitHub page: <a target="_blank" rel="nofollow" href="https://github.com/HenrikBengtsson/future.BatchJobs">https://github.com/HenrikBengtsson/future.BatchJobs</a></li>
</ul>
</li>
<li>future.batchtools package:
<ul>
<li>CRAN page: N/A</li>
<li>GitHub page: <a target="_blank" rel="nofollow" href="https://github.com/HenrikBengtsson/future.batchtools">https://github.com/HenrikBengtsson/future.batchtools</a></li>
</ul>
</li>
<li>doFuture package (an <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=foreach">foreach</a> adaptor):
<ul>
<li>CRAN page: <a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=doFuture">https://cran.r-project.org/package=doFuture</a></li>
<li>GitHub page: <a target="_blank" rel="nofollow" href="https://github.com/HenrikBengtsson/doFuture">https://github.com/HenrikBengtsson/doFuture</a></li>
</ul>
</li>
</ul>
<h2>See also</h2>
<ul>
<li><a target="_blank" rel="nofollow" href="http://www.jottr.org/2016/07/a-future-for-r-slides-from-user-2016.html">A Future for R: Slides from useR 2016</a>, 2016-07-02</li>
<li><a target="_blank" rel="nofollow" href="http://www.jottr.org/2016/10/remote-processing-using-futures.html">Remote Processing Using Futures</a>, 2016-10-21</li>
</ul>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.jottr.org/2017/02/future-reproducible-rngs-futurelapply.html"> jottR</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/upwkyOSR0lY"></a></div>