<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/run-massive-parallel-r-jobs-cheaply-with-updated-doazureparallel-package/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2017/06/doazureparallel-updated.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>At the EARL conference in San Francisco this week, JS Tan from Microsoft gave an update (<a rel="nofollow" href="http://blog.revolutionanalytics.com/downloads/doAzureParallel%20presentation.pdf" target="_blank">PDF slides here</a>) on the <a rel="nofollow" href="https://github.com/Azure/doAzureParallel" target="_blank">doAzureParallel package</a> . As we've noted <a rel="nofollow" href="http://blog.revolutionanalytics.com/2017/03/doazureparallel.html" target="_blank">here before</a>, this package allows you to easily distribute parallel R computations to an Azure cluster. The package was <a rel="nofollow" href="https://azure.microsoft.com/en-us/blog/run-massive-r-jobs-in-azure-directly-from-r-studio-at-a-fraction-of-the-price" target="_blank">recently updated</a> to support using automatically-scaling <a rel="nofollow" href="https://azure.microsoft.com/en-us/services/batch/" target="_blank">Azure Batch clusters</a> with low-priority nodes, which can be used at a <a rel="nofollow" href="https://azure.microsoft.com/en-us/pricing/details/batch/" target="_blank">discount of up to 80%</a> compared to the price of regular high-availability VMs.</p>
<blockquote class="twitter-tweet" data-lang="en">
<p dir="ltr" lang="en">JS Tan using doAzureParallel <a rel="nofollow" href="https://twitter.com/hashtag/rstats?src=hash" target="_blank">#rstats</a> package to run simulation on a cluster of 20 low-priority Azure VMs. Total cost: $0.02 <a rel="nofollow" href="https://twitter.com/hashtag/EARLConf2017?src=hash" target="_blank">#EARLConf2017</a> <a rel="nofollow" href="https://t.co/Mpl3IUa9zY" target="_blank">pic.twitter.com/Mpl3IUa9zY</a></p>
<p>— David Smith (@revodavid) <a rel="nofollow" href="https://twitter.com/revodavid/status/872589986490482688" target="_blank">June 7, 2017</a></p></blockquote>
<p></p>
<p><a rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d28ace3d970c-pi" target="_blank"><img alt="Azurepool" title="Azurepool" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d28ace3d970c-200wi"></a>Using the doAzureParallel package is simple. First, you need to define the cluster you're going to use <a rel="nofollow" href="https://github.com/Azure/doAzureParallel/blob/master/README.md#configuration-json-files" target="_blank">as a JSON file</a>. (You can see an example on the right.) Here, you'll specify your Azure credentials, the size of the cluster, and the type of nodes (CPUs and memory) to use in the cluster. You can also specify here R packages (from CRAN and/or Github) to be pre-loaded onto each node, and the maximum number of simultaneous tasks to run on each node (for within-node parallelism).</p>
<p>New to this update, the <a rel="nofollow" href="https://github.com/Azure/doAzureParallel/blob/master/docs/11-autoscale.md" target="_blank">poolSize option</a> allows you to specify the number of dedicated (standard) VM nodes to use, in addition to a number of low-priority nodes to use. Low-priority nodes can be pre-empted by the Azure system at any time, but are much cheaper to use. (Even if a node is pre-empted your parallel computation will be continue; it will just take a little longer with the reduced capacity.) You can even specify a minimum and maximum number of nodes of each class to use, in which case the cluster will <strong>automatically scale</strong> up and down according to either (your choice) the workload or the time of day (e.g. only expand the low-priority part of the cluster on weekends, when pre-emption is less likely). </p>
<p><a rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d28ac4a6970c-pi" target="_blank"><img border="0" alt="Azurebatch" title="Azurebatch" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d28ac4a6970c-800wi"></a><br>Once you've defined the parameters of your cluster, all you need to do is declare the cluster as a backend for the <a rel="nofollow" href="https://mran.microsoft.com/package/foreach/" target="_blank">foreach package</a>. The body of the <code>foreach</code> loop runs just like a <code>for</code> loop, except that multiple iterations run in parallel on the remote cluster. Here are the key parts of the <a rel="nofollow" href="https://github.com/Azure/doAzureParallel/blob/master/samples/montecarlo_pricing_simulation.R" target="_blank">option price simulation</a> example JS presented at the conference.</p>
<p></p>
<p>This same approach can be used for any &quot;embarrassingly parallel&quot; iteration in R, and you can use any R function or package within the body of the loop. For example, you could use a cluster to reduce the time required for <a rel="nofollow" href="http://github.com/Azure/doAzureParallel/blob/master/samples/caret_example.R" title="" target="_blank">parameter tuning and cross-validation with the caret package,</a> or speed up <a rel="nofollow" href="https://github.com/Azure/doAzureParallel/blob/master/samples/plyr_example.R" target="_blank">data preparation tasks when using the dplyr package</a>.</p>
<p>In addition to support for auto-scaling clusters, this update to doAzureParallel also includes a few other new features. You'll also find new utility functions for managing multiple <a rel="nofollow" href="http://github.com/Azure/doAzureParallel#long-running-jobs--job-management" title="" target="_blank">long-running R jobs,</a> functions to read data from and write data to <a rel="nofollow" href="http://github.com/Azure/doAzureParallel/blob/master/samples/sas_resource_files_example.R" title="" target="_blank">Azure Blob storage</a>, and the ability to <a rel="nofollow" href="https://github.com/Azure/doAzureParallel/blob/master/docs/21-distributing-data.md#pre-loading-data-into-the-cluster" target="_blank">pre-load data into the cluster</a> by specifying resource files.</p>
<p>The doAzureParallel package is available for download now from Github, under the open-source MIT license. For details on how to use the package, check out the <a rel="nofollow" href="https://github.com/Azure/doAzureParallel/blob/master/README.md" target="_blank">README</a> and the <a rel="nofollow" href="https://github.com/Azure/doAzureParallel/tree/master/docs" target="_blank">doAzureParallel guide</a>.</p>
<p>Github (Azure): <a rel="nofollow" href="https://github.com/azure/doazureparallel" target="_blank">doAzureParallel</a></p>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2017/06/doazureparallel-updated.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...

	<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/WxPI-XdX80g" height="1"></a></div>