<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/massively-parallel-computations-on-azure-clusters-with-r-made-easy-with-doazureparallel/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2017/03/doazureparallel.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p><em>by JS Tan (Program Manager, Microsoft)</em></p>
<p>For users of the R language, scaling up their work to take advantage of cloud-based computing has generally been a complex undertaking. We are therefore excited to announce <a target="_blank" rel="nofollow" href="https://www.github.com/azure/doazureparallel">doAzureParallel</a>, a lightweight R package built on <a target="_blank" rel="nofollow" href="https://azure.microsoft.com/en-us/services/batch/">Azure Batch</a> that allows you to easily use Azure’s flexible compute resources right from your R session. The doAzureParallel package complements <a target="_blank" rel="nofollow" href="https://www.microsoft.com/en-us/cloud-platform/r-server">Microsoft R Server</a> and provides the infrastructure you need to run massively parallel simulations on Azure directly from R.</p>
<p>The doAzureParallel package is a parallel backend for the popular <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/foreach/">foreach package</a>, making it possible to execute multiple processes across a cluster of Azure virtual machines with just a few lines of R code. The package helps you create and manage the cluster in Azure, and register it as a parallel backend to be used with foreach.</p>
<p><a target="_blank" rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c8e3a0f4970b-pi"><img title="DoAzureParallel" alt="DoAzureParallel" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c8e3a0f4970b-800wi" border="0"></a></p>
<p>With doAzureParallel, there is no need to manually create, configure and manage a cluster of individual VMs. Running your scale jobs is as easy as running algorithms on your local machine. With Azure Batch’s autoscaling capabilities, you can also increase or decrease your cluster size to fit your workloads, saving you time and money. doAzureParallel also uses the <a target="_blank" rel="nofollow" href="https://azuremarketplace.microsoft.com/en-us/marketplace/apps/microsoft-ads.standard-data-science-vm">Azure Data Science Virtual Machine</a><strong> (DSVM),</strong> allowing Azure Batch to easily and quickly configure the appropriate environment in as little time as possible.</p>
<p>doAzureParallel is ideal for running embarrassingly parallel work such as parametric sweeps or Monte Carlo simulations, making it a great fit for many financial modelling algorithms (back-testing, portfolio scenario modelling, etc).</p>
<p>The doAzureParallel is available for download on Github under the open-source MIT license, and there is no additional cost for these capabilities – you only pay for the Azure VMs you use.</p>
<p><strong>Performance Gains with doAzureParallel</strong></p>
<p>To illustrate the kind of speed up you can get with doAzureParallel, we did a test that compares the <a target="_blank" rel="nofollow" href="https://github.com/Azure/doAzureParallel/blob/master/samples/mandlebrot_performance_test.ipynb">performance of computing the Mandelbrot set</a> on a single machine of 2 cores, a 5-node clusters of 10 cores, a 10-node cluster of 20 cores, and finally a 20-node cluster of 40 cores.</p>
<div><a target="_blank" rel="nofollow" href="http://a5.typepad.com/6a0105360ba1c6970c01bb0986d555970d-pi"><img title="Speed" alt="Speed" src="http://a5.typepad.com/6a0105360ba1c6970c01bb0986d555970d-400wi"></a>
<div>Time in seconds for computing the Mandelbrot set</div>
</div>
<p>From the graph, we can see that using doAzureParallel on Azure can get you a speed up of about <strong>2X with a cluster of 5 nodes</strong>, a speed up of about <strong>3X with a cluster of 10 nodes</strong>, and a speed up of about <strong>6X with a cluster of 20 nodes</strong>. From a cost perspective, running the 20-node cluster for about 50 seconds with 6X performance ended up costing only $0.03 when using a cluster of Standard_F2 Linux VMs. (This was for the WestUS region. Pricing varies by region and can change over time.)</p>
<p>The doAzureParallel package is <a target="_blank" rel="nofollow" href="https://github.com/Azure/doAzureParallel">available now on Github</a>, where you can also find <a target="_blank" rel="nofollow" href="https://github.com/Azure/doAzureParallel/blob/master/docs/README.md">detailed documentation</a>. For more detailed information, including installation steps and demo code, check out the announcement at the Microsoft Azure blog linked below. </p>
<p>Microsoft Azure blog: <a target="_blank" rel="nofollow" href="https://azure.microsoft.com/en-us/blog/doazureparallel/">doAzureParallel: Take advantage of Azure’s flexible compute directly from your R session</a></p>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2017/03/doazureparallel.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/Lp1uEOT0H2Q"></a></div>