<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/speed-up-simulations-in-r-with-doazureparallel/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2018/01/doazureparallel-simulations.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>I’m a big fan using <a rel="nofollow" href="https://mran.microsoft.com/documents/what-is-r" target="_blank">R</a> to simulate data. When I’m trying to understand a data set, my first step is sometimes to simulate data from a model and compare the results to the data, before I go down the path of fitting an analytical model directly. Simulations are easy to code in R, but they can sometimes take a while to run — especially if there are a bunch of parameters you want to explore, which in turn requires a <em>bunch</em> of simulations.</p>
<blockquote class="twitter-tweet" data-partner="tweetdeck">
<p dir="ltr" lang="en">When your pc only has four core processors and your parallel processing across three of the.<a rel="nofollow" href="https://twitter.com/hashtag/DataScience?src=hash&ref_src=twsrc%5Etfw" target="_blank">#DataScience</a> <a rel="nofollow" href="https://twitter.com/hashtag/RStats?src=hash&ref_src=twsrc%5Etfw" target="_blank">#RStats</a> <a rel="nofollow" href="https://twitter.com/hashtag/Waiting?src=hash&ref_src=twsrc%5Etfw" target="_blank">#Waiting</a> <a rel="nofollow" href="https://twitter.com/hashtag/Multitasking?src=hash&ref_src=twsrc%5Etfw" target="_blank">#Multitasking</a> <a rel="nofollow" href="https://t.co/iVkkr7ibox" target="_blank">pic.twitter.com/iVkkr7ibox</a></p>
<p>— Patrick Williams (@unbalancedDad) <a rel="nofollow" href="https://twitter.com/unbalancedDad/status/956308800750764032?ref_src=twsrc%5Etfw" target="_blank">January 24, 2018</a></p></blockquote>
<p></p>
<p>In this post, I’ll provide a simple example of running multiple simulations in R, and show how you can speed up the process by running the simulations in parallel: either on your own machine, or on a cluster of machines in the Azure cloud using the <a rel="nofollow" href="https://github.com/Azure/doAzureParallel" target="_blank">doAzureParallel</a> package.</p>
<p>To demonstrate this, let’s use a simple simulation example: the <a rel="nofollow" href="https://en.wikipedia.org/wiki/Birthday_problem" target="_blank">birthday problem</a>. Simply stated, the goal is to calculate for a room of \(N\) people the probability that someone in the room shares a birthday with someone else in the room. Now, you can calculate this probability analytically — R even has a <a rel="nofollow" href="https://www.rdocumentation.org/packages/stats/versions/3.4.3/topics/birthday?tap_a=5644-dce66f&tap_s=10907-287229" target="_blank">function for it</a> — but this one of those situations where it’s quicker for me to write a simulation that it would be to figure out the analytical result. (Better yet, my simulation accounts for February 29 birthdays, which the standard result doesn’t. Take that, distribution analysis!) Here’s an R function that simulates 10,000 rooms, and counts the number of times a room of <code>n</code> people includes a shared birthday:</p>
<p></p>
<p>You can compare the results to the built-in function <code>pbirthday</code> to make sure it’s working, though you should include the <code>feb29=FALSE</code> option for an apples-to-apples comparison. The more simulations (<code>nsims</code>) you use, the closer the results will be.</p>
<p>We want to find the number of people in the room where the probability of a match is closest to 50%. We’re not exactly sure what that number is, but we can fund out by calculating the probability for a range of room sizes, plotting the results, and see where the probability crosses 0.50. Here’s a simple for loop that calculates the probability for room sizes from 1 to 100 people:</p>

<div>
<pre>bdayp <span>&lt;-</span> <span>1</span><span>:</span><span>100</span>
<span>for</span> (n <span>in</span> <span>1</span><span>:</span><span>100</span>) bdayp[n] <span>&lt;-</span> pbirthdaysim(n)
plot(bdayp, xlab<span>=</span><span>&quot;People in room&quot;</span>, 
            ylab<span>=</span><span>&quot;Probability of shared birthday&quot;</span>)
abline(h<span>=</span><span>0.5</span>)
</pre>
</div>
<p><a rel="nofollow" href="http://a4.typepad.com/6a0105360ba1c6970c01bb09ec71b4970d-pi" target="_blank"><img border="0" alt="Birthday" title="Birthday" src="http://a4.typepad.com/6a0105360ba1c6970c01bb09ec71b4970d-800wi"></a></p>
<p>If you look at where the horizontal line crosses the curve, you can see that with 23 people in the room there’s a 50% probability of a match. With more than 60 people, a match is practically guaranteed.</p>
<p>On my SurfaceBook (16Gb RAM, 2 2.6Gz cores) using R 3.4.3, that simulation takes about 6 minutes (361 seconds). And even though I have 2 cores, according to Task Manager the CPU utilization hovered at little over 50%, because R — as a single-threaded application — was using just one core. (The operating system and my other apps accounted for the rest of the usage.)</p>
<p>So here’s one opportunity to speed things up: we could run two R sessions on my laptop at the same time, each doing half the simulations, and then combine the results at the end. Fortunately, we don’t have to break up our simulation manually: there are R packages to streamline that process. One such package is the built-in R package <a rel="nofollow" href="https://stat.ethz.ch/R-manual/R-devel/library/parallel/doc/parallel.pdf" target="_blank">parallel</a>, which provides parallel equivalents to the “apply” family of functions. Here though, I’m going to use the <a rel="nofollow" href="http://blog.revolutionanalytics.com/2018/01/cda.ms/6Q" target="_blank"><code>foreach</code> function</a>, which is similar to a <code>for</code> loop and — with the <code><a rel="nofollow" href="https://www.rdocumentation.org/packages/foreach/versions/1.4.4/topics/foreach?tap_a=5644-dce66f&tap_s=10907-287229" target="_blank">%dopar%</a></code> operator — runs iterations in parallel. (You can find the <a rel="nofollow" href="https://cran.r-project.org/package=foreach" target="_blank">foreach package</a> on CRAN.) On my laptop, I can use the <a rel="nofollow" href="https://mran.microsoft.com/package/doParallel" target="_blank">doParallel package</a> to make <code>foreach</code> use the aforementioned parallel package in the background to run two iterations at a time.</p>
<p></p>
<p>This time, my CPU utilization stayed above 90% across both cores (while the fans whirred madly), and the simulation took 316 seconds. Not a huge improvement, but definitely faster. (To be fair, I was also on a video call while the simulation was running, which itself consumed quite a bit of computer CPU.)  Which suggests the question: can we go faster if we use more than one computer (and ideally ones dedicated to R and not other desktop stuff)? The answer, as you might have guessed, is yes.</p>
<p>The main problem here is getting access to a cluster: not many of us have a rack of machines to ourselves. But in this age of cloud computing, we can rent one for as long as we like, and for small simulations this can be pretty cheap, too. With the <a rel="nofollow" href="https://github.com/Azure/doAzureParallel" target="_blank">doAzureParallel</a> package I can spin up a cluster of any desired size and power, and use the same <code>foreach</code> code as above to run my simulation on it, all controlled from my local R installation.</p>
<p>To use doAzureParallel, you’ll need the following:</p>
<ul>
<li>An Azure account (if you don’t have one, a new <a rel="nofollow" href="http://blog.revolutionanalytics.com/2018/01/cda.ms/6N" target="_blank">free account</a> includes $200 in credits), and </li>
<li>An <a rel="nofollow" href="http://blog.revolutionanalytics.com/2018/01/cda.ms/6D" target="_blank">Azure Storage</a> service and an <a rel="nofollow" href="http://blog.revolutionanalytics.com/2018/01/cda.ms/6C" target="_blank">Azure Batch</a> service set up in your account, and they associated keys provided in a <code>credentials.json</code> file. (The <a rel="nofollow" href="https://github.com/Azure/doAzureParallel/blob/master/README.md" target="_blank">instructions</a> explain how to do this, and it only takes a few minutes to set up.) Azure Batch is the cluster management service we’ll be using to create our clusters for the R simulations.</li>
</ul>
<p>It can take a little while to boot the cluster, but once it’s up and running you can use it as much as you like. And best of all, you can launch the simulation on the cluster from your local laptop using almost exactly same code as before:</p>
<div>
<pre>library(doAzureParallel)
setCredentials(<span>&quot;credentials.json&quot;</span>)
cluster <span>&lt;-</span> makeCluster(<span>&quot;cluster.json&quot;</span>)
registerDoAzureParallel(cluster)
bdayp <span>&lt;-</span> foreach(n<span>=</span><span>1</span><span>:</span><span>100</span>) <span>%dopar%</span> pbirthdaysim(n)
</pre>
</div>
<p>(That’s why I used foreach instead of the parallel package for the local analysis above: you can change the “backend” for the computation without changing the <code>foreach</code> statement itself.) Using a cluster of 8 2-code nodes (as specified in <a rel="nofollow" href="https://gist.github.com/revodavid/30879bfd46f916379ec6b2b81549e224" target="_blank">this cluster.json file</a>), the simulation took just 54-seconds. That doesn’t include spinning up the cluster itself (the first 4 lines above), but does include sending the jobs to the nodes, waiting for tasks to complete, and merging the results (which happens in line 5). Of course, once you’ve launched a cluster you can re-use it instantly, and you can even resize it dynamically from R to provide more (or less) computing power.</p>
<p>In my local region that particular virtual machine (VM) type currently <a rel="nofollow" href="https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/" target="_blank">costs US$0.10 an hour</a>. (There are <a rel="nofollow" href="https://github.com/Azure/doAzureParallel/blob/master/docs/10-vm-sizes.md#vm-size-table" target="_blank">many VM sizes available</a>.) We used 8 VMs for about a minute, so the total cost of that simulation was less than 2 cents. And we can make things even cheaper — up to 80% discount! — by using <a rel="nofollow" href="http://blog.revolutionanalytics.com/2018/01/cda.ms/6P" target="_blank">pre-emptable VMs</a>: you can add some (or all!) to your cluster with the <code>lowPriorityNodes</code> field in the <a rel="nofollow" href="https://gist.github.com/revodavid/6a7cb047d151fce97cac8887552b8d97" target="_blank">cluster.json file</a>. Using low-priority VMs doesn’t affect VM performance, but Azure may shut down some of them if surplus capacity runs out. If this happens when you’re using doAzureParallel you might not even notice, however: it will automatically resubmit iterations to remaining nodes or wait for new VMs. You’ll still get the same results, but it may possibly take longer than usual.</p>
<p>Now, simulating the birthday problem is a particularly simple example — deliberately so. Most importantly, it doesn’t require any data to be moved to the cluster, which is an important consideration for performance. (We’ll look at that issue in a later blog post.) Our simulation also doesn’t use any specialized R packages that need to be installed on the cluster. As it happens, in the cluster specification we set things up to use the <a rel="nofollow" href="https://hub.docker.com/r/rocker/tidyverse/" target="_blank">tidyverse Docker image</a> from <a rel="nofollow" href="https://github.com/rocker-org/rocker" target="_blank">Rocker</a>, so most packages we’d need were already there. But if you want to use a different image for the nodes, or need to ship a custom R package or startup script to each node, you can specify that in the <code>cluster.json</code> file as well.</p>
<p>If you’d like to try out the doAzureParallel package yourself, you can install it and find setup instructions and documentation at the Github repository below.</p>
<p>Github (Azure): <a rel="nofollow" href="https://github.com/Azure/doAzureParallel" target="_blank">doAzureParallel</a></p>
<p> </p>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2018/01/doazureparallel-simulations.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/CI99rCOAaHg" height="1"></a></div>