<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/are-parallel-simulations-in-the-cloud-worth-it-benchmarking-my-mbp-vs-my-workstation-vs-amazon-ec2/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://rpsychologist.com/benchmark-parallel-sim"> R Psychologist - R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>If you tend to do lots of large Monte Carlo simulations, you’ve probably already discovered the benefits of multi-core CPUs and parallel computation. A simulation that takes 4 weeks without parallelization, can easily be done in 1 week on a quad core laptop with parallelization. However, for even larger simulations reducing the computation time down from e.g. 8 months down to 2 months is where you either, start to think about reducing your simulation study, or getting more CPU power. I prefer the latter option (yeah, I know it’s not always an option…), and often perform larger runs on Amazon EC2 instances, and more daily largeish runs on my workstation. Recently, I saw this post by Max Kuhn: <a rel="nofollow" href="http://appliedpredictivemodeling.com/blog/2018/1/17/parallel-processing" target="_blank">While you wait for that to finish, can I interest you in parallel processing?</a> where he compared computers with 4 to 10 cores. Since I already had a bunch of benchmarks done on Amazon’s EC2 c5.* instances (2 to 72 cores), for an article I’m writing, I figured it’d be interesting to compare those results to more everyday PhD student-level computers. So in this post I compare two different Macbook Pros, a dual CPU workstation, and different EC2 instances. </p>
<p><center><br>
<img src="https://i1.wp.com/rpsychologist.com/img/parallel_benchmark_htop.png?w=456" alt="" data-recalc-dims="1">
<p><em>The joy of running 72 R sessions in parallel.</em></p>
<h2>The Machines</h2>
<p>These are the machines I benchmarked. Logical cores indicate the maximum number of virtual cores, or threads. I didn’t remember to write down how many sockets the Amazon machines had. </p>
<table>
<thead>
<tr>
<th>Machine</th>
<th>CPU</th>
<th>GHz</th>
<th>Cores</th>
<th>Logical Cores</th>
</tr>
</thead>
<tbody>
<tr>
<td>Macbook Pro 15″ Mid 2012</td>
<td>Intel Core i7-3720QM</td>
<td>2.6 GHz</td>
<td>4</td>
<td>8</td>
</tr>
<tr>
<td>Macbook Pro 15″ Mid 2015</td>
<td>Intel Core i7-4980HQ</td>
<td>2.8 GHz</td>
<td>4</td>
<td>8</td>
</tr>
<tr>
<td>HP Z620 Workstation</td>
<td>Xeon E5-2670 (x 2)</td>
<td>2.6 GHz</td>
<td>16</td>
<td>32</td>
</tr>
<tr>
<td>Amazon EC2 c5.large</td>
<td>Xeon Platinum 8124M</td>
<td>3.0 GHz</td>
<td>1</td>
<td>2</td>
</tr>
<tr>
<td>Amazon EC2 c5.xlarge</td>
<td>Xeon Platinum 8124M</td>
<td>3.0 GHz</td>
<td>2</td>
<td>4</td>
</tr>
<tr>
<td>Amazon EC2 c5.2xlarge</td>
<td>Xeon Platinum 8124M</td>
<td>3.0 GHz</td>
<td>4</td>
<td>8</td>
</tr>
<tr>
<td>Amazon EC2 c5.4xlarge</td>
<td>Xeon Platinum 8124M</td>
<td>3.0 GHz</td>
<td>8</td>
<td>16</td>
</tr>
<tr>
<td>Amazon EC2 c5.9xlarge</td>
<td>Xeon Platinum 8124M</td>
<td>3.0 GHz</td>
<td>18</td>
<td>36</td>
</tr>
<tr>
<td>Amazon EC2 c5.18xlarge</td>
<td>Xeon Platinum 8124M</td>
<td>3.0 GHz</td>
<td>36</td>
<td>72</td>
</tr>
</tbody>
</table>
<h2>The Simulation</h2>
<p>I used my package <code>powerlmm</code> <a rel="nofollow" href="https://github.com/rpsychologist/powerlmm" target="_blank">(link)</a> to simulate longitudinally clustered data, with 6 clusters per treatment arm, 50 subjects per clusters, and 11 measures on each subject. So <strong>600 * 11 observations in total</strong>. This three-level model was fit with a random intercept and slope at the subject-level, and a random slope at the cluster level. To make the machines work even harder, I approximated <em>dfs</em> using Satterthwaite’s approximation, using <code>lmerTest</code>. The number of simulations was <strong>5000</strong>.</p>
<p>The Macbooks were running macOS, and my HP workstation and Amazon’s EC2 instances were running Ubuntu 16.04. R 3.4.3 was installed on all machines </p>
<h2>Results</h2>
<p>Unexpectedly, running only 1 core was the slowest, which took between 50 to 80 minutes. The maximum performance of each machine was:</p>
<ul>
<li>MBP (2012) <strong>17 min</strong>.</li>
<li>MPB (2015) <strong>13 min</strong>.</li>
<li>HP Z620 Workstation <strong>5 min</strong>.</li>
<li>Amazon EC2 c5.18xlarge <strong>2 min</strong>.</li>
</ul>
<p>Moreover, running the simulations in a shell tended to be slightly faster then using a GUI (RStudio), and <code>fork</code> and <code>PSOCK</code> clusters were equally fast. However, the time it took to spawn the PSOCK workers is excluded from the benchmark. The figure bellow shows the elapsed time for all machines. </p>
<p><img alt="center" title="center" src="https://i1.wp.com/rpsychologist.com/img/benchmark-parallel-sim/Parallel-simulation-elapsed-time-linear-mixed-effects-models-lme4-1.png?w=456" data-recalc-dims="1"></p>
<p>Speedup was fairly linear for the number of physical cores per machine. The figure bellow shows relative speedup for each machine. </p>
<p><img alt="center" title="center" src="https://i0.wp.com/rpsychologist.com/img/benchmark-parallel-sim/Parallel-simulation-speedup-linear-mixed-effects-models-lme4-1.png?w=456" data-recalc-dims="1"></p>
<h2>So is cloud computing worth it?</h2>
<p>Obviously, having remote access to a 72 core 3 GHz machine is an extremely convenient way to significantly decrease your simulation time. And RStudio Server makes remote R work painless. However, my ~5 year old HP workstation still performs extremely well. So if you have to pay for these things yourself, buying a refurbished workstation is a really good option, especially if you tend to do a lot of simulations. You can probably buy a refurbished Z620 for 600 EUR on Ebay, which is approximately what it costs to run the c5.9xlarge instance for 17 days (on-demand). </p>
<h1>Code</h1>
<p>Lastly, here’s the code I used.</p>
<h2>Simulation functions</h2>
<div>
<pre><span></span><span># filename: funcs.R</span>

<span>#' @param cores no. cores to use</span>
<span>#' Rest of the arguments explained in run_benchmark()</span>
benchmark_sim <span>&lt;-</span> <span>function</span><span>(</span>cores<span>,</span> object<span>,</span> type<span>,</span> label<span>,</span> nsim<span>)</span> <span>{</span>
   <span>message</span><span>(</span><span>&quot;Benchmarking &quot;</span><span>,</span> cores<span>,</span> <span>&quot; cores&quot;</span><span>)</span>
   cl <span>&lt;-</span> parallel<span>::</span>makeCluster<span>(</span>cores<span>,</span> 
                               type <span>=</span> type<span>)</span>
   res <span>&lt;-</span> simulate<span>(</span>p<span>,</span> 
                   cores <span>=</span> cores<span>,</span>
                   nsim <span>=</span> nsim<span>,</span> 
                   satterthwaite <span>=</span> <span>TRUE</span><span>,</span> 
                   cl <span>=</span> cl<span>)</span>
   parallel<span>::</span>stopCluster<span>(</span>cl<span>)</span>
   pow <span>&lt;-</span> <span>summary</span><span>(</span>res<span>)</span><span>$</span><span>summary</span><span>$</span>correct<span>$</span>FE<span>$</span>Power_satt<span>[</span><span>4</span><span>]</span>
   <span>data.frame</span><span>(</span>power <span>=</span> pow<span>,</span> 
              time <span>=</span> res<span>$</span>time<span>,</span>
              cores <span>=</span> cores<span>,</span>
              label <span>=</span> <span>paste</span><span>(</span>label<span>,</span> type<span>,</span> sep <span>=</span> <span>&quot;_&quot;</span><span>))</span>

<span>}</span>

<span>#' @param object simulation_parameters()-object</span>
<span>#' @param max_cores benchmark 1, ..., max_cores</span>
<span>#' @param type type of cluster to use</span>
<span>#' @param label a label for the experiment</span>
<span>#' @param nsim number of simulations</span>
<span>#' @param type FORK or PSOCK</span>
run_benchmark <span>&lt;-</span> <span>function</span><span>(</span>object<span>,</span> max_cores<span>,</span> nsim<span>,</span> label<span>,</span> type <span>=</span> <span>NULL</span><span>)</span> <span>{</span>
  <span>if</span><span>(</span><span>is.null</span><span>(</span>type<span>))</span> <span>{</span>
    <span># only use forking in non-interactive session</span>
    <span>if</span><span>(</span><span>interactive</span><span>())</span> <span>{</span>
      type <span>&lt;-</span> <span>&quot;PSOCK&quot;</span>
    <span>}</span> <span>else</span> <span>{</span>
      type <span>&lt;-</span> <span>&quot;FORK&quot;</span>
    <span>}</span>
  <span>}</span>

  res <span>&lt;-</span> <span>lapply</span><span>(</span><span>1</span><span>:</span>max_cores<span>,</span> 
                benchmark_sim<span>,</span> 
                object <span>=</span> object<span>,</span>
                type <span>=</span> type<span>,</span>
                label <span>=</span> label<span>,</span>
                nsim <span>=</span> nsim<span>)</span>
  res <span>&lt;-</span> <span>do.call</span><span>(</span><span>rbind</span><span>,</span> res<span>)</span>

  <span># save</span>
  fname <span>&lt;-</span> <span>paste0</span><span>(</span><span>unique</span><span>(</span>res<span>$</span>label<span>),</span> <span>&quot;_shell.rds&quot;</span><span>)</span>
  <span>saveRDS</span><span>(</span>res<span>,</span> file <span>=</span> fname<span>)</span>
<span>}</span>
</pre>
</div>
<h2>Run benchmark</h2>
<div>
<pre><span></span><span>source</span><span>(</span><span>&quot;funcs.R&quot;</span><span>)</span>
<span>library</span><span>(</span>powerlmm<span>)</span>
<span>library</span><span>(</span>methods<span>)</span>


<span># setup sim</span>
p <span>&lt;-</span> study_parameters<span>(</span>n1 <span>=</span> <span>11</span><span>,</span>
                      n2 <span>=</span> <span>50</span><span>,</span>
                      n3 <span>=</span> <span>6</span><span>,</span>
                      fixed_intercept <span>=</span> <span>37</span><span>,</span>
                      fixed_slope <span>=</span> <span>-0.64</span><span>,</span>
                      sigma_subject_intercept <span>=</span> <span>2.8</span><span>,</span>
                      sigma_subject_slope <span>=</span> <span>0.4</span><span>,</span>
                      sigma_cluster_intercept <span>=</span> <span>0</span><span>,</span>
                      cor_subject <span>=</span> <span>-0.5</span><span>,</span>
                      icc_slope <span>=</span> <span>0.05</span><span>,</span>
                      sigma_error <span>=</span> <span>2.6</span><span>,</span>
                      dropout <span>=</span> dropout_weibull<span>(</span>proportion <span>=</span> <span>0.3</span><span>,</span> 
                                                rate <span>=</span> <span>1</span><span>/</span><span>2</span><span>),</span>
                      cohend <span>=</span> <span>-0.5</span><span>)</span>

<span># will save results to wd</span>
run_benchmark<span>(</span>p<span>,</span> 
              max_cores <span>=</span> <span>8</span><span>,</span> 
              nsim <span>=</span> <span>5000</span><span>,</span>
              label <span>=</span> <span>&quot;AWS_c5.18xlarge&quot;</span><span>,</span>
              type <span>=</span> <span>&quot;FORK&quot;</span><span>)</span>
</pre>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://rpsychologist.com/benchmark-parallel-sim"> R Psychologist - R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/d1D204JUO74" height="1"></a></div></center>