<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/while-you-wait-for-that-to-finish-can-i-interest-you-in-parallel-processing/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://appliedpredictivemodeling.com/blog/2018/1/17/parallel-processing"> Blog - Applied Predictive Modeling</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><code>caret</code> has been able to <a rel="nofollow" href="https://topepo.github.io/caret/parallel-processing.html" target="_blank">utilize parallel processing</a> for some time (before it was on CRAN in October 2007) using slightly different versions of the package. Around September of 2011, <code>caret</code> started using the <a rel="nofollow" href="https://cran.r-project.org/package=foreach" target="_blank"><code>foreach</code></a> package was used to “harmonize” the parallel processing technologies thanks to a super smart guy named Steve Weston. </p>
<p>I’ve done a few benchmarks to quantify the benefits to parallel processing over the years. I just did another set since I just got a new iMacPro. This time, I put the code and results in a <a rel="nofollow" href="https://github.com/topepo/par-caret-bench" target="_blank">GitHub repo</a>. The main questions that I wanted to answer were:</p>
<ul>
<li>Is the benefit the same across computers and operating systems?</li>
<li>Is there any performance difference between parallel processing technologies (or packages)?</li>
<li>For my test cases, is it better to parallelize resampling loops or the individual model fit?</li>
</ul>
<p>The task I used was to tune a boosted classification tree model (via the <a rel="nofollow" href="https://cran.r-project.org/package=xgboost" target="_blank"><code>xgboost</code></a>) package over 25 random values of the seven tuning parameters. The data were simulated and contained 2,000 training set points and 115 numeric predictors. Five repeats of 10-fold cross-validation was used so that 1,250 separate models could be run in parallel. An example of the code is <a rel="nofollow" href="https://github.com/topepo/par-caret-bench/blob/master/macOS/2017%20iMacPro1%2C1%203%20GHz%20Intel%20Xeon%20W%2010core/xgb_01.R" target="_blank">here</a>. </p>
<p>I tested this using three machines:</p>
<ul>
<li>A 2015 MacBookPro (version 11,4) with a 2.5 GHz Intel Core i7 processor. It has 4 physical cores.</li>
<li>A <a rel="nofollow" href="https://pcpartpicker.com/list/4xzqr7" target="_blank">DIY machine</a> built in January of 2016 using a 3.4 GHz Intel Core i7 processor (6 cores).</li>
<li>A 2017 iMacPro (1,1) with a 3 GHz Intel Xeon W processor. There are 10 cores.</li>
</ul>
<p>All the computers have >=8GB of memory. The DIY machine was running Ubuntu 16.04 as well as Windows 10. All of the builds used R 3.4.2 or higher. </p>
<p>The methods for parallelization were:</p>
<ul>
<li><strong>forking</strong> uses the <code>fork</code> system call to create new instances of the session (i.e. workers) that run parts of the computations. This is available on MacOS, unix, and linux. It is available to <code>foreach</code> via the <code>doMC</code> and <code>doParallel</code> packages. Historically, I have used <code>doMC</code> for my work but both were tested here.</li>
</ul>
<ul>
<li>
<p><strong>sockets</strong> via the <code>doParallel</code> package. I don’t know a whole lot about socket connections but they have been around a long time via the <code>snow</code> package and can run on all major operating systems.</p>
</li>
<li>
<p><strong>threading</strong>: <code>xgboost</code> has the option to use <a rel="nofollow" href="http://www.openmp.org/" target="_blank"><code>openMP</code></a> to parallelize the individual model fits.</p>
</li>
</ul>
<p>This <a rel="nofollow" href="https://www.jstatsoft.org/article/view/v031i01" target="_blank">JSS paper from 2009</a> might be helpful. </p>
<p>Once nice thing about parallelism in <code>caret</code> is that you do not have to modify the package code. With <code>foreach</code>, you simply “register” a parallel backend and the <code>caret</code> functions automatically utilize the workers. </p>
<p>To be clear, the forking/socket approaches split the 1,250 model fits across the workers while the threading affects each individual model fit (one-at-a-time). </p>
<p>In each case, different number of workers were applied to the problems and the execution time was measured. Since these are all recent Intel processors, <a rel="nofollow" href="https://en.wikipedia.org/wiki/Hyper-threading" target="_blank">hyper-threading</a> can be used. This means that, even though you have <em>X</em> physical cores, there are <em>2X</em> “logical cores” that can perform independent operations. For that reason, we can add more workers than physical cores. The order of all runs were randomized and run in clear R sessions.</p>
<p>First, let’s look at the non-threading results. The raw execution times were:</p>
<p> <img data-image-dimensions="1536x864" data-image-focal-point="0.5,0.5" src="https://i2.wp.com/static1.squarespace.com/static/51156277e4b0b8b2ffe11c00/t/5a600246f9619a7f88fcf9a3/1516241500926/par-plot-1.png?w=456&ssl=1" alt="par-plot-1.png" data-load="false" data-type="image" data-image-id="5a600246f9619a7f88fcf9a3" data-recalc-dims="1" data-image="https://static1.squarespace.com/static/51156277e4b0b8b2ffe11c00/t/5a600246f9619a7f88fcf9a3/1516241500926/par-plot-1.png"></p>
<p>When converted to <em>speed-ups</em>, we get:</p>
<p> <img data-image-dimensions="1536x864" data-image-focal-point="0.5,0.5" src="https://i1.wp.com/static1.squarespace.com/static/51156277e4b0b8b2ffe11c00/t/5a60026b9140b7408de4caf8/1516241525571/par-plot-su-1.png?w=456&ssl=1" alt="par-plot-su-1.png" data-load="false" data-type="image" data-image-id="5a60026b9140b7408de4caf8" data-recalc-dims="1" data-image="https://static1.squarespace.com/static/51156277e4b0b8b2ffe11c00/t/5a60026b9140b7408de4caf8/1516241525571/par-plot-su-1.png"></p>
<p>The speed-up is the sequential time divided by the parallel time; a speed-up of 3 means that the parallel execution took a third less time than the sequential code. </p>
<p>My observations:</p>
<ul>
<li>As expected, the desktops were substantially faster than the laptop. The speeds also rank-ordered with the age of the computer.</li>
<li>On the same machine, linux ran faster than windows by slightly less than a fold.</li>
<li>It appears that the speed-ups are <em>linear</em> up until we go beyond the number of physical cores. There is incremental benefit to using logical cores but it is small.</li>
<li>I’ve never seen a case where performance get’s substantially worse as you increase the number of workers. If the model is already fast (e.g. PLS) or your workers exceed system memory, this isn’t an issue.</li>
<li>There was no real difference in performance between forking and sockets (at least using <code>doParallel</code>).</li>
<li>The new iMacPro does really well. The speed-ups are linear up until about 8 or 9 cores. This means that an execution time of 38 minutes can be cut down to 4 or 5 minutes.</li>
</ul>
<p>The <code>doMC</code> results are strange. When the number of workers is a multiple of 5, there is a performance hit. At first I thought this was an aberration until it was apparent that it occurred over two operating systems and three computers. Also, it is specific to <code>doMC</code>; forking using <code>doParallel</code> does not have the same issue. I’ve notified the <code>doMC</code> package maintainer about it. </p>
<p>What about threading? These runs had <code>foreach</code> run sequentially and allowed <code>xgb.train</code> to run in parallel by adjusting the <code>nthread</code> argument. The results:</p>
<p> <img data-image-dimensions="1344x1344" data-image-focal-point="0.5,0.5" src="https://i2.wp.com/static1.squarespace.com/static/51156277e4b0b8b2ffe11c00/t/5a6002850d9297648e7ffcd1/1516241550862/threads-1.png?w=456&ssl=1" alt="threads-1.png" data-load="false" data-type="image" data-image-id="5a6002850d9297648e7ffcd1" data-recalc-dims="1" data-image="https://static1.squarespace.com/static/51156277e4b0b8b2ffe11c00/t/5a6002850d9297648e7ffcd1/1516241550862/threads-1.png"></p>
<p>This isn’t so great and really surprising (or fair). We are starting and stopping threads 1,250 times across the model tuning process. In general, we want to parallelize the longest running part of the algorithm. I have no doubt of the utility of threading the computations <em>for a single model</em> but it can be counterproductive when model is called repeatedly. </p>
<p>The code and results can be found <a rel="nofollow" href="https://github.com/topepo/par-caret-bench" target="_blank">here</a> if you are interested in more details. </p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://appliedpredictivemodeling.com/blog/2018/1/17/parallel-processing"> Blog - Applied Predictive Modeling</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/dO_WkD9CyEA" height="1"></a></div>