<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/can-you-nest-parallel-operations-in-r/"></div></div><div><a data-url="https://www.r-bloggers.com/can-you-nest-parallel-operations-in-r/" data-counturl="https://www.r-bloggers.com/can-you-nest-parallel-operations-in-r/" data-count="horizontal" data-text="Can you nest parallel operations in R?" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2016/08/can-you-nest-parallel-operations-in-r/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>When we teach <a target="_blank" href="https://cran.r-project.org/web/views/HighPerformanceComputing.html">parallel programming</a> in <a target="_blank" href="https://cran.r-project.org/">R</a> we start with the basic use of <code>parallel</code> (please see <a target="_blank" href="http://www.win-vector.com/blog/2016/01/parallel-computing-in-r/">here for example</a>). This is, in our opinion, a necessary step <em>before</em> getting into clever notation and wrapping such as <a target="_blank" href="https://cran.r-project.org/package=doParallel">doParallel</a> and <a target="_blank" href="https://cran.r-project.org/package=foreach">foreach</a>. Only then do the students have a sufficiently explicit interface to frame important questions about the semantics of parallel computing. Beginners really need a solid mental model of what services are really being provided by their tools and to test edge cases early.</p>
<p>One question that comes up over and over again is “can you nest <code>parLapply</code>?”</p>
<p>The answer is “no.” This is in fact an advanced topic, but it is one of the things that pops up when you start worrying about parallel programming. Please read on for what that is the right answer and how to work around that (simulate a “yes”).</p>
<p>I don’t think the above question is usually given sufficient consideration (nesting parallel operations can in fact make a lot of sense). You can’t directly nest <code>parLapply</code>, but that is a different issue than can one invent a work-around. For example: a “yes” answer (really meaning there are work-arounds) can be found <a target="_blank" href="http://stackoverflow.com/questions/8413188/can-i-nest-parallelparlapply">here</a>. Again this is a different question than “is there a way to nest foreach loops” (which is possible through <a target="_blank" href="https://cran.r-project.org/web/packages/foreach/vignettes/nested.pdf">the nesting operator <code>%.%</code></a> which presumably handles working around nesting issues in <code>parLapply</code>).</p>
<p><span></span></p>
<p>Let’s set up a concrete example, so we can ask and answer a precise question. Suppose we have a list of jobs (coming from an external source) that we will simulate with the code fragment below.</p>
<pre><code>jobs &lt;- list(1:3,5:10,100:200)</code></pre>
<p>Notice the jobs have wildly diverging sizes, this is an important consideration.</p>
<p>Suppose the task we want to perform is some the square roots of the entries. The standard (non-parallel) calculation would look like the following.</p>
<pre><code>worker1 &lt;- function(x) {
  sum(sqrt(x))
}

lapply(jobs,worker1)</code></pre>
<pre><code>## [[1]]
## [1] 4.146264
## 
## [[2]]
## [1] 16.32201
## 
## [[3]]
## [1] 1231.021</code></pre>
<p>For didactic purposes please pretend that the <code>sum</code> function is very expensive and the <code>sqrt</code> function is somewhat expensive.</p>
<p>If it was obvious we always had a great number of small sub-lists we would want to use parallelization to make sure we are performing many <code>sum</code>s at the same time. We would then parallelize over the first level as below.</p>
<pre><code>clus &lt;- parallel::makeCluster(4)</code></pre>
<pre><code>parallel::parLapplyLB(clus,jobs,worker1)</code></pre>
<pre><code>## [[1]]
## [1] 4.146264
## 
## [[2]]
## [1] 16.32201
## 
## [[3]]
## [1] 1231.021</code></pre>
<p>Notice that <code>parallel::parLapplyLB</code> uses almost the same calling convention as <code>lapply</code> and returns the exact same answer.</p>
<p>If it was obvious we had a single large sub-list we would want to make sure we were always parallelizing the <code>sqrt</code> operations so we would prefer to parallelize as follows:</p>
<pre><code>mkWorker2 &lt;- function(clus) {
  force(clus)
  function(x) {
    xs &lt;- parallel::parLapplyLB(clus,x,sqrt)
    sum(as.numeric(xs))
  }
}

worker2 &lt;- mkWorker2(clus)

lapply(jobs,worker2)</code></pre>
<pre><code>## [[1]]
## [1] 4.146264
## 
## [[2]]
## [1] 16.32201
## 
## [[3]]
## [1] 1231.021</code></pre>
<p>(For the details of building functions and passing values to remote workers please see <a target="_blank" href="http://www.win-vector.com/blog/2016/01/parallel-computing-in-r/">here</a>.)</p>
<p>If we were not sure if in the future what structure we would encounter we would prefer to schedule all operations for possible parallel execution. This would minimize the number of idle resources and minimize the time to finish the jobs. Ideally that would look like the following (a nested use of parallel):</p>
<pre><code>parallel::parLapplyLB(clus,jobs,worker2)</code></pre>
<pre><code>## Error in checkForRemoteErrors(val): 3 nodes produced errors; first error: invalid connection</code></pre>
<p>Notice the above fails with an error. Wishing for flexible code is what beginners intuitively mean when they as if you can nest parallel calls. They may not be able to explain it, but they are worried they don’t have a good characterization of the work they are trying to parallelize over. They are not asking if things get magically faster by “parallelizing parallel.”</p>
<p>It isn’t too hard to find out what the nature of the error is: the communication connection socket file descriptors (<code>con</code>) are passed as integers to each machine, but they are not valid descriptors where they arrive (they are just integers). We can see this by looking at the structure of the cluster:</p>
<pre><code>str(clus)</code></pre>
<pre><code>## List of 4
##  $ :List of 3
##   ..$ con :Classes 'sockconn', 'connection'  atomic [1:1] 5
##   .. .. ..- attr(*, &quot;conn_id&quot;)=&lt;externalptr> 
##   ..$ host: chr &quot;localhost&quot;
##   ..$ rank: int 1
##   ..- attr(*, &quot;class&quot;)= chr &quot;SOCKnode&quot;
##  $ :List of 3
##   ..$ con :Classes 'sockconn', 'connection'  atomic [1:1] 6
##   .. .. ..- attr(*, &quot;conn_id&quot;)=&lt;externalptr> 
##   ..$ host: chr &quot;localhost&quot;
##   ..$ rank: int 2
##   ..- attr(*, &quot;class&quot;)= chr &quot;SOCKnode&quot;
##  $ :List of 3
##   ..$ con :Classes 'sockconn', 'connection'  atomic [1:1] 7
##   .. .. ..- attr(*, &quot;conn_id&quot;)=&lt;externalptr> 
##   ..$ host: chr &quot;localhost&quot;
##   ..$ rank: int 3
##   ..- attr(*, &quot;class&quot;)= chr &quot;SOCKnode&quot;
##  $ :List of 3
##   ..$ con :Classes 'sockconn', 'connection'  atomic [1:1] 8
##   .. .. ..- attr(*, &quot;conn_id&quot;)=&lt;externalptr> 
##   ..$ host: chr &quot;localhost&quot;
##   ..$ rank: int 4
##   ..- attr(*, &quot;class&quot;)= chr &quot;SOCKnode&quot;
##  - attr(*, &quot;class&quot;)= chr [1:2] &quot;SOCKcluster&quot; &quot;cluster&quot;</code></pre>
<pre><code>mkWorker3 &lt;- function(clus) {
  force(clus)
  function(x) {
    as.character(clus)
  }
}

worker3 &lt;- mkWorker3(clus)
parallel::parLapplyLB(clus,jobs,worker3)</code></pre>
<pre><code>## [[1]]
## [1] &quot;list(con = 5, host = \&quot;localhost\&quot;, rank = 1)&quot;
## [2] &quot;list(con = 6, host = \&quot;localhost\&quot;, rank = 2)&quot;
## [3] &quot;list(con = 7, host = \&quot;localhost\&quot;, rank = 3)&quot;
## [4] &quot;list(con = 8, host = \&quot;localhost\&quot;, rank = 4)&quot;
## 
## [[2]]
## [1] &quot;list(con = 5, host = \&quot;localhost\&quot;, rank = 1)&quot;
## [2] &quot;list(con = 6, host = \&quot;localhost\&quot;, rank = 2)&quot;
## [3] &quot;list(con = 7, host = \&quot;localhost\&quot;, rank = 3)&quot;
## [4] &quot;list(con = 8, host = \&quot;localhost\&quot;, rank = 4)&quot;
## 
## [[3]]
## [1] &quot;list(con = 5, host = \&quot;localhost\&quot;, rank = 1)&quot;
## [2] &quot;list(con = 6, host = \&quot;localhost\&quot;, rank = 2)&quot;
## [3] &quot;list(con = 7, host = \&quot;localhost\&quot;, rank = 3)&quot;
## [4] &quot;list(con = 8, host = \&quot;localhost\&quot;, rank = 4)&quot;</code></pre>
<p>What we are getting wrong is: we can’t share control of the cluster to each worker just by passing the cluster object around. This would require some central registry and call-back scheme (which is one of the things packages like <code>foreach</code> and <code>doParallel</code> accomplish when they “register a parallel back-end to use”). Base <code>parallel</code> depends more on explicit reference to the cluster data structure, so it isn’t “idiomatic parLapply” to assume we can find “the parallel cluster” (there could in fact be more than one at the same time).</p>
<p>So what is the work around?</p>
<p>One work around is to move to sophisticated wrappers (like <code>doParallel</code> or even <a target="_blank" href="https://cran.r-project.org/package=future"><code>future</code></a>, also see <a target="_blank" href="http://www.jottr.org/2016/07/a-future-for-r-slides-from-user-2016.html">here</a>).</p>
<p>Another fix is to (at the cost of time effort and space) to re-organize the calculation into two sequenced phases, each of which is parallel- but not nested. It is a bit involved, but we show how to do that below (using <code>R</code>’s <code>Reduce</code> and <code>split</code> functions to reorganize the data, though one could also use so-called “tidyverse” methods).</p>
<pre><code># Preparation 1: collect all items into one flat list
sqrtjobs &lt;- as.list(Reduce(c,jobs))
# Phase 1: sqrt every item in parallel
sqrts &lt;- parallel::parLapplyLB(clus,sqrtjobs,sqrt)
# Preparation 2: re-assemble new job list that needs only sums
lengths &lt;- vapply(jobs,length,numeric(1))
pattern &lt;- lapply(seq_len(length(lengths)),
                  function(i) {rep(i,lengths[[i]])})
pattern &lt;- Reduce(c,pattern)
sumjobs &lt;- split(sqrts,Reduce(c,pattern))
sumjobs &lt;- lapply(sumjobs,as.numeric)
names(sumjobs) &lt;- names(jobs)
# Phase 2: sum all items in parallel
parallel::parLapplyLB(clus,sumjobs,sum)</code></pre>
<pre><code>## [[1]]
## [1] 4.146264
## 
## [[2]]
## [1] 16.32201
## 
## [[3]]
## [1] 1231.021</code></pre>
<p>In conclusion: you can’t <em>directly</em> nest <code>parLapply</code>, but you can usefully sequence through it.</p>
<pre><code>parallel::stopCluster(clus)</code></pre>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/can-you-nest-parallel-operations-in-r/"></div></div><div><a data-url="https://www.r-bloggers.com/can-you-nest-parallel-operations-in-r/" data-counturl="https://www.r-bloggers.com/can-you-nest-parallel-operations-in-r/" data-count="vertical" data-text="Can you nest parallel operations in R?" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2016/08/can-you-nest-parallel-operations-in-r/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/Rt6K-Ph1aHs"></a></div>