<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/speed-up-your-code-parallel-processing-with-multidplyr/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html"> mattdancho.com - Articles</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>There’s nothing more frustrating than waiting for long-running <em>R</em> scripts to iteratively run. I’ve recently come across a new-ish package for parallel processing that plays nicely with the tidyverse: <code>multidplyr</code>. The package has saved me countless hours when applied to long-running, iterative scripts. In this post, I’ll discuss the workflow to parallelize your code, and I’ll go through a real world example of collecting stock prices where it improves speed by over 5X for a process that normally takes 2 minutes or so. Once you grasp the workflow, the parallelization can be applied to almost any iterative scripts regardless of application.</p>
<h1>Table of Contents</h1>
<ul>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#prerequisites">Prerequisites</a></li>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#why">Why Parallel Processing?</a></li>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#workflow">Workflow</a></li>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#example">Real World Example</a>
<ul>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#prep">Prep-Work</a></li>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#series">Processing in Series</a></li>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#parallel">Processing in Parallel</a>
<ul>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#get-cores">Step 0: Get Number of Cores (Optional)</a></li>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#add-groups">Step 1: Add Groups</a></li>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#create-clusters">Step 2: Create Clusters</a></li>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#partition">Step 3: Partition by Group</a></li>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#setup-clusters">Step 4: Setup Clusters</a></li>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#visualize-results">Step 5: Run Parallelized Code</a></li>
</ul>
</li>
</ul>
</li>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#conclusions">Conclusion</a></li>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#recap">Recap</a></li>
<li><a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#further-reading">Further Reading</a></li>
</ul>
<h1>Prerequisites <a></a></h1>
<p>The <code>multidplyr</code> package is not available on CRAN, but you can install it using <code>devtools</code>:</p>
<figure>
<pre><code data-lang="r"><span>install.packages</span><span>(</span><span>&quot;devtools&quot;</span><span>)</span><span>
</span><span>devtools</span><span>::</span><span>install_github</span><span>(</span><span>&quot;hadley/multidplyr&quot;</span><span>)</span></code></pre>
</figure>
<p>For those following along in <em>R</em>, you’ll need to load the following packages, which are available on CRAN:</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>multidplyr</span><span>)</span><span> </span><span># parallel processing
</span><span>library</span><span>(</span><span>rvest</span><span>)</span><span> </span><span># web scraping
</span><span>library</span><span>(</span><span>quantmod</span><span>)</span><span> </span><span># get stock prices; useful stock analysis functions
</span><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span> </span><span># ggplot2, purrr, dplyr, tidyr, readr, tibble
</span><span>library</span><span>(</span><span>stringr</span><span>)</span><span> </span><span># working with strings
</span><span>library</span><span>(</span><span>lubridate</span><span>)</span><span> </span><span>#</span><span> </span><span>working</span><span> </span><span>with</span><span> </span><span>dates</span><span> </span></code></pre>
</figure>
<p>If you don’t have these installed, run <code>install.packages(pkg_names)</code> with the package names as a character vector (<code>pkg_names &lt;- c(&quot;rvest&quot;, &quot;quantmod&quot;, ...)</code>).</p>
<p>I also recommend the open-source <a target="_blank" rel="nofollow" href="https://www.rstudio.com/">RStudio</a> IDE, which makes <em>R Programming</em> easy and efficient.</p>
<h1>Why Parallel Processing? <a></a></h1>
<p>Computer programming languages, including <em>R</em> and <em>python</em>, by default run scripts using only one processor (i.e. core). Under many circumstances this is fine since the computation speed is relatively fast. However, some scripts just take a long time to process, particularly during iterative programming (i.e. using loops to process a lot of data and/or very complex calculations).</p>
<p>Most modern PC’s have multiple cores that are underutilized. Parallel processing takes advantage of this by splitting the work across the multiple cores for maximum processor utilization. The result is a dramatic improvement in processing time.</p>
<p>While you may not realize it, most computations in <em>R</em> are loops that can be split using parallel processing. However, parallel processing takes more code and may not improve speeds, especially during fast computations because it takes time to transmit and recombine data. Therefore, parallel processing should only be used when speed is a significant issue.</p>
<p>When processing time is long, parallel processing could result in a significant improvement. Let’s check out how to parallelie your <em>R</em> code using the <code>multidplyr</code> package.</p>
<h1>Workflow <a></a></h1>
<p>The <code>multidplyr</code> workflow can be broken down into six basic steps shown in <a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#figure1"><em>Figure 1</em></a>. The six steps are implemented in <a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#parallel">Processing in Parallel</a>.</p>
<p><a></a></p>
<p><img alt="multidplyr worfkflow" src="https://i2.wp.com/www.mattdancho.com/assets/multidplyr.png?w=456" data-recalc-dims="1"></p>
<p>Figure 1: multidplyr Workflow</p>
<p>Essentially, you start with some data set that you need to do things to multiple times. Your situation generally falls into one of two types:</p>
<ol>
<li>It could be a really large data set that you want to split up into several small data sets and perform the same thing on each.</li>
<li>It could be one data set that you want to perform multiple things on (e.g. apply many models).</li>
</ol>
<p>The good news is both situations follow the same basic workflow. The toughest part is getting your data in the format needed to process using the workflow. Don’t worry, we’ll go through a real world example next so you can see how this is accomplished.</p>
<h1>Real World Example <a></a></h1>
<p>We’ll go through the <code>multidplyr</code> workflow using a real world example that I routinely use: collecting stock prices from the inter-web. Other uses include using modeling functions over grouped data sets, using many models on the same data set, and processing text (e.g. getting n-grams on large corpora). Basically anything with a loop.</p>
<h2>Prep-Work <a></a></h2>
<p>In preparation for collecting stock prices, we need two things:</p>
<ol>
<li>A list of stocks</li>
<li>A function to get stock prices from a stock symbol</li>
</ol>
<p>The code below comes from my <a target="_blank" rel="nofollow" href="http://www.mattdancho.com/investments/2016/10/23/SP500_Analysis.html">S&amp;P500 Stock Analysis Post</a> and it is also used in my more advanced <a target="_blank" rel="nofollow" href="http://www.mattdancho.com/investments/2016/11/30/Russell2000_Analysis.html">Russell 2000 Analysis Post</a>.</p>
<p>First, we use <code>rvest</code> to get the list of S&amp;P500 stocks, <code>sp500</code>.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>rvest</span><span>)</span><span>
</span><span># Web-scrape SP500 stock list from Wikipedia
</span><span>sp_500</span><span> </span><span>&lt;-</span><span> </span><span>read_html</span><span>(</span><span>&quot;https://en.wikipedia.org/wiki/List_of_S%26P_500_companies&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>html_node</span><span>(</span><span>&quot;table.wikitable&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>html_table</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>`Ticker symbol`</span><span>,</span><span> </span><span>Security</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>as_tibble</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>rename</span><span>(</span><span>symbol</span><span> </span><span>=</span><span> </span><span>`Ticker symbol`</span><span>,</span><span>
</span><span>company</span><span> </span><span>=</span><span> </span><span>Security</span><span>)</span><span>
</span><span># Show results
</span><span>sp_500</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 505 × 2
##    symbol             company
##     &lt;chr>               &lt;chr>
## 1     MMM          3M Company
## 2     ABT Abbott Laboratories
## 3    ABBV              AbbVie
## 4     ACN       Accenture plc
## 5    ATVI Activision Blizzard
## 6     AYI   Acuity Brands Inc
## 7    ADBE   Adobe Systems Inc
## 8     AAP  Advance Auto Parts
## 9     AES            AES Corp
## 10    AET           Aetna Inc
## # ... with 495 more rows</code></pre>
</figure>
<p>Second, we create a function that leverages the <code>quantmod::getSymbols</code> to return the historical stock prices in tidy format. This function will be mapped to all of the 500+ stocks next.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>quantmod</span><span>)</span><span>
</span><span># Function to Get Stock Prices in Tidy Form
</span><span>get_stock_prices</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>symbol</span><span>,</span><span> </span><span>return_format</span><span> </span><span>=</span><span> </span><span>&quot;tibble&quot;</span><span>,</span><span> </span><span>...</span><span>)</span><span> </span><span>{</span><span>
</span><span># Get stock prices; Handle errors
</span><span> </span><span>stock_prices</span><span> </span><span>&lt;-</span><span> </span><span>tryCatch</span><span>({</span><span>
</span><span>getSymbols</span><span>(</span><span>Symbols</span><span> </span><span>=</span><span> </span><span>symbol</span><span>,</span><span> </span><span>auto.assign</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>...</span><span>)</span><span>
</span><span>},</span><span> </span><span>error</span><span> </span><span>=</span><span> </span><span>function</span><span>(</span><span>e</span><span>)</span><span> </span><span>{</span><span>
</span><span>return</span><span>(</span><span>NA</span><span>)</span><span> </span><span># Return NA on error
</span><span> </span><span>})</span><span>
</span><span>if</span><span> </span><span>(</span><span>!</span><span>is.na</span><span>(</span><span>stock_prices</span><span>[[</span><span>1</span><span>]]))</span><span> </span><span>{</span><span>
</span><span># Rename
</span><span> </span><span>names</span><span>(</span><span>stock_prices</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;Open&quot;</span><span>,</span><span> </span><span>&quot;High&quot;</span><span>,</span><span> </span><span>&quot;Low&quot;</span><span>,</span><span> </span><span>&quot;Close&quot;</span><span>,</span><span> </span><span>&quot;Volume&quot;</span><span>,</span><span> </span><span>&quot;Adjusted&quot;</span><span>)</span><span>
</span><span># Return in xts format if tibble is not specified
</span><span> </span><span>if</span><span> </span><span>(</span><span>return_format</span><span> </span><span>==</span><span> </span><span>&quot;tibble&quot;</span><span>)</span><span> </span><span>{</span><span>
</span><span>stock_prices</span><span> </span><span>&lt;-</span><span> </span><span>stock_prices</span><span> </span><span>%>%</span><span>
</span><span>as_tibble</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>rownames_to_column</span><span>(</span><span>var</span><span> </span><span>=</span><span> </span><span>&quot;Date&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>Date</span><span> </span><span>=</span><span> </span><span>ymd</span><span>(</span><span>Date</span><span>))</span><span>
</span><span>}</span><span>
</span><span>return</span><span>(</span><span>stock_prices</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span></code></pre>
</figure>
<h2>Processing In Series <a></a></h2>
<p>The next computation is the routine that we wish to parallelize, but first we’ll time the script running on one processor, looping in series. We are collecting ten years of historical daily stock prices for each of the 500+ stocks. To do this, the script below uses the <code>purrr::map()</code> function to map <code>get_stock_prices()</code> to each stock <code>symbol</code> in <code>sp_500</code>. The loop in our case is the iterative application of a function to each stock. This operation will be split by group in the next section. The <code>proc.time()</code> function is used to time the routine running without parallel processing.</p>
<figure>
<pre><code data-lang="r"><span># Map get_stock_prices() to list of stock symbols
</span><span>from</span><span> </span><span>&lt;-</span><span> </span><span>&quot;2007-01-01&quot;</span><span>
</span><span>to</span><span> </span><span>&lt;-</span><span> </span><span>today</span><span>()</span><span>
</span><span>start</span><span> </span><span>&lt;-</span><span> </span><span>proc.time</span><span>()</span><span> </span><span># Start clock
</span><span>sp_500_processed_in_series</span><span> </span><span>&lt;-</span><span> </span><span>sp_500</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>
</span><span>stock.prices</span><span> </span><span>=</span><span> </span><span>map</span><span>(</span><span>.x</span><span> </span><span>=</span><span> </span><span>symbol</span><span>,</span><span> </span><span>~</span><span> </span><span>get_stock_prices</span><span>(</span><span>symbol</span><span> </span><span>=</span><span> </span><span>.x</span><span>,</span><span>
</span><span>return_format</span><span> </span><span>=</span><span> </span><span>&quot;tibble&quot;</span><span>,</span><span>
</span><span>from</span><span> </span><span>=</span><span> </span><span>from</span><span>,</span><span>
</span><span>to</span><span> </span><span>=</span><span> </span><span>to</span><span>)</span><span>
</span><span>)</span><span>
</span><span>)</span><span>
</span><span>time_elapsed_series</span><span> </span><span>&lt;-</span><span> </span><span>proc.time</span><span>()</span><span> </span><span>-</span><span> </span><span>start</span><span> </span><span>#</span><span> </span><span>End</span><span> </span><span>clock</span></code></pre>
</figure>
<p>The result, <code>sp_500_processed_in_series</code> is a <code>tibble</code> (tidy data frame) that is nested with two levels: the first has the stock <code>symbol</code>, <code>company</code>, and <code>stock.prices</code>. The variable, <code>stock.prices</code>, contains the historical stock prices for each stock.</p>
<figure>
<pre><code data-lang="r"><span>sp_500_processed_in_series</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 505 × 3
##    symbol             company         stock.prices
##     &lt;chr>               &lt;chr>               &lt;list>
## 1     MMM          3M Company &lt;tibble [2,506 × 7]>
## 2     ABT Abbott Laboratories &lt;tibble [2,506 × 7]>
## 3    ABBV              AbbVie   &lt;tibble [996 × 7]>
## 4     ACN       Accenture plc &lt;tibble [2,506 × 7]>
## 5    ATVI Activision Blizzard &lt;tibble [2,506 × 7]>
## 6     AYI   Acuity Brands Inc &lt;tibble [2,506 × 7]>
## 7    ADBE   Adobe Systems Inc &lt;tibble [2,506 × 7]>
## 8     AAP  Advance Auto Parts &lt;tibble [2,506 × 7]>
## 9     AES            AES Corp &lt;tibble [2,506 × 7]>
## 10    AET           Aetna Inc &lt;tibble [2,506 × 7]>
## # ... with 495 more rows</code></pre>
</figure>
<p>Let’s verify we got the daily stock prices for every stock. We’ll use the <code>tidyr::unnest()</code> function to expand the <code>stock.prices</code> for the list of stocks. At 1,203,551 rows, the full list has been obtained.</p>
<figure>
<pre><code data-lang="r"><span>sp_500_processed_in_series</span><span> </span><span>%>%</span><span> </span><span>unnest</span><span>()</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 1,203,551 × 9
##    symbol    company       Date  Open  High   Low Close  Volume
##     &lt;chr>      &lt;chr>     &lt;date> &lt;dbl> &lt;dbl> &lt;dbl> &lt;dbl>   &lt;dbl>
## 1     MMM 3M Company 2007-01-03 77.53 78.85 77.38 78.26 3781500
## 2     MMM 3M Company 2007-01-04 78.40 78.41 77.45 77.95 2968400
## 3     MMM 3M Company 2007-01-05 77.89 77.90 77.01 77.42 2765200
## 4     MMM 3M Company 2007-01-08 77.42 78.04 76.97 77.59 2434500
## 5     MMM 3M Company 2007-01-09 78.00 78.23 77.44 77.68 1896800
## 6     MMM 3M Company 2007-01-10 77.31 77.96 77.04 77.85 1787500
## 7     MMM 3M Company 2007-01-11 78.05 79.03 77.88 78.65 2372500
## 8     MMM 3M Company 2007-01-12 78.41 79.50 78.22 79.36 2582200
## 9     MMM 3M Company 2007-01-16 79.48 79.62 78.92 79.56 2526600
## 10    MMM 3M Company 2007-01-17 79.33 79.51 78.75 78.91 2711300
## # ... with 1,203,541 more rows, and 1 more variables: Adjusted &lt;dbl></code></pre>
</figure>
<p>And, let’s see how long it took when processing in series. The processing time is the time elapsed in seconds. Converted to minutes this is approximately 1.68 minutes.</p>
<figure>
<pre><code data-lang="r"><span>time_elapsed_series</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##    user  system elapsed 
##   43.12    1.08  100.58</code></pre>
</figure>
<h2>Processing in Parallel <a></a></h2>
<p>We just collected ten years of daily stock prices for over 500 stocks in about 1.68 minutes. Let’s parallelize the computation to get an improvement. We will follow the six steps shown in <a target="_blank" rel="nofollow" href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html#figure1"><em>Figure 1</em></a>.</p>
<h3>Step 0: Get Number of Cores (Optional) <a></a></h3>
<p>Prior to starting, you may want to determine how many cores your machine has. An easy way to do this is using <code>parallel::detectCores()</code>. This will be used to determine the number of groups to split the data into in the next set.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>parallel</span><span>)</span><span>
</span><span>cl</span><span> </span><span>&lt;-</span><span> </span><span>detectCores</span><span>()</span><span>
</span><span>cl</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## [1] 8</code></pre>
</figure>
<h3>Step 1: Add Groups <a></a></h3>
<p>Let’s add groups to <code>sp_500</code>. The groups are needed to divide the data across your <code>cl</code> number cores. For me, this is 8 cores. We create a <code>group</code> vector, which is a sequential vector of <code>1:cl</code> (1 to 8) repeated the length of the number of rows in <code>sp_500</code>. We then add the <code>group</code> vector to the <code>sp_500</code> tibble using the <code>dplyr::bind_cols()</code> function.</p>
<figure>
<pre><code data-lang="r"><span>group</span><span> </span><span>&lt;-</span><span> </span><span>rep</span><span>(</span><span>1</span><span>:</span><span>cl</span><span>,</span><span> </span><span>length.out</span><span> </span><span>=</span><span> </span><span>nrow</span><span>(</span><span>sp_500</span><span>))</span><span>
</span><span>sp_500</span><span> </span><span>&lt;-</span><span> </span><span>bind_cols</span><span>(</span><span>tibble</span><span>(</span><span>group</span><span>),</span><span> </span><span>sp_500</span><span>)</span><span>
</span><span>sp_500</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 505 × 3
##    group symbol             company
##    &lt;int>  &lt;chr>               &lt;chr>
## 1      1    MMM          3M Company
## 2      2    ABT Abbott Laboratories
## 3      3   ABBV              AbbVie
## 4      4    ACN       Accenture plc
## 5      5   ATVI Activision Blizzard
## 6      6    AYI   Acuity Brands Inc
## 7      7   ADBE   Adobe Systems Inc
## 8      8    AAP  Advance Auto Parts
## 9      1    AES            AES Corp
## 10     2    AET           Aetna Inc
## # ... with 495 more rows</code></pre>
</figure>
<h3>Step 2: Create Clusters <a></a></h3>
<p>Use the <code>create_cluster()</code> function from the <code>multidplyr</code> package. Think of a cluster as a work environment on a core. Therefore, the code below establishes a work environment on each of the 8 cores.</p>
<figure>
<pre><code data-lang="r"><span>cluster</span><span> </span><span>&lt;-</span><span> </span><span>create_cluster</span><span>(</span><span>cores</span><span> </span><span>=</span><span> </span><span>cl</span><span>)</span><span>
</span><span>cluster</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## socket cluster with 8 nodes on host 'localhost'</code></pre>
</figure>
<h3>Step 3: Partition by Group <a></a></h3>
<p>Next is partitioning. Think of partitioning as sending a subset of the initial <code>tibble</code> to each of the clusters. The result is a partitioned data frame (<code>party_df</code>), which we explore next. Use the <code>partition()</code> function from the <code>multidplyr</code> package to split the <code>sp_500</code> list by group and send each group to a different cluster.</p>
<figure>
<pre><code data-lang="r"><span>by_group</span><span> </span><span>&lt;-</span><span> </span><span>sp_500</span><span> </span><span>%>%</span><span>
</span><span>partition</span><span>(</span><span>group</span><span>,</span><span> </span><span>cluster</span><span> </span><span>=</span><span> </span><span>cluster</span><span>)</span><span>
</span><span>by_group</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## Source: party_df [505 x 3]
## Groups: group
## Shards: 8 [63--64 rows]
## 
## # S3: party_df
##    group symbol                           company
##    &lt;int>  &lt;chr>                             &lt;chr>
## 1      6    AYI                 Acuity Brands Inc
## 2      6    APD      Air Products &amp; Chemicals Inc
## 3      6    ADS             Alliance Data Systems
## 4      6    AEP           American Electric Power
## 5      6   AMGN                         Amgen Inc
## 6      6   AAPL                        Apple Inc.
## 7      6    ADP         Automatic Data Processing
## 8      6     BK The Bank of New York Mellon Corp.
## 9      6   BIIB                  BIOGEN IDEC Inc.
## 10     6   AVGO                          Broadcom
## # ... with 495 more rows</code></pre>
</figure>
<p>The result, <code>by_group</code>, looks similar to our original <code>tibble</code>, but it is a <code>party_df</code>, which is very different. The key is to notice that the there are 8 <code>Shards</code>. Each <code>Shard</code> has between 63 and 64 rows, which evenly splits our data among each shard. Now that our <code>tibble</code> has been partitioned into a <code>party_df</code>, we are ready to move onto setting up the clusters.</p>
<h3>Step 4: Setup Clusters <a></a></h3>
<p>The clusters have a local, bare-bones <em>R work environment</em>, which doesn’t work for the vast majority of cases. Code typically depends on libraries, functions, expressions, variables, and/or data that are not available in base <em>R</em>. Fortunately, there is a way to add these items to the clusters. Let’s see how.</p>
<p>For our computation, we are going to need to add several libraries along with the <code>get_stock_prices()</code> function to the clusters. We do this by using the <code>cluster_library()</code> and <code>cluster_assign_value()</code> functions, respectively.</p>
<figure>
<pre><code data-lang="r"><span>from</span><span> </span><span>&lt;-</span><span> </span><span>&quot;2007-01-01&quot;</span><span>
</span><span>to</span><span> </span><span>&lt;-</span><span> </span><span>today</span><span>()</span><span>
</span><span># Utilize pipe (%>%) to assign libraries, functions, and values to clusters
</span><span>by_group</span><span> </span><span>%>%</span><span>
</span><span># Assign libraries
</span><span> </span><span>cluster_library</span><span>(</span><span>&quot;tidyverse&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>cluster_library</span><span>(</span><span>&quot;stringr&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>cluster_library</span><span>(</span><span>&quot;lubridate&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>cluster_library</span><span>(</span><span>&quot;quantmod&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span># Assign values (use this to load functions or data to each core)
</span><span> </span><span>cluster_assign_value</span><span>(</span><span>&quot;get_stock_prices&quot;</span><span>,</span><span> </span><span>get_stock_prices</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>cluster_assign_value</span><span>(</span><span>&quot;from&quot;</span><span>,</span><span> </span><span>from</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>cluster_assign_value</span><span>(</span><span>&quot;to&quot;</span><span>,</span><span> </span><span>to</span><span>)</span></code></pre>
</figure>
<p>We can verify that the libraries are loaded using the <code>cluster_eval()</code> function.</p>
<figure>
<pre><code data-lang="r"><span>cluster_eval</span><span>(</span><span>by_group</span><span>,</span><span> </span><span>search</span><span>())[[</span><span>1</span><span>]]</span><span> </span><span>#</span><span> </span><span>results</span><span> </span><span>for</span><span> </span><span>first</span><span> </span><span>cluster</span><span> </span><span>shown</span><span> </span><span>only</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##  [1] &quot;.GlobalEnv&quot;        &quot;package:quantmod&quot;  &quot;package:TTR&quot;      
##  [4] &quot;package:xts&quot;       &quot;package:zoo&quot;       &quot;package:lubridate&quot;
##  [7] &quot;package:stringr&quot;   &quot;package:dplyr&quot;     &quot;package:purrr&quot;    
## [10] &quot;package:readr&quot;     &quot;package:tidyr&quot;     &quot;package:tibble&quot;   
## [13] &quot;package:ggplot2&quot;   &quot;package:tidyverse&quot; &quot;package:stats&quot;    
## [16] &quot;package:graphics&quot;  &quot;package:grDevices&quot; &quot;package:utils&quot;    
## [19] &quot;package:datasets&quot;  &quot;package:methods&quot;   &quot;Autoloads&quot;        
## [22] &quot;package:base&quot;</code></pre>
</figure>
<p>We can also verify that the functions are loaded using the <code>cluster_get()</code> function.</p>
<figure>
<pre><code data-lang="r"><span>cluster_get</span><span>(</span><span>by_group</span><span>,</span><span> </span><span>&quot;get_stock_prices&quot;</span><span>)[[</span><span>1</span><span>]]</span><span> </span><span>#</span><span> </span><span>results</span><span> </span><span>for</span><span> </span><span>first</span><span> </span><span>cluster</span><span> </span><span>shown</span><span> </span><span>only</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## function(symbol, return_format = &quot;tibble&quot;, ...) {
##     # Get stock prices; Handle errors
##     stock_prices &lt;- tryCatch({
##         getSymbols(Symbols = symbol, auto.assign = FALSE, ...)
##     }, error = function(e) {
##         return(NA) # Return NA on error
##     })
##     if (!is.na(stock_prices[[1]])) {
##         # Rename
##         names(stock_prices) &lt;- c(&quot;Open&quot;, &quot;High&quot;, &quot;Low&quot;, &quot;Close&quot;, &quot;Volume&quot;, &quot;Adjusted&quot;)
##         # Return in xts format if tibble is not specified
##         if (return_format == &quot;tibble&quot;) {
##             stock_prices &lt;- stock_prices %>%
##                 as_tibble() %>%
##                 rownames_to_column(var = &quot;Date&quot;) %>%
##                 mutate(Date = ymd(Date))
##         }
##         return(stock_prices)
##     }
## }</code></pre>
</figure>
<h3>Step 5: Run Parallelized Code <a></a></h3>
<p>Now that we have our clusters and partitions set up and everything looks good, we can run the parallelized code. The code chunk is the same as the series code chunk with two exceptions:</p>
<ol>
<li>Instead of starting with the <code>sp_500 tibble</code>, we start with the <code>by_group party_df</code></li>
<li>We combine the results at the end using the <code>collect()</code> function</li>
</ol>
<figure>
<pre><code data-lang="r"><span>start</span><span> </span><span>&lt;-</span><span> </span><span>proc.time</span><span>()</span><span> </span><span># Start clock
</span><span>sp_500_processed_in_parallel</span><span> </span><span>&lt;-</span><span> </span><span>by_group</span><span> </span><span>%>%</span><span> </span><span># Use by_group party_df
</span><span> </span><span>mutate</span><span>(</span><span>
</span><span>stock.prices</span><span> </span><span>=</span><span> </span><span>map</span><span>(</span><span>.x</span><span> </span><span>=</span><span> </span><span>symbol</span><span>,</span><span> </span><span>~</span><span> </span><span>get_stock_prices</span><span>(</span><span>symbol</span><span> </span><span>=</span><span> </span><span>.x</span><span>,</span><span>
</span><span>return_format</span><span> </span><span>=</span><span> </span><span>&quot;tibble&quot;</span><span>,</span><span>
</span><span>from</span><span> </span><span>=</span><span> </span><span>from</span><span>,</span><span>
</span><span>to</span><span> </span><span>=</span><span> </span><span>to</span><span>)</span><span>
</span><span>)</span><span>
</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>collect</span><span>()</span><span> </span><span>%>%</span><span> </span><span># Special collect() function to recombine partitions
</span><span> </span><span>as_tibble</span><span>()</span><span> </span><span># Convert to tibble
</span><span>time_elapsed_parallel</span><span> </span><span>&lt;-</span><span> </span><span>proc.time</span><span>()</span><span> </span><span>-</span><span> </span><span>start</span><span> </span><span>#</span><span> </span><span>End</span><span> </span><span>clock</span></code></pre>
</figure>
<p>Let’s verify we got the list of stock prices. We’ll use the <code>tidyr::unnest()</code> function to expand the <code>stock.prices</code> for the list of stocks. This is the same list as <code>sp_500_processed_in_series</code>, but it’s sorted in order by which groups finished first. If we want to return the <code>tibble</code> in the same order of <code>sp_500</code>, we can easily pipe (<code>%>%</code>) <code>arrange(company)</code> after <code>as_tibble()</code> in the code chunk above.</p>
<figure>
<pre><code data-lang="r"><span>sp_500_processed_in_parallel</span><span> </span><span>%>%</span><span> </span><span>unnest</span><span>()</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## Source: local data frame [1,204,056 x 10]
## Groups: group [8]
## 
##    group symbol           company       Date  Open  High   Low Close
##    &lt;int>  &lt;chr>             &lt;chr>     &lt;date> &lt;dbl> &lt;dbl> &lt;dbl> &lt;dbl>
## 1      6    AYI Acuity Brands Inc 2007-01-03 52.20 53.27 51.52 51.92
## 2      6    AYI Acuity Brands Inc 2007-01-04 53.00 53.39 49.69 52.32
## 3      6    AYI Acuity Brands Inc 2007-01-05 51.74 51.85 50.95 51.09
## 4      6    AYI Acuity Brands Inc 2007-01-08 51.15 52.64 50.70 52.01
## 5      6    AYI Acuity Brands Inc 2007-01-09 51.94 52.59 51.66 52.52
## 6      6    AYI Acuity Brands Inc 2007-01-10 52.13 53.61 52.01 53.48
## 7      6    AYI Acuity Brands Inc 2007-01-11 53.60 55.47 53.52 55.08
## 8      6    AYI Acuity Brands Inc 2007-01-12 54.84 55.68 54.61 55.65
## 9      6    AYI Acuity Brands Inc 2007-01-16 55.65 56.56 55.40 55.61
## 10     6    AYI Acuity Brands Inc 2007-01-17 55.56 56.32 55.20 55.90
## # ... with 1,204,046 more rows, and 2 more variables: Volume &lt;dbl>,
## #   Adjusted &lt;dbl></code></pre>
</figure>
<p>And, let’s see how long it took when processing in parallel.</p>
<figure>
<pre><code data-lang="r"><span>time_elapsed_parallel</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">##    user  system elapsed 
##    0.17    0.28   18.66</code></pre>
</figure>
<p><strong>The processing time is approximately 0.31 minutes, which is 5.4X faster!</strong> Note that it’s not a full 8X faster because of transmission time as data is sent to and from the nodes. With that said, the speed will approach 8X improvement as calculations become longer since the transmission time is fixed whereas the computation time is variable.</p>
<h1>Conclusion <a></a></h1>
<p>Parallelizing code can drastically improve speed on multi-core machines. It makes the most sense in situations involving many iterative computations. On an 8 core machine, processing time significantly improves. It will not be quite 8X faster, but the longer the computation the closer the speed gets to the full 8X improvement. <strong>For a computation that takes two minutes under normal conditions, we improved the processing speed by over 5X through parallel processing!</strong></p>
<h1>Recap <a></a></h1>
<p>The focus of this post was on the <code>multidplyr</code> package, a package designed to do parallel processing within the <code>tidyverse</code>. We worked through the five main steps in the <code>multidplyr</code> workflow on a real world example of collecting historical stock prices for a large set of stocks. The beauty is that the package and workflow can be applied to anything from collecting stocks to applying many models to computing n-grams on textual data and more!</p>
<h1>Further Reading <a></a></h1>
<ol>
<li><strong><a target="_blank" rel="nofollow" href="https://github.com/hadley/multidplyr"><code>multidplyr</code> on GitHub</a></strong>: The vignette explains the <code>multidplyr</code> workflow using the <code>flights</code> data set from the <code>nycflights13</code> package.</li>
</ol>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.mattdancho.com/code-tools/2016/12/18/multidplyr.html"> mattdancho.com - Articles</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/rSMTzFoxlfU"></a></div>