<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/reading-list-faster-with-parallel-doparallel-and-pbapply/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://danielmarcelino.github.io/2018/reading-list-faster.html"> Daniel Marcelino's Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img alt="center" title="Parallel isn't always fast" src="https://i0.wp.com/danielmarcelino.github.io/img/2018-12-12-reading-list-faster/Multithreading.png?w=456" data-recalc-dims="1"></p>
<p>I have several tables that I would like to load as a sole data frame. Derived functions from read. table () have a lot of convenient features, but it seems like there is a lot of steps in the implementation that would slow things down. The gain in performance of reading 29 CSV files (about 2.2 GB) shows quite different picture. While the parallelization process does bring some improvement considering the ‘user time’, i.e. the CPU time charged for the process execution at the machine level, the ‘elapsed time’, i.e. the ‘real’ elapsed time since the process was started doesn’t show much difference. Let’s go through it.</p>
<h2>Data</h2>
<figure>
<pre><code data-lang="r"><span>list_of_datasets</span><span> </span><span>&lt;-</span><span> </span><span>list.files</span><span>(</span><span>pattern</span><span> </span><span>=</span><span> </span><span>&quot;*.csv&quot;</span><span>)</span><span>
</span><span>list_of_datasets</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text"> [1] &quot;despesas_contratadas_candidatos_2018_AC.csv&quot;    
 [2] &quot;despesas_contratadas_candidatos_2018_AL.csv&quot;    
 [3] &quot;despesas_contratadas_candidatos_2018_AM.csv&quot;    
 [4] &quot;despesas_contratadas_candidatos_2018_AP.csv&quot;    
 [5] &quot;despesas_contratadas_candidatos_2018_BA.csv&quot;    
 [6] &quot;despesas_contratadas_candidatos_2018_BR.csv&quot;    
 [7] &quot;despesas_contratadas_candidatos_2018_BRASIL.csv&quot;
 [8] &quot;despesas_contratadas_candidatos_2018_CE.csv&quot;    
 [9] &quot;despesas_contratadas_candidatos_2018_DF.csv&quot;    
[10] &quot;despesas_contratadas_candidatos_2018_ES.csv&quot;    
[11] &quot;despesas_contratadas_candidatos_2018_GO.csv&quot;    
[12] &quot;despesas_contratadas_candidatos_2018_MA.csv&quot;    
[13] &quot;despesas_contratadas_candidatos_2018_MG.csv&quot;    
[14] &quot;despesas_contratadas_candidatos_2018_MS.csv&quot;    
[15] &quot;despesas_contratadas_candidatos_2018_MT.csv&quot;    
[16] &quot;despesas_contratadas_candidatos_2018_PA.csv&quot;    
[17] &quot;despesas_contratadas_candidatos_2018_PB.csv&quot;    
[18] &quot;despesas_contratadas_candidatos_2018_PE.csv&quot;    
[19] &quot;despesas_contratadas_candidatos_2018_PI.csv&quot;    
[20] &quot;despesas_contratadas_candidatos_2018_PR.csv&quot;    
[21] &quot;despesas_contratadas_candidatos_2018_RJ.csv&quot;    
[22] &quot;despesas_contratadas_candidatos_2018_RN.csv&quot;    
[23] &quot;despesas_contratadas_candidatos_2018_RO.csv&quot;    
[24] &quot;despesas_contratadas_candidatos_2018_RR.csv&quot;    
[25] &quot;despesas_contratadas_candidatos_2018_RS.csv&quot;    
[26] &quot;despesas_contratadas_candidatos_2018_SC.csv&quot;    
[27] &quot;despesas_contratadas_candidatos_2018_SE.csv&quot;    
[28] &quot;despesas_contratadas_candidatos_2018_SP.csv&quot;    
[29] &quot;despesas_contratadas_candidatos_2018_TO.csv&quot;    </code></pre>
</figure>
<h2>Function</h2>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>pbapply</span><span>)</span><span>
</span><span>library</span><span>(</span><span>parallel</span><span>)</span><span>
</span><span>library</span><span>(</span><span>doParallel</span><span>)</span><span>
</span><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>#' Reads a list of datasets</span><span>
</span><span>#' @param x A list of datasets (names of datasets are strings)</span><span>
</span><span>#' @param func A function, the read function to use to read the data</span><span>
</span><span>#' @param parallelize Parallelize the code</span><span>
</span><span>#' @param ... Further arguments passed to func</span><span>
</span><span>readListFaster</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>x</span><span>,</span><span> </span><span>func</span><span>,</span><span> </span><span>...</span><span>,</span><span> </span><span>parallelize</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>rbind</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>){</span><span>
</span><span>stopifnot</span><span>(</span><span>length</span><span>(</span><span>x</span><span>)</span><span> </span><span>></span><span> </span><span>0</span><span>)</span><span>
</span><span>read_and_assign</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>dataset</span><span>,</span><span> </span><span>func</span><span>){</span><span>
</span><span>dataset_name</span><span> </span><span>&lt;-</span><span> </span><span>as.name</span><span>(</span><span>dataset</span><span>)</span><span>
</span><span>dataset_name</span><span> </span><span>&lt;-</span><span> </span><span>func</span><span>(</span><span>dataset</span><span>,</span><span> </span><span>...</span><span>)</span><span>
</span><span>}</span><span>
</span><span>if</span><span> </span><span>(</span><span>parallelize</span><span>)</span><span> </span><span>{</span><span>
</span><span>message</span><span>(</span><span>&quot;Reading in data in parallel&quot;</span><span>)</span><span>
</span><span>clusters</span><span> </span><span>&lt;-</span><span> </span><span>parallel</span><span>::</span><span>detectCores</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>parallel</span><span>::</span><span>makeCluster</span><span>()</span><span>
</span><span>doParallel</span><span>::</span><span>registerDoParallel</span><span>(</span><span>clusters</span><span>)</span><span>
</span><span>output</span><span> </span><span>&lt;-</span><span> </span><span>invisible</span><span>(</span><span> </span><span># invisible is used to suppress the unneeded output</span><span>
</span><span>pbapply</span><span>::</span><span>pblapply</span><span>(</span><span>x</span><span>,</span><span>
</span><span>read_and_assign</span><span>,</span><span>
</span><span>func</span><span> </span><span>=</span><span> </span><span>func</span><span>,</span><span>
</span><span>...</span><span>,</span><span>
</span><span>cl</span><span> </span><span>=</span><span> </span><span>clusters</span><span>)</span><span>
</span><span>)</span><span>
</span><span>parallel</span><span>::</span><span>stopCluster</span><span>(</span><span>clusters</span><span>)</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>if</span><span> </span><span>(</span><span>!</span><span>parallelize</span><span>)</span><span> </span><span>{</span><span>
</span><span>output</span><span> </span><span>&lt;-</span><span> </span><span>invisible</span><span>(</span><span>
</span><span>pbapply</span><span>::</span><span>pblapply</span><span>(</span><span>x</span><span>,</span><span>
</span><span>read_and_assign</span><span>,</span><span>
</span><span>func</span><span> </span><span>=</span><span> </span><span>func</span><span>)</span><span>
</span><span>)</span><span>
</span><span>}</span><span>
</span><span># Remove what's after the &quot;.&quot; at the end of the data set names and what's before any / for url files.</span><span>
</span><span>x</span><span> </span><span>&lt;-</span><span> </span><span>stringr</span><span>::</span><span>str_replace_all</span><span>(</span><span>x</span><span>,</span><span>&quot;.*/|\\..*&quot;</span><span>,</span><span> </span><span>&quot;&quot;</span><span>)</span><span>
</span><span>names</span><span>(</span><span>output</span><span>)</span><span> </span><span>&lt;-</span><span> </span><span>x</span><span>
</span><span>if</span><span> </span><span>(</span><span>rbind</span><span>)</span><span> </span><span>{</span><span>
</span><span>dplyr</span><span>::</span><span>bind_rows</span><span>(</span><span>output</span><span>)</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>output</span><span>
</span><span>}</span><span>
</span><span>}</span></code></pre>
</figure>
<h2>Results</h2>
<h3>Without Paralelize</h3>
<figure>
<pre><code data-lang="r"><span>system.time</span><span>(</span><span>loaded_datasets1</span><span> </span><span>&lt;-</span><span> </span><span>readListFaster</span><span>(</span><span>list_of_datasets</span><span>,</span><span> </span><span>func</span><span> </span><span>=</span><span> </span><span>read.csv2</span><span>,</span><span> </span><span>parallelize</span><span>=</span><span>FALSE</span><span>,</span><span> </span><span>rbind</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>))</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">   user  system elapsed 
229.852  15.221 264.272 </code></pre>
</figure>
<h3>With Paralelize</h3>
<figure>
<pre><code data-lang="r"><span>system.time</span><span>(</span><span>loaded_datasets2</span><span> </span><span>&lt;-</span><span> </span><span>readListFaster</span><span>(</span><span>list_of_datasets</span><span>,</span><span> </span><span>func</span><span> </span><span>=</span><span> </span><span>read.csv2</span><span>,</span><span> </span><span>parallelize</span><span>=</span><span>TRUE</span><span>,</span><span> </span><span>rbind</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>))</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">   user  system elapsed 
 74.463  24.082 261.827 </code></pre>
</figure>
<p>It’s important to realise that every bit of optimisation matters. But it would not help us if the outcome data frames were different, don’t you agree? Luckily all 174 million records do match.</p>
<figure>
<pre><code data-lang="r"><span>table</span><span>(</span><span>loaded_datasets1</span><span> </span><span>==</span><span> </span><span>loaded_datasets2</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">     TRUE 
174053476 </code></pre>
</figure>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://danielmarcelino.github.io/2018/reading-list-faster.html"> Daniel Marcelino's Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/X6fz-_BF0g4" height="1"></a></div>