<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/understanding-overhead-issues-in-parallel-computation/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://matloff.wordpress.com/2017/07/29/understanding-overhead-issues-in-parallel-computation/"> Mad (Data) Scientist</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In my talk at useR! earlier this month, I emphasized the fact that a major impediment to obtaining good speed from parallelizing an algorithm is systems overhead of various kinds, including:</p>
<ul>
<li>Contention for memory/network.</li>
<li>Bandwidth limits — CPU/memory, CPU/network, CPU/GPU.</li>
<li>Cache coherency problems.</li>
<li>Contention for I/O ports.</li>
<li>OS and/or R limits on number of sockets (network connections).</li>
<li>Serialization.</li>
</ul>
<p>During the Q&amp;A at the end, one person in the audience asked how R programmers without a computer science background might acquire this information. A similar question was posed here today by a reader on this blog, to which I replied,</p>
<blockquote><p>That question was asked in my talk. I answered by saying that I have an introduction to such things in my book, but that this is not enough. One builds this knowledge in haphazard ways, e.g. by search terms like “cache miss” and “network latency” on the Web, and above all, by giving it careful thought and reasoning things out. (When Nobel laureate Richard Feynman was a kid, someone said in awe, “He fixes radios by thinking!”)</p>
<p>Join an R Users Group, if there is one in your area. (And if not, then start one!) Talk about these things with them (though if you follow my above advice, you may find you soon know more than they do).</p>
</blockquote>
<p>The book I was referring to was <em>Parallel Computing for Data Science: With Examples in R, C++ and CUDA</em> (Chapman &amp; Hall/CRC, The R Series, Jun 4, 2015.</p>
<p><strong>I have decided that the topic of system overhead issues in parallel computation is important enough for me to <a rel="nofollow" href="http://heather.cs.ucdavis.edu/rparbookch2.pdf" target="_blank">place Chapter 2 on the Web</a>, which I have now done. </strong>Enjoy. I’d be happy to answer your questions (of a general nature, not on your specific code).</p>
<p>We are continuing to add more features to our R parallel computation package, <strong><a rel="nofollow" href="https://github.com/matloff/partools" target="_blank">partools</a></strong>. Watch this space for news!</p>
<p>By the way, the useR! 2017 videos are now <a rel="nofollow" href="https://channel9.msdn.com/events/useR-international-R-User-conferences/useR-International-R-User-2017-Conference" target="_blank">on the Web</a>, including <a rel="nofollow" href="https://channel9.msdn.com/events/useR-international-R-User-conferences/useR-International-R-User-2017-Conference/KEYNOTE-Parallel-Computation-in-R-What-We-Want-and-How-We-Might-Get-It?term=Matloff" target="_blank">my talk on parallel computing</a>.</p>
<p> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/matloff.wordpress.com/1940/" target="_blank"><img border="0" alt="" src="http://feeds.wordpress.com/1.0/comments/matloff.wordpress.com/1940/"></a> <img border="0" alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" data-recalc-dims="1"></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://matloff.wordpress.com/2017/07/29/understanding-overhead-issues-in-parallel-computation/"> Mad (Data) Scientist</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/P0SNSWCSdbM" height="1"></a></div>