<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/make-pleasingly-parallel-r-code-with-rxexecby/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2017/04/rxexecby.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>Some things are easy to convert from a long-running sequential process to a system where each part runs at the same time, thus reducing the required time overall. We often call these &quot;<a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Embarrassingly_parallel">embarrassingly parallel</a>&quot; problems, but given how easy it is to reduce the time it takes to execute them by converting them into a parallel process, &quot;pleasingly parallel&quot; may well be a more appropriate name.</p>
<p>Using the <a target="_blank" rel="nofollow" href="https://mran.microsoft.com/package/foreach/">foreach package</a> (available on CRAN) is one simple way of speeding up pleasingly parallel problems using R. A <a target="_blank" rel="nofollow" href="https://www.rdocumentation.org/packages/foreach/versions/1.4.3/topics/foreach"><code>foreach</code></a> loop is much like a regular <code>for</code> loop in R, and by default will run each iteration in sequence (again, just like a <code>for</code> loop). But by registering a parallel &quot;backend&quot; for foreach, you can run many (or maybe even all) iterations at the same time, using <a target="_blank" rel="nofollow" href="https://www.rdocumentation.org/packages/doParallel/versions/1.0.10">multiple processors on the same machine</a>, or even <a target="_blank" rel="nofollow" href="http://blog.revolutionanalytics.com/2017/03/doazureparallel.html">multiple machines in the cloud</a>.</p>
<p>For many applications, though, you need to provide a different chunk of data to each iteration to process. (For example, you may need to fit a statistical model within each country — each iteration will then only need the subset for one country.) You could just pass the entire data set into each iteration and subset it there, but that's inefficient and may even be impractical when dealing with very large datasets sitting in a remote repository. A better idea would be to leave the data where it is, and run R <em>within</em> the data repository, in parallel.</p>
<p>Microsoft R 9.1 introduces a new function, <code><a target="_blank" rel="nofollow" href="https://msdn.microsoft.com/en-us/microsoft-r/scaler/packagehelp/rxexecby">rxExecBy</a></code>, for exactly this purpose. When your data is sitting in SQL Server or Spark, you can specify a set of keys to partition the data by, and an R function (any R function, built-in or user-defined) to apply to the partitions. The data doesn't actually move: R runs directly on the data platform. You can also run it on local data in various formats</p>
<p><a target="_blank" rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c8f2b7cb970b-pi"><img title="RxExecBy" alt="RxExecBy" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c8f2b7cb970b-800wi" border="0"></a></p>
<p>The <code>rxExecBy</code> function is included in <a target="_blank" rel="nofollow" href="https://msdn.microsoft.com/en-us/microsoft-r/r-client-get-started">Microsoft R Client</a> (available free) and <a target="_blank" rel="nofollow" href="https://www.microsoft.com/en-us/cloud-platform/r-server">Microsoft R Server</a>. For some examples of using <code>rxExecBy</code>, take a look at the Microsoft R Blog post linked below.</p>
<p>Microsoft R Blog: <a target="_blank" rel="nofollow" href="https://blogs.msdn.microsoft.com/rserver/2017/04/12/pleasingly-parallel-using-rxexecby/">Running Pleasingly Parallel workloads using rxExecBy on Spark, SQL, Local and Localpar compute contexts</a></p>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2017/04/rxexecby.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/ST_4GeKCm10"></a></div>