<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/new-series-r-and-big-data-concentrating-on-spark-and-sparklyr/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2017/05/new-series-r-and-big-data-concentrating-on-spark-and-sparklyr/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a rel="nofollow" href="http://www.win-vector.com/" target="_blank">Win-Vector LLC</a> has recently been teaching how to use <a rel="nofollow" href="https://cran.r-project.org/" target="_blank"><code>R</code></a> with big data through <a rel="nofollow" href="https://github.com/WinVector/BigDataRStrata2017" target="_blank"><code>Spark</code> and <code>sparklyr</code></a>. We have also been helping clients become productive on <code>R/Spark</code> infrastructure through direct consulting and bespoke training. I thought this would be a good time to talk about the power of working with big-data using <code>R</code>, share some hints, and even admit to some of the warts found in this combination of systems. </p>
<p>The ability to perform sophisticated analyses and modeling on “big data” with <code>R</code> is rapidly improving, and this is the time for businesses to invest in the technology. Win-Vector can be your key partner in methodology development and training (through our consulting and training practices).</p>
<p><center><br>
<a rel="nofollow" href="https://en.wikipedia.org/wiki/We_Can_Do_It!" target="_blank"><img border="0" alt="We Can Do It" title="We_Can_Do_It!.jpg" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/05/We_Can_Do_It.jpg?resize=399%2C517" data-recalc-dims="1">
<p>J. Howard Miller’s, 1943.</a><br>
</center></p>
<p>The field is exciting, rapidly evolving, and even a touch dangerous. We invite you to start using <code>Spark</code> through <code>R</code> and are starting a new series of articles tagged “R and big data” to help you produce production quality solutions quickly.</p>
<p>Please read on for a brief description of our new articles series: “R and big data.”<span></span></p>
<h2>Background</h2>
<p><code>R</code> is a best of breed in-memory analytics platform. <code>R</code> allows the analyst to write programs that operate over their data and bring in a huge suite of powerful statistical techniques and machine learning procedures. <code>Spark</code> is an analytics platform designed to operate over big data that exposes some of its own statistical and machine learning capabilities. <code>R</code> can now be operated “over <code>Spark</code>“. That is: <code>R</code> programs can delegate tasks to <code>Spark</code> clusters and issue commands to <code>Spark</code> clusters. In some cases the syntax for operating over <code>Spark</code> is deliberately identical to working over data stored in <code>R</code>.</p>
<h2>Why <code>R</code> and <code>Spark</code></h2>
<p>The advantages are: </p>
<ul>
<li><code>Spark</code> can work at a scale and speed far larger than native <code>R</code> . The ability to send work to <code>Spark</code> increases <code>R</code>‘s capabilities.</li>
<li><code>R</code> has machine learning and statistical capabilities that go far beyond what is available on <code>Spark</code> or any other “big data” system (many of which are descended from report generation or basic analytics). The ability to use specialized <code>R</code> methods on data samples yields additional capabilities.</li>
<li><code>R</code> and <code>Spark</code> can share code and data.</li>
</ul>
<p>The <code>R</code>/<code>Spark</code> combination is not the only show in town; but it is a powerful capability that may not be safe to ignore. We will also talk about additional tools that can be brought into the mix: such as the powerful large scale machine learning capabilities from <code>h2o</code></p>
<h2>The warts</h2>
<p>Frankly a lot of this is very new, and still on the “bleeding edge.” <code>Spark 2.x</code> has only been <a rel="nofollow" href="http://spark.apache.org/downloads.html" target="_blank">available in stable form since July 26, 2016</a> (or just under a year). <code>Spark 2.x</code> is much more capable than the <code>Spark 1.x</code> series in terms of both data manipulation and machine learning, so we strongly suggest clients strongly insist on <code>Spark 2.x</code> clusters from their infrastructure vendors (such as Couldera, Hortonworks, MapR, and others) despite having only become available in these packaged solutions recently. The <code>sparklyr</code> adapter itself was first available on <code>CRAN</code> only as of September 24th, 2016. And <code>SparkR</code> only <a rel="nofollow" href="https://databricks.com/blog/2015/06/09/announcing-sparkr-r-on-spark.html" target="_blank">started distributing with <code>Spark 1.4</code> as of June 2015</a>.</p>
<p>While <code>R</code>/<code>Spark</code> is indeed a powerful combination, nobody seems to sharing a lot of production experiences and best practices whith it yet.</p>
<p>Some of the problems are sins of optimism. A lot of people still confuse successfully standing a cluster up with effectively using it. Other people confuse statistical and procedures available in in-memory <code>R</code> (which are very broad and often quite mature) with those available in <code>Spark</code> (which are less numerous and less mature). </p>
<h2>Our goal</h2>
<p>What we want to do with the “<code>R</code> and big data” series is:</p>
<ul>
<li>Give a taste of some of the power of the <code>R</code>/<code>Spark</code> combination.</li>
<li>Share a “capabilities and readiness” checklist you should apply when evaluating infrastructure.</li>
<li>Start to publicly document <code>R</code>/<code>Spark</code> best practices.</li>
<li>Describe some of the warts and how to work around them.</li>
<li>Share fun tricks and techniques that make working with <code>R</code>/<code>Spark</code> much easier and more effective.</li>
</ul>
<h2>The start</h2>
<p>Our next article in this series will be up soon and will discuss the nature of data-handles in <code>Sparklyr</code> (one of the <code>R</code>/<code>Spark</code> interfaces) and how to manage your data inventory neatly.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2017/05/new-series-r-and-big-data-concentrating-on-spark-and-sparklyr/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/UYulhAJuzCY" height="1"></a></div>