<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/tutorial-scalable-r-on-spark-with-sparkr-sparklyr-and-revoscaler/"></div></div><div><a data-url="https://www.r-bloggers.com/tutorial-scalable-r-on-spark-with-sparkr-sparklyr-and-revoscaler/" data-counturl="https://www.r-bloggers.com/tutorial-scalable-r-on-spark-with-sparkr-sparklyr-and-revoscaler/" data-count="horizontal" data-text="Tutorial: Scalable R on Spark with SparkR, sparklyr and RevoScaleR" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2016/10/tutorial-scalable-r-on-spark.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>If you'd like to manipulate and analyze very large data sets with the <a target="_blank" href="https://mran.microsoft.com/documents/what-is-r/">R language</a>, one option is to use R and <a target="_blank" href="http://spark.apache.org/">Apache Spark</a> together. R provides the simple, data-oriented language for specifying transformations and models; Spark provides the storage and computation engine to handle data much larger than R alone can handle.</p>
<p>At the KDD 2016 conference last October, a team from Microsoft presented a tutorial on <a target="_blank" href="http://www.kdd.org/kdd2016/tutorials/view/scalable-r-on-spark">Scalable R on Spark</a>, and made <a target="_blank" href="https://github.com/Azure/Azure-MachineLearning-DataScience/tree/master/Misc/KDDCup2016">all of the materials available on Github</a>. The materials include an 80-slide presentation covering several tutorials (you can <a target="_blank" href="https://github.com/Azure/Azure-MachineLearning-DataScience/blob/master/Misc/KDDCup2016/Slides/Scalable%20Data%20Analytics%20Using%20R%20Single%20Machines%20to%20Spark%20Clusters%20-%20Final.pptx?raw=true">download the 13Mb PowerPoint file here</a>).</p>
<p>Slides 1-29 form an introduction which covers:</p>
<ul>
<li>Scaling R scripts on a single machine with the bigmemory and ff packages</li>
<li>Interfacing to Spark from R with SparkR 1.6</li>
<li>Installing and using the <a target="_blank" href="http://spark.rstudio.com/">sparklyr package</a></li>
<li>Using Microsoft R Server and the &quot;RevoScaleR&quot; package to offload its computations to Spark</li>
<li>Comparisons and benchmarks of the techniques to scale R described above</li>
</ul>
<p><a target="_blank" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb0941fe21970d-pi"><img title="KDD comparison" alt="KDD comparison" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb0941fe21970d-800wi" border="0"></a></p>
<p>Slides 32-44 form a hands-on tutorial working with the airline arrival data to predict flight delays. In the tutorial, you use SparkR to clean and join the data, R Server's &quot;rxDTree&quot; function to fit a random forest model to predict delays, and then publish a prediction function to Azure with the AzureML package to create a cloud-based flight-delay prediction service. <a target="_blank" href="https://github.com/Azure/Azure-MachineLearning-DataScience/tree/master/Misc/KDDCup2016/Code/MRS">The Microsoft R scripts are available here</a>.</p>
<p>Slides 46-50 form another tutorial, this time working with the <a target="_blank" href="http://www.andresmh.com/nyctaxitrips">NYC Taxi dataset</a>. The <a target="_blank" href="https://github.com/Azure/Azure-MachineLearning-DataScience/blob/master/Misc/KDDCup2016/Code/SparkR/sparklyr_NYCTaxi.Rmd">first tutorial script</a> uses the sparklyr package to visualize the data and create models to predict the tip amount. This <a target="_blank" href="https://github.com/Azure/Azure-MachineLearning-DataScience/blob/master/Misc/KDDCup2016/Code/SparkR/SparkR_sparklyr_NYCTaxi.Rmd">second tutorial script</a> goes further with models, fitting Elastic Net, Random Forest and Gradient Boosted Tree models with both SparkR and sparklyr. In addition <a target="_blank" href="https://github.com/Azure/Azure-MachineLearning-DataScience/blob/master/Misc/KDDCup2016/Code/SparkSQL/SparkSQL.R">this script</a> uses SparkR and SparkSQL to create a map of the trips.</p>
<p>Slides 51-59 demonstrate optimizing the performance of a time series forecasting model, by searching over a large parameter space with the <a target="_blank" href="https://mran.microsoft.com/package/hts/">hts package</a>. By running the models in parallel to optimize the MAPE (mean absolute percent error), the total execution time was reduced to 1 day compared to the 40 days to complete the computations serially. The parallelization was achieved with the Microsoft R Server &quot;rxExec&quot; function, which you can replicate with <a target="_blank" href="https://github.com/Azure/Azure-MachineLearning-DataScience/blob/master/Misc/KDDCup2016/Code/UseCaseHTS/sample_demo.R">the script available here</a>.</p>
<p>Slides 60-72 includes background on calculating learning curves for various predictive models (see <a target="_blank" href="http://blog.revolutionanalytics.com/2015/09/why-big-data-learning-curves.html">here</a> and <a target="_blank" href="http://blog.revolutionanalytics.com/2016/03/learning-from-learning-curves.html">here</a> for more information). There's also a tutorial: <a target="_blank" href="https://github.com/Azure/Azure-MachineLearning-DataScience/tree/master/Misc/KDDCup2016/Code/learning_curves">the R scripts are available here</a>.</p>
<p>To work though the materials from the tutorial, you'll need access to a Spark cluster configured with Microsoft R Server and the necessary scripts and data files. You can easily create an HDInsight Premium cluster including Microsoft R Server on Microsoft Azure: these <a target="_blank" href="https://github.com/Azure/Azure-MachineLearning-DataScience/blob/master/Misc/KDDCup2016/Instructions%20for%20preparing%20HDI%20Spark%20clusters%20running%20MRS%20for%20KDD%202016%20tutorial.pdf">instructions provide the details</a>. Once the cluster is ready, you can remotely access it from your desktop using <code>ssh</code> as described. The clusters are charged by the hour (according to their size and power), so be sure to shut down the cluster when you're done with the tutorial.</p>
<p>These tutorials are hopefully useful to anyone who is trying to learn to use R with Spark. The full collection of materials and slides is available at the Github repository below.</p>
<p>Github (Azure): <a target="_blank" href="http://www.kdd.org/kdd2016/tutorials/view/scalable-r-on-spark">KDD 2016 tutorial: Scalable Data Science with R, from Single Nodes to Spark Clusters</a></p>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/tutorial-scalable-r-on-spark-with-sparkr-sparklyr-and-revoscaler/"></div></div><div><a data-url="https://www.r-bloggers.com/tutorial-scalable-r-on-spark-with-sparkr-sparklyr-and-revoscaler/" data-counturl="https://www.r-bloggers.com/tutorial-scalable-r-on-spark-with-sparkr-sparklyr-and-revoscaler/" data-count="vertical" data-text="Tutorial: Scalable R on Spark with SparkR, sparklyr and RevoScaleR" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2016/10/tutorial-scalable-r-on-spark.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/pce_L8X0Fnw"></a></div>