<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/tuning-apache-spark-for-faster-analysis-with-microsoft-r-server/"></div></div><div><a data-url="https://www.r-bloggers.com/tuning-apache-spark-for-faster-analysis-with-microsoft-r-server/" data-counturl="https://www.r-bloggers.com/tuning-apache-spark-for-faster-analysis-with-microsoft-r-server/" data-count="horizontal" data-text="Tuning Apache Spark for faster analysis with Microsoft R Server" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2016/08/tuning-apache-spark.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>My colleagues Max Kaznady, Jason Zhang, Arijit Tarafdar and Miguel Fierro recently posted a really useful guide with lots of <a target="_blank" href="https://blogs.msdn.microsoft.com/azuredatalake/2016/08/09/rapid-big-data-prototyping-with-microsoft-r-server-on-apache-spark-context-switching-spark-tuning/">tips to speed up prototyping models with Microsoft R Server on Apache Spark</a>. These tips apply when using <a target="_blank" href="https://azure.microsoft.com/en-us/services/hdinsight/apache-spark/">Spark on Azure HDInsight</a>, where you can spin up a Spark cluster the cloud with Microsoft R installed on the head node and worker nodes with just a single request on the Azure portal:</p>
<p><a target="_blank" href="http://a6.typepad.com/6a0105360ba1c6970c01bb0929c17e970d-pi"><img title="MRS cluster" alt="MRS cluster" src="http://a6.typepad.com/6a0105360ba1c6970c01bb0929c17e970d-400wi"></a></p>
<p>The blog post provides detailed instructions, but the three main efficiency tips are:</p>
<p><strong>Install RStudio Server on the head node</strong>. <a target="_blank" href="https://www.visualstudio.com/en-us/features/rtvs-vs.aspx">R Tools for Visual Studio</a> doesn't yet support remote execution, and while you can interact with the R command line directly via SSH or Putty, having an IDE makes things much easier and faster. There's a simple <a target="_blank" href="https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-r-server-install-r-studio">guide</a> to install <a target="_blank" href="https://www.rstudio.com/products/rstudio/#Server">RStudio Server</a> on the head node, whcih you can then use remotely to drive both the head node and the worker nodes.</p>
<p><strong>Use just the head node for iterating on your model</strong>. While you probably want to use all of the data to train your final model, you can speed up the iterative process of developing your model (selecting variables, creating features and transformations, and comparing performance of different model types) by working with a sample of your data on the head node only. While this mightn't have the predictive power of using all the data, it's more than sufficient to determine variable importance and compare performance between competing models. Ultimately, the more iterations you have time for when developing the model, the better the final production model (trained on all the data) will be.</p>
<p><a target="_blank" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d2102496970c-pi"><img title="Tree" alt="Tree" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d2102496970c-800wi" border="0"></a></p>
<p><strong>Tune the Spark cluster to optimize performance for model training</strong>. Now that you've saved time on the model development, save some time for the full-data model training by tuning the Spark cluster. (This is particularly useful if the production model is going to be retrained on a regular basis, say on a weekly or overnight schedule.) Max and Arjit provide some guidelines for setting the on-heap and off-heap memory settings for each executor to maintain the ideal ratio.</p>
<p>You can read the full details at the Azure blog post linked below, and you can try tuning a <a target="_blank" href="https://github.com/Azure/blog-mrs-spark-samples/blob/master/nyc_taxi_context_switch.R">sample analysis on the Taxi dataset yourself</a>, using the R code provided on GitHub.</p>
<p>Azure HDInsight blog: <a target="_blank" href="https://blogs.msdn.microsoft.com/azuredatalake/2016/08/09/rapid-big-data-prototyping-with-microsoft-r-server-on-apache-spark-context-switching-spark-tuning/">Rapid Big Data Prototyping with Microsoft R Server on Apache Spark: Context Switching &amp; Spark Tuning</a></p>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/tuning-apache-spark-for-faster-analysis-with-microsoft-r-server/"></div></div><div><a data-url="https://www.r-bloggers.com/tuning-apache-spark-for-faster-analysis-with-microsoft-r-server/" data-counturl="https://www.r-bloggers.com/tuning-apache-spark-for-faster-analysis-with-microsoft-r-server/" data-count="vertical" data-text="Tuning Apache Spark for faster analysis with Microsoft R Server" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2016/08/tuning-apache-spark.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/NeZ1Kcy0ZJ4"></a></div>