<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/spark-2-0-more-performance-more-statistical-models/"></div></div><div><a data-url="http://www.r-bloggers.com/spark-2-0-more-performance-more-statistical-models/" data-counturl="http://www.r-bloggers.com/spark-2-0-more-performance-more-statistical-models/" data-count="horizontal" data-text="Spark 2.0: more performance, more statistical models" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2016/05/spark-20-to-include-more-r-models.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p><a target="_blank" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c85e06a8970b-pi"><img title="Spark-logo" alt="Spark-logo" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c85e06a8970b-800wi" border="0"></a><a target="_blank" href="http://spark.apache.org/">Apache Spark</a>, the open-source cluster computing framework, will soon see a major update with the <a target="_blank" href="https://databricks.com/blog/2016/05/11/spark-2-0-technical-preview-easier-faster-and-smarter.html">upcoming release of Spark 2.0</a>. This update promises to be faster than Spark 1.6, thanks to a run-time compiler that generates optimized bytecode. It also promises to be easier for developers to use, with streamlined APIs and a more complete SQL implementation. (Here's a <a target="_blank" href="https://azure.microsoft.com/en-us/documentation/articles/hdinsight-apache-spark-jupyter-spark-sql/">tutorial on using SQL with Spark</a>.) Spark 2.0 will also include a new &quot;structured streaming&quot; API, which will allow developers to write algorithm for streaming data without having to worry about the fact that streaming data is always incomplete; algorithms written for complete DataFrame objects will work for streams as well.</p>
<p>This update also includes some news for R users. First, the <a target="_blank" href="http://blog.revolutionanalytics.com/2015/06/sparkr-announcement.html">DataFrame object</a> continues to be the primary interface for R (and Python users). Although the DataSets structure in Spark is more general, using the single-table DataFrame construct makes sense for R and Python which have analogous native (or near-native, in Python's case) data structures. In addition, Spark 2.0 is set to add a few new distributed statistical modeling algorithms: generalized linear models (in addition to the Normal least-squares and logistic regression models in Spark 1.6); Naive Bayes; survival (censored) regression; and K-means clustering. The addition of survival regression is particularly interesting. It's a type of model used in situations where the outcome isn't always completely known: for example, some (but not all) patients may have yet experienced remission in a cancer study. It's also used for reliability analysis and lifetime estimation in manufacturing, where some (but not all) parts may have failed by the end of the observation period. To my knowledge, this is the first distributed implementation of the survival regression algorithm.</p>
<p>For R users, these models can be applied to large data sets stored in Spark DataFrames, and then computed using the Spark distributed computing framework. Access to the algorithms is via the <a target="_blank" href="https://spark.apache.org/docs/latest/sparkr.html#machine-learning">SparkR package</a>, which hews closely to standard R interfaces for model training. You'll need to first create a <a target="_blank" href="https://people.apache.org/~pwendell/spark-nightly/spark-master-docs/latest/sparkr.html#upgrading-from-sparkr-16x-to-20">SparkDataFrame object in R</a>, as a reference to a Spark DataFrame. Then, to perform logistic regression (for example), you'll use R's standard glm function, using the SparkDataFrame object as the data argument. (This elegantly uses R's object-oriented dispatch architecture to call the Spark-specific GLM code.) The example below creates a Spark DataFrame, and then uses Spark to fit a logisic regression to it:</p>
<pre><code data-lang="r">df <span>&lt;-</span> createDataFrame<span>(</span>sqlContext<span>,</span> iris<span>)</span>
model <span>&lt;-</span> glm<span>(</span>Species <span>~</span> Sepal_Length <span>+</span> Sepal_Width<span>,</span> data <span>=</span> training<span>,</span> family <span>=</span> <span>&quot;binomial&quot;</span><span>)</span></code></pre>
<p>The object <span>model</span> contains most (but not all) of the output created by R's traditional glm algorithm, so most standard R functions that work with GLM objects work here as well.</p>
<p>For more on what's coming in Spark 2.0 check out the DataBricks blog post below, and the <a target="_blank" href="https://people.apache.org/~pwendell/spark-nightly/spark-master-docs/latest/sparkr.html">preview documentation for SparkR</a> contains more info as well. Also, you might want to check out how you can use <a target="_blank" href="https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-r-server-compute-contexts/">Spark on HDInsight with Microsoft R Server</a>.</p>
<p>Databricks: <a target="_blank" href="https://databricks.com/blog/2016/05/11/spark-2-0-technical-preview-easier-faster-and-smarter.html">Preview of Apache Spark 2.0 now on Databricks Community Edition</a></p>
<p> </p>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/spark-2-0-more-performance-more-statistical-models/"></div></div><div><a data-url="http://www.r-bloggers.com/spark-2-0-more-performance-more-statistical-models/" data-counturl="http://www.r-bloggers.com/spark-2-0-more-performance-more-statistical-models/" data-count="vertical" data-text="Spark 2.0: more performance, more statistical models" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2016/05/spark-20-to-include-more-r-models.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/4vffKZNcm5M"></a></div>