<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/extending-sparklyr-to-compute-cost-for-k-means-on-yarn-cluster-with-spark-ml-library/"></div></div><div><a data-url="https://www.r-bloggers.com/extending-sparklyr-to-compute-cost-for-k-means-on-yarn-cluster-with-spark-ml-library/" data-counturl="https://www.r-bloggers.com/extending-sparklyr-to-compute-cost-for-k-means-on-yarn-cluster-with-spark-ml-library/" data-count="horizontal" data-text="Extending sparklyr to Compute Cost for K-means on YARN Cluster with Spark ML Library" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://r-addict.com/2016/08/25/Extending-Sparklyr.html"> http://r-addict.com</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img src="https://i0.wp.com/r-addict.com/images/fulls/sparklyr.png?w=456" data-recalc-dims="1"> Machine and statistical learning wizards are becoming more eager to perform analysis with <a target="_blank" href="https://spark.apache.org/docs/2.0.0/ml-clustering.html">Spark ML</a> library if this is only possible. It’s trendy, posh, spicy and gives the feeling of doing state of the art machine learning and being up to date with the newest computational trends. It is even more sexy and powerful when computations can be performed on the extraordinarily enormous computation cluster – let’s say 100 machines on <a target="_blank" href="https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/YARN.html">YARN hadoop</a> cluster makes you the real data cruncher! In this post I present <a target="_blank" href="http://spark.rstudio.com/">sparklyr</a> package (by <a target="_blank" href="http://r-addict.com/2016/08/25/github.com/rstudio/sparklyr">RStudio</a>), the connector that will transform you from a regular R user, to the supa! data scientist that can invoke Scala code to perform machine learning algorithms on YARN cluster just from RStudio! Moreover, I present how I have extended the interface to K-means procedure, so that now it is also possible to compute cost for that model, which might be beneficial in determining the number of clusters in segmentation problems. <strong>Thought about learnig Scala? Leave it – user sparklyr!</strong></p>
<ul>
<li><a target="_blank" href="http://r-addict.com/2016/08/25/Extending-Sparklyr.html#sparklyr-basics">sparklyr basics</a></li>
<li><a target="_blank" href="http://r-addict.com/2016/08/25/Extending-Sparklyr.html#dplyr-and-dbi-interface-on-spark">dplyr and DBI interface on Spark</a></li>
<li><a target="_blank" href="http://r-addict.com/2016/08/25/Extending-Sparklyr.html#running-spark-ml-machine-learning-k-means-algorithm-from-r">Running Spark ML Machine Learning K-means Algorithm from R</a></li>
</ul>
<p>If you don’t know much about Spark yet, you can read my April post <a target="_blank" href="http://r-addict.com/2016/04/06/FAQ-SparkR.html">Answers to FAQ about SparkR for R users</a> – where I explained how could we use <code>SparkR</code> package that is distributed with Spark. Many things (code) might have changed since that time, due to the rapid development caused by great popularity of Spark. Now we can use version 2.0.0 of Spark. If you are migrating from previous versions I suggest you should look at <a target="_blank" href="http://spark.apache.org/docs/latest/sparkr.html#migration-guide">Migration Guide – Upgrading From SparkR 1.6.x to 2.0</a>.</p>
<h1>sparklyr basics</h1>
<p>This packages is based on <a target="_blank" href="http://github.com/rstudio/sparkapi/">sparkapi</a> package that enables to run Spark applications locally or on YARN cluster just from R. It translates R code to bash invocation of spark-shell. It’s biggest advantage is <a target="_blank" href="https://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html">dplyr</a> interface for working with Spark Data Frames (that might be Hive Tables) and possibility to invoke algorithms from Spark ML library.</p>
<p>Installation of sparklyr, then Spark itself and simple application initiation is described by this code</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>devtools</span><span>)</span><span>
</span><span>install_github</span><span>(</span><span>'rstudio/sparklyr'</span><span>)</span><span>
</span><span>library</span><span>(</span><span>sparklyr</span><span>)</span><span>
</span><span>spark_install</span><span>(</span><span>version</span><span> </span><span>=</span><span> </span><span>&quot;2.0.0&quot;</span><span>)</span><span>
</span><span>sc</span><span> </span><span>&lt;-</span><span> </span><span>spark_connect</span><span>(</span><span>master</span><span>=</span><span>&quot;yarn&quot;</span><span>,</span><span>
</span><span>config</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>
</span><span>default</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>
</span><span>spark.submit.deployMode</span><span>=</span><span> </span><span>&quot;client&quot;</span><span>,</span><span>
</span><span>spark.executor.instances</span><span>=</span><span> </span><span>20</span><span>,</span><span> </span><span>spark.executor.memory</span><span>=</span><span> </span><span>&quot;2G&quot;</span><span>,</span><span>
</span><span>spark.executor.cores</span><span>=</span><span> </span><span>4</span><span>,</span><span>
</span><span>spark.driver.memory</span><span>=</span><span> </span><span>&quot;4G&quot;</span><span>)))</span></code></pre>
</figure>
<p>One don’t have to specify config by himself, but if this is desired then remember that you could also specify parameters for Spark application with <a target="_blank" href="http://spark.rstudio.com/deployment.html#configuration">config.yml</a> files so that you can benefit from many profiles (development, production). In version 2.0.0 it is desired to name master <code>yarn</code> instead of <code>yarn-client</code> and passing the <code>deployMode</code> parameter, which is different from version 1.6.x. All available parameters can be found in <a target="_blank" href="http://spark.apache.org/docs/latest/running-on-yarn.html">Running Spark on YARN</a> documentation page.</p>
<h1>dplyr and DBI interface on Spark</h1>
<p>When connecting to YARN, it is most probable that you would like to use data tables that are stored on Hive. Remember that</p>
<blockquote>
<p>Configuration of Hive is done by placing your hive-site.xml, core-site.xml (for security configuration), and hdfs-site.xml (for HDFS configuration) file in conf/.</p>
</blockquote>
<p>where <code>conf/</code> is set as <code>HADOOP_CONF_DIR</code>. Read more about using <a target="_blank" href="http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables">Hive tables from Spark</a></p>
<p>If everything is set up and the application runs properly, you can use dplyr interface to provide lazy evaluation for data manipulations. Data are stored on Hive, Spark application runs on YARN cluster, and the code is invoked from R in the simple language of data transformations (dplyr) – everything thanks to sparklyr team great job! Easy example is below</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span># give the list of tables
</span><span>src_tbls</span><span>(</span><span>sc</span><span>)</span><span> </span><span># copies iris from R to Hive
</span><span>iris_tbl</span><span> </span><span>&lt;-</span><span> </span><span>copy_to</span><span>(</span><span>sc</span><span>,</span><span> </span><span>iris</span><span>,</span><span> </span><span>&quot;iris&quot;</span><span>)</span><span> </span><span># create a hook for data stored on Hive
</span><span>data_tbl</span><span> </span><span>&lt;-</span><span> </span><span>tbl</span><span>(</span><span>sc</span><span>,</span><span> </span><span>&quot;table_name&quot;</span><span>)</span><span> </span><span>data_tbl2</span><span> </span><span>&lt;-</span><span> </span><span>tbl</span><span>(</span><span>sc</span><span>,</span><span> </span><span>sql</span><span>(</span><span>&quot;SELECT * from table_name&quot;</span><span>))</span></code></pre>
</figure>
<p>You can also perform any operation on datasets use by Spark</p>
<figure>
<pre><code data-lang="r"><span>iris_tbl</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>Petal_Length</span><span>,</span><span> </span><span>Petal_Width</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>top_n</span><span>(</span><span>40</span><span>,</span><span> </span><span>Petal_Width</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>Petal_Length</span><span>)</span></code></pre>
</figure>
<p>Note that original commas in iris names have been translated to <code>_</code>.</p>
<p>This package also provides interface for functions defined in <code>DBI</code> package</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>DBI</span><span>)</span><span>
</span><span>dbListTables</span><span>(</span><span>sc</span><span>)</span><span>
</span><span>dbGetQuery</span><span>(</span><span>sc</span><span>,</span><span> </span><span>&quot;use database_name&quot;</span><span>)</span><span>
</span><span>data_tbl3</span><span> </span><span>&lt;-</span><span> </span><span>dbGetQuery</span><span>(</span><span>sc</span><span>,</span><span> </span><span>&quot;SELECT * from table_name&quot;</span><span>)</span><span>
</span><span>dbListFields</span><span>(</span><span>sc</span><span>,</span><span> </span><span>data_tbl3</span><span>)</span></code></pre>
</figure>
<h1>Running Spark ML Machine Learning K-means Algorithm from R</h1>
<p>The basic example on how sparklyr invokes Scala code from Spark ML will be presented on K-means algorithm.<br>
If you check the code of <code>sparklyr::ml_kmeans</code> function you will see that for input <code>tbl_spark</code> object, named x and character vector containing features’ names (<code>featuers</code>)</p>
<figure>
<pre><code data-lang="r"><span>envir</span><span> </span><span>&lt;-</span><span> </span><span>new.env</span><span>(</span><span>parent</span><span> </span><span>=</span><span> </span><span>emptyenv</span><span>())</span><span>
</span><span>df</span><span> </span><span>&lt;-</span><span> </span><span>spark_dataframe</span><span>(</span><span>x</span><span>)</span><span>
</span><span>sc</span><span> </span><span>&lt;-</span><span> </span><span>spark_connection</span><span>(</span><span>df</span><span>)</span><span>
</span><span>df</span><span> </span><span>&lt;-</span><span> </span><span>ml_prepare_features</span><span>(</span><span>df</span><span>,</span><span> </span><span>features</span><span>)</span><span>
</span><span>tdf</span><span> </span><span>&lt;-</span><span> </span><span>ml_prepare_dataframe</span><span>(</span><span>df</span><span>,</span><span> </span><span>features</span><span>,</span><span> </span><span>ml.options</span><span> </span><span>=</span><span> </span><span>ml.options</span><span>,</span><span> </span><span>envir</span><span> </span><span>=</span><span> </span><span>envir</span><span>)</span></code></pre>
</figure>
<p>sparklyr ensures that you have proper connection to spark data frame and prepares features in convenient form and naming convention. At the end it prepares a Spark DataFrame for Spark ML routines.</p>
<p>This is done in a new environment, so that we can store arguments for future ML algorithm and the model itself in its own environment. This is safe and clean solution. You can construct a simple model calling a Spark ML class like this</p>
<figure>
<pre><code data-lang="r"><span>envir</span><span>$</span><span>model</span><span> </span><span>&lt;-</span><span> </span><span>&quot;org.apache.spark.ml.clustering.KMeans&quot;</span><span>
</span><span>kmeans</span><span> </span><span>&lt;-</span><span> </span><span>invoke_new</span><span>(</span><span>sc</span><span>,</span><span> </span><span>envir</span><span>$</span><span>model</span><span>)</span></code></pre>
</figure>
<p>which invokes new object of class <code>KMeans</code> on which we can invoke parameters setters to change default parameters like this</p>
<figure>
<pre><code data-lang="r"><span>model</span><span> </span><span>&lt;-</span><span> </span><span>kmeans</span><span> </span><span>%>%</span><span>
</span><span>invoke</span><span>(</span><span>&quot;setK&quot;</span><span>,</span><span> </span><span>centers</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>invoke</span><span>(</span><span>&quot;setMaxIter&quot;</span><span>,</span><span> </span><span>iter.max</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>invoke</span><span>(</span><span>&quot;setTol&quot;</span><span>,</span><span> </span><span>tolerance</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>invoke</span><span>(</span><span>&quot;setFeaturesCol&quot;</span><span>,</span><span> </span><span>envir</span><span>$</span><span>features</span><span>)</span><span>
</span><span>#</span><span> </span><span>features</span><span> </span><span>where</span><span> </span><span>set</span><span> </span><span>in</span><span> </span><span>ml_prepare_dataframe</span></code></pre>
</figure>
<p>For an existing object of <code>KMeans</code> class we can invoke its method called <code>fit</code> that is responsible for starting the K-means clustering algorithm</p>
<figure>
<pre><code data-lang="r"><span>fit</span><span> </span><span>&lt;-</span><span> </span><span>model</span><span> </span><span>%>%</span><span>
</span><span>invoke</span><span>(</span><span>&quot;fit&quot;</span><span>,</span><span> </span><span>tdf</span><span>)</span></code></pre>
</figure>
<p>which returns new object on which we can compute, e.g centers of outputted clustering</p>
<figure>
<pre><code data-lang="r"><span>kmmCenters</span><span> </span><span>&lt;-</span><span> </span><span>invoke</span><span>(</span><span>fit</span><span>,</span><span> </span><span>&quot;clusterCenters&quot;</span><span>)</span></code></pre>
</figure>
<p>or the Within Set Sum of Squared Errors (called Cost) (which is mine small contribution <a target="_blank" href="https://github.com/rstudio/sparklyr/pull/173">#173</a> )</p>
<figure>
<pre><code data-lang="r"><span>kmmCost</span><span> </span><span>&lt;-</span><span> </span><span>invoke</span><span>(</span><span>fit</span><span>,</span><span> </span><span>&quot;computeCost&quot;</span><span>,</span><span> </span><span>tdf</span><span>)</span></code></pre>
</figure>
<p>This sometimes helps to decide <a target="_blank" href="http://stackoverflow.com/a/15376462/3857701">how many clusters should we specify for clustering problem</a></p>
<p><img src="https://i0.wp.com/r-addict.com/images/fulls/kmeans.png?w=456" data-recalc-dims="1"></p>
<p>and is presented in <code>print</code> method for <code>ml_model_kmeans</code> object</p>
<figure>
<pre><code data-lang="r"><span>iris_tbl</span><span> </span><span>%>%</span><span>
</span><span>select</span><span>(</span><span>Petal_Width</span><span>,</span><span> </span><span>Petal_Length</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ml_kmeans</span><span>(</span><span>centers</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span> </span><span>compute.cost</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>print</span><span>()</span><span>
</span><span>K</span><span>-</span><span>means</span><span> </span><span>clustering</span><span> </span><span>with</span><span> </span><span>3</span><span> </span><span>clusters</span><span>
</span><span>Cluster</span><span> </span><span>centers</span><span>:</span><span>
</span><span>Petal_Width</span><span> </span><span>Petal_Length</span><span>
</span><span>1</span><span> </span><span>1.359259</span><span> </span><span>4.292593</span><span>
</span><span>2</span><span> </span><span>2.047826</span><span> </span><span>5.626087</span><span>
</span><span>3</span><span> </span><span>0.246000</span><span> </span><span>1.462000</span><span>
</span><span>Within</span><span> </span><span>Set</span><span> </span><span>Sum</span><span> </span><span>of</span><span> </span><span>Squared</span><span> </span><span>Errors</span><span> </span><span>=</span><span> </span><span>31.41289</span></code></pre>
</figure>
<p>All that can be better understood if we’ll have a look on <a target="_blank" href="https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.ml.clustering.KMeans">Spark ML docuemtnation for KMeans</a> (<a target="_blank" href="https://github.com/rstudio/sparklyr/issues/178#issuecomment-240472368">be carefull not to confuse with Spark MLlib</a> where methods and parameters have different names than those in Spark ML). This enabled me to provide simple update for <code>ml_kmeans()</code> (<a target="_blank" href="https://github.com/rstudio/sparklyr/pull/179">#179</a>) so that we can specify <code>tol</code> (tolerance) parameter in <code>ml_kmeans()</code> to support tolerance of convergence.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/extending-sparklyr-to-compute-cost-for-k-means-on-yarn-cluster-with-spark-ml-library/"></div></div><div><a data-url="https://www.r-bloggers.com/extending-sparklyr-to-compute-cost-for-k-means-on-yarn-cluster-with-spark-ml-library/" data-counturl="https://www.r-bloggers.com/extending-sparklyr-to-compute-cost-for-k-means-on-yarn-cluster-with-spark-ml-library/" data-count="vertical" data-text="Extending sparklyr to Compute Cost for K-means on YARN Cluster with Spark ML Library" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://r-addict.com/2016/08/25/Extending-Sparklyr.html"> http://r-addict.com</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/vJdeGyUg4TE"></a></div>