<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/sparklyr-0-6/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://blog.rstudio.com/2017/07/31/sparklyr-0-6/"> RStudio Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>We’re excited to announce a new release of the <a rel="nofollow" href="http://github.com/rstudio/sparklyr/" target="_blank">sparklyr</a> package, available in <a rel="nofollow" href="https://cran.r-project.org/package=sparklyr" target="_blank">CRAN</a> today! <code>sparklyr 0.6</code> introduces new features to:</p>
<ul>
<li><strong>Distribute R</strong> computations using <code>spark_apply()</code> to execute arbitrary R code across your Spark cluster. You can now use all of your favorite R packages and functions in a distributed context.</li>
<li>Connect to <strong>External Data Sources</strong> using <code>spark_read_source()</code>, <code>spark_write_source()</code>, <code>spark_read_jdbc()</code> and <code>spark_write_jdbc()</code>.</li>
<li><strong>Use the Latest Frameworks</strong> including <a rel="nofollow" href="https://blog.rstudio.com/2017/06/13/dplyr-0-7-0/" target="_blank">dplyr 0.7</a>, <a rel="nofollow" href="https://cran.r-project.org/package=DBI" target="_blank">DBI 0.7</a>, <a rel="nofollow" href="https://www.rstudio.com/products/rstudio/download/preview/" target="_blank">RStudio 1.1</a> and <a rel="nofollow" href="https://databricks.com/blog/2017/07/11/introducing-apache-spark-2-2.html" target="_blank">Spark 2.2</a>.</li>
</ul>
<p>and several improvements across:</p>
<ul>
<li><strong>Spark Connections</strong> add a new <a rel="nofollow" href="https://databricks.com/product/databricks" target="_blank">Databricks</a> connection that enables <a rel="nofollow" href="https://databricks.com/blog/2017/05/25/using-sparklyr-databricks.html" target="_blank">using sparklyr in Databricks</a> through <code>mode=&quot;databricks&quot;</code>, add support for <a rel="nofollow" href="https://spark.apache.org/docs/latest/running-on-yarn.html" target="_blank">Yarn Cluster</a> through <code>master=&quot;yarn-cluster&quot;</code> and connection speed was also improved.</li>
<li><strong>Dataframes</strong> add support for <code>sdf_pivot()</code>, <code>sdf_broadcast()</code>, <code>cbind()</code>, <code>rbind()</code>, <code>sdf_separate_column()</code>, <code>sdf_bind_cols()</code>, <code>sdf_bind_rows()</code>, <code>sdf_repartition()</code> and <code>sdf_num_partitions()</code>.</li>
<li><strong>Machine Learning</strong> adds support for multinomial regression in <code>ml_logistic_regression()</code>, <code>weights.column</code> for GLM, <code>ml_model_data()</code> and a new <code>ft_count_vectorizer()</code> function for <code>ml_lda()</code>.</li>
<li>Many other improvements, from initial support for <strong>broom</strong> over <code>ml_linear_regression()</code> and <code>ml_generalized_linear_regression()</code>, <strong>dplyr</strong> support for <code>%like%</code>, <code>%rlike%</code> and <code>%regexp%</code>, sparklyr <strong>extensions</strong> now support <code>download_scalac()</code> to help you install the required Scala compilers while developing extensions, Hive <strong>database</strong> management got simplified with <code>tbl_change_db()</code> and <code>src_databases()</code> to query and switch between Hive databases. RStudio started a joint effort with <a rel="nofollow" href="https://www.microsoft.com/" target="_blank">Microsoft</a> to support a <strong>cross-platform Spark installer</strong> under <a rel="nofollow" href="https://github.com/rstudio/spark-install" target="_blank">github.com/rstudio/spark-install</a>.</li>
</ul>
<p>Additional changes and improvements can be found in the <a rel="nofollow" href="https://github.com/rstudio/sparklyr/blob/master/NEWS.md" target="_blank">sparklyr NEWS</a> file.</p>
<p>Updated documentation and examples are available at <a rel="nofollow" href="http://spark.rstudio.com/" target="_blank">spark.rstudio.com</a>. For questions or feedback, please feel free to open a <a rel="nofollow" href="https://github.com/rstudio/sparklyr/issues" target="_blank">sparklyr github issue</a> or a <a rel="nofollow" href="http://stackoverflow.com/questions/tagged/sparklyr" target="_blank">sparklyr stackoverflow question</a>.</p>
<div>
<h2>Distributed R</h2>
<p><code>sparklyr 0.6</code> provides support for executing distributed R code through <code>spark_apply()</code>. For instance, after connecting and copying some data:</p>
<pre><code>library(sparklyr)
sc &lt;- spark_connect(master = &quot;local&quot;)
iris_tbl &lt;- sdf_copy_to(sc, iris)</code></pre>
<p>We can apply an arbitrary R function, say <code>jitter()</code>, to each column over each row as follows:</p>
<pre><code>iris_tbl %>% spark_apply(function(e) sapply(e[,1:4], jitter))</code></pre>
<pre><code>## # Source:   table</code></pre></div>