<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/sparklyr-0-7/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://blog.rstudio.com/2018/01/29/sparklyr-0-7/"> RStudio Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>We are excited to share that <a rel="nofollow" href="https://cran.r-project.org/web/packages/sparklyr/index.html" target="_blank">sparklyr 0.7</a> is now available on CRAN! Sparklyr provides an R interface to Apache Spark. It supports dplyr syntax for working with Spark DataFrames and exposes the full range of machine learning algorithms available in Spark. You can also learn more about Apache Spark and sparklyr in <a rel="nofollow" href="http://spark.rstudio.com/" target="_blank">spark.rstudio.com</a> and our new <a rel="nofollow" href="https://www.rstudio.com/resources/webinars/introducing-an-r-interface-for-apache-spark/" target="_blank">webinar series on Apache Spark</a>. Features in this release:</p>
<ul>
<li>Adds support for <strong>ML Pipelines</strong> which provide a uniform set of high-level APIs to help create, tune, and deploy machine learning pipelines at scale.</li>
<li>Enhances <strong>Machine Learning</strong> capabilities by supporting the full range of ML algorithms and feature transformers.</li>
<li>Improves <strong>Data Serialization</strong>, specifically by adding support for date columns.</li>
<li>Adds support for <a rel="nofollow" href="https://spark.rstudio.com/guides/connections/#cluster-mode" target="_blank">YARN cluster mode</a> connections.</li>
<li>Adds various other improvements as listed in the <a rel="nofollow" href="http://spark.rstudio.com/news/" target="_blank">NEWS</a> file.</li>
</ul>
<p>In this blog post, we highlight Pipelines, new ML functions, and enhanced support for data serialization. To follow along in the examples below, you can upgrade to the latest stable version from CRAN with:</p>
<pre><code>install.packages(&quot;sparklyr&quot;)</code></pre>
<div>
<h2>ML Pipelines</h2>
<p>The <a rel="nofollow" href="https://spark.apache.org/docs/latest/ml-pipeline.html" target="_blank">ML Pipelines</a> API is a high-level interface for building ML workflows in Spark. Pipelines provide a uniform approach to compose feature transformers and ML routines, and are interoperable across the different Spark APIs (R/sparklyr, Scala, and Python.)</p>
<p>First, let’s go over a quick overview of terminology. A <code>Pipeline</code> consists of a sequence of stages—<code>PipelineStage</code>s—that act on some data in order. A <code>PipelineStage</code> can be either a <code>Transformer</code> or an <code>Estimator</code>. A <code>Transformer</code> takes a data frame and returns a transformed data frame, whereas an <code>Estimator</code> take a data frame and returns a <code>Transformer</code>. You can think of an <code>Estimator</code> as an algorithm that can be fit to some data, e.g. the ordinary least squares (OLS) method, and a <code>Transformer</code> as the fitted model, e.g. the linear formula that results from OLS. A <code>Pipeline</code> is itself a <code>PipelineStage</code> and can be an element in another <code>Pipeline</code>. Lastly, a <code>Pipeline</code> is always an <code>Estimator</code>, and its fitted form is called <code>PipelineModel</code> which is a <code>Transformer</code>.</p>
<p>Let’s look at some examples of creating pipelines. We establish a connection and copy some data to Spark:</p>
<pre><code>library(sparklyr)
library(dplyr)

# If needed, install Spark locally via `spark_install()`
sc &lt;- spark_connect(master = &quot;local&quot;)
iris_tbl &lt;- copy_to(sc, iris)

# split the data into train and validation sets
iris_data &lt;- iris_tbl %>%
  sdf_partition(train = 2/3, validation = 1/3, seed = 123)</code></pre>
<p>Then, we can create a new <code>Pipeline</code> with <code>ml_pipeline()</code> and add stages to it via the <code>%>%</code> operator. Here we also define a transformer using dplyr transformations using the newly available <code>ft_dplyr_transformer()</code>.</p>
<pre><code>pipeline &lt;- ml_pipeline(sc) %>%
  ft_dplyr_transformer(
    iris_data$train %>%
      mutate(Sepal_Length = log(Sepal_Length),
             Sepal_Width = Sepal_Width ^ 2)
  ) %>%
  ft_string_indexer(&quot;Species&quot;, &quot;label&quot;)
pipeline</code></pre>
<pre><code>## Pipeline (Estimator) with 2 stages
## </code></pre></div>