<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/sparklyr-0-8/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://blog.rstudio.com/2018/05/14/sparklyr-0-8/"> RStudio Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p> 

<br>
</p>
<p>We’re pleased to announce that <a rel="nofollow" href="https://cran.r-project.org/package=sparklyr" target="_blank">sparklyr 0.8</a> is now available on CRAN! Sparklyr provides an R interface to Apache Spark. It supports dplyr syntax for working with Spark DataFrames and exposes the full range of machine learning algorithms available in Spark ML. You can also learn more about Apache Spark and sparklyr at <a rel="nofollow" href="http://spark.rstudio.com/" target="_blank">spark.rstudio.com</a> and the <a rel="nofollow" href="https://www.rstudio.com/resources/webinars/introducing-an-r-interface-for-apache-spark/" target="_blank">sparklyr webinar series</a>. In this version, we added support for Spark 2.3, Livy 0.5, and various enhancements and bugfixes. For this post, we’d like to highlight a new feature from Spark 2.3 and introduce the mleap and graphframes extensions.</p>
<div>
<h2>Parallel Cross-Validation</h2>
<p>Spark 2.3 supports parallelism in hyperparameter tuning. In other words, instead of training each model specification serially, you can now train them in parallel. This can be enabled by setting the <code>parallelism</code> parameter in <code>ml_cross_validator()</code> or <code>ml_train_split_validation()</code>. Here’s an example:</p>
<pre><code>library(sparklyr)
sc &lt;- spark_connect(master = &quot;local&quot;, version = &quot;2.3.0&quot;)
iris_tbl &lt;- sdf_copy_to(sc, iris)

# Define the pipeline
labels &lt;- c(&quot;setosa&quot;, &quot;versicolor&quot;, &quot;virginica&quot;)
pipeline &lt;- ml_pipeline(sc) %>%
  ft_vector_assembler(
    c(&quot;Sepal_Width&quot;, &quot;Sepal_Length&quot;, &quot;Petal_Width&quot;, &quot;Petal_Length&quot;),
    &quot;features&quot;
  ) %>%
  ft_string_indexer_model(&quot;Species&quot;, &quot;label&quot;, labels = labels) %>%
  ml_logistic_regression()

# Specify hyperparameter grid
grid &lt;- list(
  logistic = list(
    elastic_net_param = c(0.25, 0.75),
    reg_param = c(1e-3, 1e-4)
  )
)

# Create the cross validator object
cv &lt;- ml_cross_validator(
  sc, estimator = pipeline, estimator_param_maps = grid,
  evaluator = ml_multiclass_classification_evaluator(sc),
  num_folds = 3, parallelism = 4
)

# Train the models
cv_model &lt;- ml_fit(cv, iris_tbl)</code></pre>
<p>Once the models are trained, you can inspect the performance results by using the newly available helper function <code>ml_validation_metrics()</code>:</p>
<pre><code>ml_validation_metrics(cv_model)</code></pre>
<pre><code>##       f1 elastic_net_param_1 reg_param_1
## 1 0.9506                0.25       1e-03
## 2 0.9384                0.75       1e-03
## 3 0.9384                0.25       1e-04
## 4 0.9569                0.75       1e-04</code></pre>
<pre><code>spark_disconnect(sc)</code></pre>
</div>
<div>
<h2>Pipelines in Production</h2>
<p>Earlier this year, we <a rel="nofollow" href="https://blog.rstudio.com/2018/01/29/sparklyr-0-7/" target="_blank">announced support for ML Pipelines in sparklyr</a>, and discussed how one can persist models onto disk. While that workflow is appropriate for batch scoring of large datasets, we also wanted to enable real-time, low-latency scoring using pipelines developed with sparklyr. To enable this, we’ve developed the <a rel="nofollow" href="https://cran.r-project.org/package=mleap" target="_blank">mleap</a> package, available on CRAN, which provides an interface to the <a rel="nofollow" href="https://github.com/combust/mleap" target="_blank">MLeap</a> open source project.</p>
<p>MLeap allows you to use your Spark pipelines in any Java-enabled device or service. This works by serializing Spark pipelines which can later be loaded into the Java Virtual Machine (JVM) for scoring without requiring a Spark cluster. This means that software engineers can take Spark pipelines exported with sparklyr and easily embed them in web, desktop or mobile applications.</p>
<p>To get started, simply grab the package from CRAN and install the necessary dependencies:</p>
<pre><code>install.packages(&quot;mleap&quot;)
library(mleap)
install_maven()
install_mleap()</code></pre>
<p>Then, build a pipeline as usual:</p>
<pre><code>library(sparklyr)
sc &lt;- spark_connect(master = &quot;local&quot;, version = &quot;2.2.0&quot;)
mtcars_tbl &lt;- sdf_copy_to(sc, mtcars)

# Create a pipeline and fit it
pipeline &lt;- ml_pipeline(sc) %>%
  ft_binarizer(&quot;hp&quot;, &quot;big_hp&quot;, threshold = 100) %>%
  ft_vector_assembler(c(&quot;big_hp&quot;, &quot;wt&quot;, &quot;qsec&quot;), &quot;features&quot;) %>%
  ml_gbt_regressor(label_col = &quot;mpg&quot;)
pipeline_model &lt;- ml_fit(pipeline, mtcars_tbl)</code></pre>
<p>Once we have the pipeline model, we can export it via <code>ml_write_bundle()</code>:</p>
<pre><code># Export model
model_path &lt;- file.path(tempdir(), &quot;mtcars_model.zip&quot;)
transformed_tbl &lt;- ml_transform(pipeline_model, mtcars_tbl)
ml_write_bundle(pipeline_model, transformed_tbl, model_path)
spark_disconnect(sc)</code></pre>
<p>At this point, we’re ready to use <code>mtcars_model.zip</code> in other applications. Notice that the following code does not require Spark:</p>
<pre><code># Import model
model &lt;- mleap_load_bundle(model_path)

# Create a data frame to be scored
newdata &lt;- tibble::tribble(
  ~qsec, ~hp, ~wt,
  16.2,  101, 2.68,
  18.1,  99,  3.08
)

# Transform the data frame
transformed_df &lt;- mleap_transform(model, newdata)
dplyr::glimpse(transformed_df)</code></pre>
<pre><code>## Observations: 2
## Variables: 6
## $ qsec       </code></pre></div>