Today we are excited to share that a new release of sparklyr is available on CRAN! This 0.9 release enables you to:
Create Spark structured streams to process real time data from many data sources using dplyr, SQL, pipelines, and arbitrary R code.
Monitor connection progress with upcoming RStudio Preview 1.2 features and support for properly interrupting Spark jobs from R.
Use Kubernetes clusters with sparklyr to simplify deployment and maintenance.
In addition, sparklyr 0.9 adds support for Spark 2.3.1 and Spark 2.2.3 and extends broom models in sparklyr. An extensive list of improvements and fixes is available in the sparklyr NEWS file.
Streams
Spark structured streams provide parallel and fault-tolerant data processing, useful when analyzing real time data. You create a stream in sparklyr by defining sources, transformations and a destination:
The sources are defined using any of the stream_read_*() functions to read streams of data from various data sources.
The transformations can be specified using dplyr, SQL, scoring pipelines or R code through spark_apply().
The destination is defined with the stream_write_*() functions, it is often also referenced as a sink.
For instance, after connecting with sc % stream_write_text(&quot;destination/&quot;)
Stream: 1857a67b-38f7-4f78-8a4c-959594bf0c70
Status: Waiting for next trigger
Active: TRUE
Once this is executed, sparklyr creates the stream and starts running it; the stream will be destroyed when the R session terminates or when stream_stop() is called on the stream instance.
There are many useful use cases for streams. For example, you can use streams to analyze access logs in an Amazon S3 bucket in real time. The following example creates a stream over an S3 bucket containing access logs, parses the log entries using the webreadr through spark_apply(), finds the most accessed objects using dplyr and, writes the results into an in-memory data frame:
urls_stream %
spark_apply(
~webreadr::read_s3(paste(c(.x$line, &quot;&quot;), collapse = &quot;\n&quot;)),
columns = lapply(webreadr::read_s3(&quot;\n&quot;), class)
) %__%
group_by(uri) %__%
summarize(n = n()) %__%
arrange(desc(n)) %__%
stream_write_memory(&quot;urls_stream&quot;, mode = &quot;complete&quot;)
Now that the urls_stream is running, we can view data being processed through:
stream_view(urls_stream)
You can also easily display streaming data using Shiny. Use the sparklyr::reactiveSpark() function to create a Shiny reactive from streaming data that can then be used to interact with other Shiny components and visualizations.
For instance, we can create a Shiny app using Spark streams that counts words from text files under a source/ folder as follows:
library(shiny)
library(sparklyr)
library(ggplot2)
library(dplyr)
sc %
ft_stop_words_remover(&quot;tokens&quot;, &quot;words&quot;) %__%
transmute(words = explode(words)) %__%
filter(nchar(words) __ 0) %__%
group_by(words) %__%
summarize(n = n()) %__%
arrange(desc(n)) %__%
filter(n __ 100) %__%
reactiveSpark()
ui % spark_apply(~ Sys.sleep(60 * 10))
# Start a new operation without having to restart the Spark context.
sdf_len(sc, 10)
While running sparklyr 0.9 under RStudio Preview 1.2, long running jobs will be displayed in the console tab and under the Jobs panel:
Kubernetes
sparklyr 0.9 enables support for Kubernetes. A cluster from a properly configured client can be launched as follows:
sc