<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/sparklyr-0-9/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://blog.rstudio.com/2018/10/01/sparklyr-0-9/"> RStudio Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Today we are excited to share that a new release of <a rel="nofollow" href="https://spark.rstudio.com/" target="_blank">sparklyr</a> is <a rel="nofollow" href="https://cran.r-project.org/package=sparklyr" target="_blank">available on CRAN</a>! This <code>0.9</code> release enables you to:</p>
<ul>
<li>Create Spark structured <strong>streams</strong> to process real time data from many data sources using <a rel="nofollow" href="https://dplyr.tidyverse.org/" target="_blank">dplyr</a>, <a rel="nofollow" href="https://cran.r-project.org/package=DBI" target="_blank">SQL</a>, <a rel="nofollow" href="https://spark.rstudio.com/guides/pipelines/" target="_blank">pipelines</a>, and arbitrary R code.</li>
<li>Monitor connection progress with upcoming <a rel="nofollow" href="https://www.rstudio.com/products/rstudio/download/preview/" target="_blank">RStudio Preview 1.2</a> features and support for properly <strong>interrupting</strong> Spark jobs from R.</li>
<li>Use <strong>Kubernetes</strong> clusters with <code>sparklyr</code> to simplify deployment and maintenance.</li>
</ul>
<p>In addition, <code>sparklyr 0.9</code> adds support for <strong>Spark 2.3.1</strong> and <strong>Spark 2.2.3</strong> and <a rel="nofollow" href="https://github.com/rstudio/sparklyr/blob/master/NEWS.md#broom" target="_blank">extends</a> <a rel="nofollow" href="https://broom.tidyverse.org/" target="_blank">broom</a> models in <code>sparklyr</code>. An extensive list of improvements and fixes is available in the <a rel="nofollow" href="https://github.com/rstudio/sparklyr/blob/master/NEWS.md" target="_blank">sparklyr NEWS</a> file.</p>
<h2>Streams</h2>
<p>Spark <a rel="nofollow" href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html" target="_blank">structured streams</a> provide parallel and fault-tolerant data processing, useful when analyzing real time data. You create a stream in <code>sparklyr</code> by defining sources, transformations and a destination:</p>
<ul>
<li>The <strong>sources</strong> are defined using any of the <code>stream_read_*()</code> functions to read streams of data from various data sources.</li>
<li>The <strong>transformations</strong> can be specified using <code>dplyr</code>, <code>SQL</code>, scoring pipelines or R code through <code>spark_apply()</code>.</li>
<li>The <strong>destination</strong> is defined with the <code>stream_write_*()</code> functions, it is often also referenced as a sink.</li>
</ul>
<p>For instance, after connecting with <code>sc &lt;- spark_connect(master = &quot;local&quot;)</code>, the simplest stream we can define is one to continuously copy text files between a <code>source</code> folder and a <code>destination</code> folder as follows:</p>
<pre><code>stream_read_text(sc, &quot;source/&quot;) %>% stream_write_text(&quot;destination/&quot;)
</code></pre>
<pre><code>Stream: 1857a67b-38f7-4f78-8a4c-959594bf0c70
Status: Waiting for next trigger
Active: TRUE
</code></pre>
<p>Once this is executed, <code>sparklyr</code> creates the stream and starts running it; the stream will be destroyed when the R session terminates or when <code>stream_stop()</code> is called on the stream instance.</p>
<p>There are many useful use cases for streams. For example, you can use streams to analyze access logs in an Amazon S3 bucket in real time. The following example creates a stream over an S3 bucket containing access logs, parses the log entries using the <a rel="nofollow" href="https://cran.r-project.org/package=webreadr" target="_blank">webreadr</a> through <a rel="nofollow" href="https://spark.rstudio.com/guides/distributed-r/" target="_blank">spark_apply()</a>, finds the most accessed objects using <code>dplyr</code> and, writes the results into an in-memory data frame:</p>
<pre><code>urls_stream &lt;- stream_read_text(sc, &quot;s3a://your-s3-bucket/&quot;) %>%
  spark_apply(
    ~webreadr::read_s3(paste(c(.x$line, &quot;&quot;), collapse = &quot;\n&quot;)),
    columns = lapply(webreadr::read_s3(&quot;\n&quot;), class)
  ) %>%
  group_by(uri) %>%
  summarize(n = n()) %>%
  arrange(desc(n)) %>%
  stream_write_memory(&quot;urls_stream&quot;, mode = &quot;complete&quot;)
</code></pre>
<p>Now that the <code>urls_stream</code> is running, we can view data being processed through:</p>
<pre><code>stream_view(urls_stream)
</code></pre>
<p><img alt="RStudio monitoring sparklyr job" src="https://i2.wp.com/blog.rstudio.com/images/2018-10-01-sparklyr-stream-view.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>You can also easily display streaming data using <a rel="nofollow" href="https://shiny.rstudio.com/" target="_blank">Shiny</a>. Use the <code>sparklyr::reactiveSpark()</code> function to create a Shiny reactive from streaming data that can then be used to interact with other Shiny components and visualizations.</p>
<p>For instance, we can create a Shiny app using Spark streams that counts words from text files under a <code>source/</code> folder as follows:</p>
<pre><code>library(shiny)
library(sparklyr)
library(ggplot2)
library(dplyr)

sc &lt;- spark_connect(master = &quot;local&quot;)

dir.create(&quot;source&quot;)
reactiveCount &lt;- stream_read_text(sc, &quot;source/&quot;) %>%
    ft_tokenizer(&quot;line&quot;, &quot;tokens&quot;) %>%
    ft_stop_words_remover(&quot;tokens&quot;, &quot;words&quot;) %>%
    transmute(words = explode(words)) %>%
    filter(nchar(words) > 0) %>%
    group_by(words) %>%
    summarize(n = n()) %>%
    arrange(desc(n)) %>%
    filter(n > 100) %>%
    reactiveSpark()

ui &lt;- fluidPage(plotOutput(&quot;wordsPlot&quot;))

server &lt;- function(input, output) {
    output$wordsPlot &lt;- renderPlot({
        reactiveCount() %>%
          head(n=10) %>%
          ggplot() + aes(x=words, y=n) + geom_bar(stat=&quot;identity&quot;)
    })
}

shinyApp(ui = ui, server = server)
</code></pre>
<p>We can then write Jane Austen’s books to this folder starting with <code>writeLines(janeaustenr::emma, &quot;source/emma.txt&quot;)</code> and similar code for the remaining ones, each time a book is saved, the Shiny app updates accordingly:</p>
<p><img alt="Shiny app using Spark stream to count words in Emma" src="https://i2.wp.com/blog.rstudio.com/images/2018-10-01-sparklyr-shiny-app-books.gif?w=456&ssl=1" data-recalc-dims="1"></p>
<p>You can learn more about <code>sparklyr</code> streaming at <a rel="nofollow" href="https://spark.rstudio.com/guides/streaming/" target="_blank">https://spark.rstudio.com/guides/streaming/</a>.</p>
<h2>Monitoring and Interrupting Jobs</h2>
<p>In <code>sparklyr 0.9</code>, you can now gracefully interrupt long-running operations and reuse the Spark connection to execute other operations. This is useful when you execute a query or modeling function that is taking longer than expected, or when you didn’t quite execute the code you wanted to. For example:</p>
<pre><code># Stop the following long-running operation with `ctrl+c` or 'stop' in RStudio
sdf_len(sc, 10) %>% spark_apply(~ Sys.sleep(60 * 10))

# Start a new operation without having to restart the Spark context.
sdf_len(sc, 10)
</code></pre>
<p>While running <code>sparklyr 0.9</code> under <a rel="nofollow" href="https://www.rstudio.com/products/rstudio/download/preview/" target="_blank">RStudio Preview 1.2</a>, long running jobs will be displayed in the console tab and under the Jobs panel:</p>
<p><img alt="RStudio monitoring sparklyr job" src="https://i1.wp.com/blog.rstudio.com/images/2018-10-01-sparklyr-monitored-connections.png?w=456&ssl=1" data-recalc-dims="1"></p>
<h2>Kubernetes</h2>
<p><code>sparklyr 0.9</code> enables support for Kubernetes. A cluster from a properly configured client can be launched as follows:</p>
<pre><code>sc &lt;- spark_connect(config = spark_config_kubernetes(&quot;k8s://hostname:8443&quot;))
</code></pre>
<p>Please note that Spark on Kubernetes requires a proper container image, see <code>?spark_config_kubernetes</code> for details. In addition, Kubernetes support <a rel="nofollow" href="https://spark.apache.org/docs/2.3.0/running-on-kubernetes.html" target="_blank">was just added in Spark 2.3.0</a> and the Kubernetes scheduler is currently experimental in Spark.</p>
<p>We hope you enjoy all new features in sparklyr 0.9! You can read more about these features and others at <a rel="nofollow" href="https://spark.rstudio.com/" target="_blank">https://spark.rstudio.com/</a>, get help from the R community at <a rel="nofollow" href="https://community.rstudio.com/tags/sparklyr" target="_blank">https://community.rstudio.com/tags/sparklyr</a>, and report issues with sparklyr at <a rel="nofollow" href="https://github.com/rstudio/sparklyr" target="_blank">https://github.com/rstudio/sparklyr</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://blog.rstudio.com/2018/10/01/sparklyr-0-9/"> RStudio Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/YzptpleZzQE" height="1"></a></div>