We’re excited to announce a new release of the sparklyr package, available in CRAN today! sparklyr 0.6 introduces new features to:
Distribute R computations using spark_apply() to execute arbitrary R code across your Spark cluster. You can now use all of your favorite R packages and functions in a distributed context.
Connect to External Data Sources using spark_read_source(), spark_write_source(), spark_read_jdbc() and spark_write_jdbc().
Use the Latest Frameworks including dplyr 0.7, DBI 0.7, RStudio 1.1 and Spark 2.2.
and several improvements across:
Spark Connections add a new Databricks connection that enables using sparklyr in Databricks through mode=&quot;databricks&quot;, add support for Yarn Cluster through master=&quot;yarn-cluster&quot; and connection speed was also improved.
Dataframes add support for sdf_pivot(), sdf_broadcast(), cbind(), rbind(), sdf_separate_column(), sdf_bind_cols(), sdf_bind_rows(), sdf_repartition() and sdf_num_partitions().
Machine Learning adds support for multinomial regression in ml_logistic_regression(), weights.column for GLM, ml_model_data() and a new ft_count_vectorizer() function for ml_lda().
Many other improvements, from initial support for broom over ml_linear_regression() and ml_generalized_linear_regression(), dplyr support for %like%, %rlike% and %regexp%, sparklyr extensions now support download_scalac() to help you install the required Scala compilers while developing extensions, Hive database management got simplified with tbl_change_db() and src_databases() to query and switch between Hive databases. RStudio started a joint effort with Microsoft to support a cross-platform Spark installer under github.com/rstudio/spark-install.
Additional changes and improvements can be found in the sparklyr NEWS file.
Updated documentation and examples are available at spark.rstudio.com. For questions or feedback, please feel free to open a sparklyr github issue or a sparklyr stackoverflow question.
Distributed R
sparklyr 0.6 provides support for executing distributed R code through spark_apply(). For instance, after connecting and copying some data:
library(sparklyr)
sc %
ft_tokenizer(&quot;text&quot;, &quot;tokens&quot;) %__%
ft_count_vectorizer(&quot;tokens&quot;, &quot;features&quot;, vocabulary.only = TRUE)
## [1] &quot;jane&quot; &quot;sense&quot; &quot;austen&quot; &quot;by&quot; &quot;sensibility&quot;
## [6] &quot;and&quot;
That’s all for now, disconnecting:
spark_disconnect(sc)