<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/sparklyr-a-test-drive-on-yarn/"></div></div><div><a data-url="https://www.r-bloggers.com/sparklyr-a-test-drive-on-yarn/" data-counturl="https://www.r-bloggers.com/sparklyr-a-test-drive-on-yarn/" data-count="horizontal" data-text="sparklyr: a test drive on YARN" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.nodalpoint.com/sparklyr-on-yarn/"> R – Nodalpoint</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a target="_blank" rel="nofollow" href="http://spark.rstudio.com/index.html"><tt>sparklyr</tt></a> is a <a target="_blank" rel="nofollow" href="https://blog.rstudio.org/2016/09/27/sparklyr-r-interface-for-apache-spark/">new</a> R front-end for Apache Spark, developed by the good people at RStudio. It offers much more functionality compared to the existing <a target="_blank" rel="nofollow" href="http://spark.apache.org/docs/2.0.1/api/R/index.html"><tt>SparkR</tt></a> interface by Databricks, allowing both <a target="_blank" rel="nofollow" href="http://spark.rstudio.com/dplyr.html"><tt>dplyr</tt>-based</a> data transformations, as well as access to the machine learning libraries of both <a target="_blank" rel="nofollow" href="http://spark.rstudio.com/mllib.html">Spark</a> and <a target="_blank" rel="nofollow" href="http://spark.rstudio.com/h2o.html">H2O Sparkling Water</a>. Moreover, the <a target="_blank" rel="nofollow" href="https://blog.rstudio.org/2016/11/01/announcing-rstudio-v1-0/">latest</a> RStudio IDE v1.0 now offers native support for managing Spark connections and viewing Spark context data in a dedicated tab.</p>
<p>Although there are already lots of <a target="_blank" rel="nofollow" href="http://spark.rstudio.com/examples.html">examples</a> provided by RStudio, rather unsurprisingly almost all of them focus on using Spark in local mode (the <a target="_blank" rel="nofollow" href="https://beta.rstudioconnect.com/content/1705/taxiDemo.nb.html">billion NYC taxi trips analysis</a> is an exception, but we’ll see later where it falls short as an example). So, I thought of getting the new package for a test drive on YARN, and here are my preliminary findings, hints, and tips.</p>
<p>In what follows, we use <tt>sparklyr</tt> version 0.4, i.e. the latest stable version available in CRAN at the time of writing.</p>
<h4>RStudio Server required</h4>
<p>If you expected to be able to connect to a remote cluster from your RStudio Desktop, you’d better forget it, at least for the time being; here is the message you get when attempt to connect to a remote cluster:</p>
<div class="wp-caption"><a target="_blank" rel="nofollow" href="https://i2.wp.com/www.nodalpoint.com/wp-content/uploads/2016/11/1-IDE.png"><img alt="RStudio Desktop message when attempting to connect to a remote cluster" class="wp-image-3314" src="https://i2.wp.com/www.nodalpoint.com/wp-content/uploads/2016/11/1-IDE.png?resize=456%2C358" data-recalc-dims="1"></a>
<p class="wp-caption-text">RStudio Desktop message when attempting to connect to a remote cluster</p>
</div>
<p>This means that, even if your machine is directly connected to the cluster, you still cannot use the connection functionality provided by RStudio Desktop.</p>
<p>That said, truth is that in what follows I found little use of this functionality even with RStudio Server, ending up managing the connection in-script – and I cannot imagine that one cannot do the same from RStudio Desktop, provided that the machine has direct access to the cluster (not tested though).</p>
<h4>Configuring your resources</h4>
<p>When working in local mode, you don’t actually have to specifically define the resources to be used; but when working on YARN, this is necessary. Here is the configuration used in the <a target="_blank" rel="nofollow" href="https://beta.rstudioconnect.com/content/1705/taxiDemo.nb.html">billion NYC taxi trips analysis</a> example mentioned above:</p>
<pre>
config &lt;- spark_config()
config$spark.driver.cores &lt;- 32
config$spark.executor.cores &lt;- 32
config$spark.executor.memory &lt;- &quot;40g&quot;
</pre>
<p>That’s great, only I want to also specify a number of executors. Of course, I cannot remember by heart all the different configuration arguments – what’s the documentation for? So, I went through the Spark 1.6.1 <a target="_blank" rel="nofollow" href="http://spark.apache.org/docs/1.6.1/configuration.html">configuration documents</a>, looking for the exact argument to use.</p>
<p>But it’s not there… Neither it is in the relevant Spark 2.0.0 <a target="_blank" rel="nofollow" href="http://spark.apache.org/docs/2.0.0/configuration.html">docs</a>. I can find how to configure memory and cores for both driver and executors, along with dozens of other things, but nothing about the number of executors…</p>
<p>I finally located it in some old Spark 1.5 <a target="_blank" rel="nofollow" href="https://spark.apache.org/docs/1.5.2/running-on-yarn.html">documents</a> – it is <code>spark.executor.instances</code>, with a default value of 2; it is a mystery to me why they chose not to include it in later versions of the documentation – but again, I have already argued <a target="_blank" rel="nofollow" href="http://www.nodalpoint.com/spark-classification/">elsewhere</a> about puzzling and missing things in Spark docs.</p>
<p>With this last piece, we are now ready to setup a Spark connection to YARN via <tt>sparklyr</tt> from an RStudio Server instance within our cluster:</p>
<pre>
library(sparklyr)

Sys.setenv(SPARK_HOME = &quot;/home/ctsats/spark-1.6.1-bin-hadoop2.6&quot;)
Sys.setenv(HADOOP_CONF_DIR = '/etc/hadoop/conf.cloudera.hdfs')
Sys.setenv(YARN_CONF_DIR = '/etc/hadoop/conf.cloudera.yarn')

config &lt;- spark_config()
config$spark.executor.instances &lt;- 4
config$spark.executor.cores &lt;- 4
config$spark.executor.memory &lt;- &quot;4G&quot;

sc &lt;- spark_connect(master=&quot;yarn-client&quot;, config=config, version = '1.6.1')

</pre>
<p>And, I am happy to report, it works like a charm.</p>
<p>It is advisable that the definitions of the relevant environmental variables be included in the <tt>Renviron.site</tt> configuration file; while this sounds like a good idea for <code>HADOOP_CONF_DIR</code> and <code>YARN_CONF_DIR</code>, I prefer to have a more explicit and dynamic control of <code>SPARK_HOME</code>, since I maintain several different Spark versions in my cluster.</p>
<h4><tt>dplyr</tt> is required</h4>
<p>Let’s try some elementary operations from the examples first; the following command should upload the <code>iris</code> data to our Spark context:</p>
<pre>
> iris_tbl &lt;- copy_to(sc, iris, &quot;iris&quot;, overwrite = TRUE)
Error: could not find function &quot;copy_to&quot;
</pre>
<p>This error puzzled me, and it took some time to figure it out; after all, <code>copy_to</code> is indeed a <tt>sparklyr</tt> <a target="_blank" rel="nofollow" href="http://spark.rstudio.com/reference/sparklyr/latest/copy_to.html">function</a>.</p>
<p>It turns out that some <tt>sparklyr</tt> functions, including <code>copy_to</code>, require <tt>dplyr</tt> in order to be available:</p>
<pre>
> library(dplyr)
> iris_tbl &lt;- copy_to(sc, iris, &quot;iris&quot;, overwrite = TRUE) 
The following columns have been renamed: 
- 'Sepal.Length' => 'Sepal_Length' (#1)
- 'Sepal.Width'  => 'Sepal_Width'  (#2)
- 'Petal.Length' => 'Petal_Length' (#3)
- 'Petal.Width'  => 'Petal_Width'  (#4)
> iris_tbl
Source:   query [?? x 5]
Database: spark connection master=yarn-client app=sparklyr local=FALSE

   Sepal_Length Sepal_Width Petal_Length Petal_Width Species
          &lt;dbl>       &lt;dbl>        &lt;dbl>       &lt;dbl>   &lt;chr>
1           5.1         3.5          1.4         0.2  setosa
2           4.9         3.0          1.4         0.2  setosa
3           4.7         3.2          1.3         0.2  setosa
4           4.6         3.1          1.5         0.2  setosa
5           5.0         3.6          1.4         0.2  setosa
6           5.4         3.9          1.7         0.4  setosa
7           4.6         3.4          1.4         0.3  setosa
8           5.0         3.4          1.5         0.2  setosa
9           4.4         2.9          1.4         0.2  setosa
10          4.9         3.1          1.5         0.1  setosa
# ... with more rows

</pre>
<p>Although not explicitly mentioned, this (i.e. loading both <tt>sparklyr</tt> and <tt>dplyr</tt>) is the advised usage, as implied by the suggested connection code popping up in the Spark tab of RStudio:</p>
<div class="wp-caption"><a target="_blank" rel="nofollow" href="https://i0.wp.com/www.nodalpoint.com/wp-content/uploads/2016/11/2-connect.png"><img alt="Connection code snippet" class="wp-image-3373" src="https://i0.wp.com/www.nodalpoint.com/wp-content/uploads/2016/11/2-connect.png?resize=456%2C108" data-recalc-dims="1"></a>
<p class="wp-caption-text">Connection code snippet</p>
</div>
<h4>Replicating the machine learning examples with Titanic data</h4>
<p>There is a nice example by RStudio, <a target="_blank" rel="nofollow" href="https://beta.rstudioconnect.com/content/1518/notebook-classification.html">comparing six Spark ML algorithms</a> on the Titanic data. It is not entirely reproducible though, due to an enigmatic line:</p>
<pre>
spark_read_parquet(sc, name = &quot;titanic&quot;, path = &quot;titanic-parquet&quot;)
</pre>
<p>which loads the Titanic data from an external Parquet file.</p>
<p>Trying to directly use the Titanic data as provided by the <tt>titanic</tt> R package produces an error in YARN mode:</p>
<pre>
> library(titanic)
> titanic_tbl &lt;- copy_to(sc, titanic::titanic_train, &quot;titanic&quot;, overwrite = TRUE)
Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 5, host-hd-03.corp.nodalpoint.com): java.lang.ArrayIndexOutOfBoundsException: 11
[...]
</pre>
<p>while, curiously enough, in local mode the command works OK (I have opened an <a target="_blank" rel="nofollow" href="https://github.com/rstudio/sparklyr/issues/307">issue</a> at Github).</p>
<p>Fortunately, saving the Titanic data in a local CSV file and then reading them from there works OK:</p>
<pre>
> write.csv(titanic_train, file=&quot;/home/ctsats/R/titanic.csv&quot;, row.names=FALSE)
> library(readr)
> titanic &lt;- read_csv(&quot;~/R/titanic.csv&quot;) [...] 
> titanic_tbl &lt;- copy_to(sc, titanic, &quot;titanic&quot;, overwrite = TRUE) 
> titanic_tbl
Source:   query [?? x 12]
Database: spark connection master=yarn-client app=sparklyr local=FALSE

   PassengerId Survived Pclass                                                Name    Sex   Age SibSp Parch
         &lt;int>    &lt;int>  &lt;int>                                               &lt;chr>  &lt;chr> &lt;dbl> &lt;int> &lt;int>
1            1        0      3                             Braund, Mr. Owen Harris   male    22     1     0
2            2        1      1 Cumings, Mrs. John Bradley (Florence Briggs Thayer) female    38     1     0
3            3        1      3                              Heikkinen, Miss. Laina female    26     0     0
4            4        1      1        Futrelle, Mrs. Jacques Heath (Lily May Peel) female    35     1     0
5            5        0      3                            Allen, Mr. William Henry   male    35     0     0
6            6        0      3                                    Moran, Mr. James   male   NaN     0     0
7            7        0      1                             McCarthy, Mr. Timothy J   male    54     0     0
8            8        0      3                      Palsson, Master. Gosta Leonard   male     2     3     1
9            9        1      3   Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg) female    27     0     2
10          10        1      2                 Nasser, Mrs. Nicholas (Adele Achem) female    14     1     0
# ... with more rows, and 4 more variables: Ticket &lt;chr>, Fare &lt;dbl>, Cabin &lt;chr>, Embarked &lt;chr>
</pre>
<h4>File access and paths</h4>
<p>You can use <code>spark_read_csv</code> to access local files, but only in local mode – it will not work in YARN client mode:</p>
<pre>
> spark_disconnect(sc) # disconnect from YARN
> sc &lt;- spark_connect(master = &quot;local&quot;, version = '1.6.1') # connect in local mode 
> spark_read_csv(sc, name=&quot;titanic_local&quot;, path=&quot;file:///home/ctsats/R/titanic.csv&quot;, header=TRUE) # local file
Source:   query [?? x 12]
Database: spark connection master=local[12] app=sparklyr local=TRUE

   PassengerId Survived Pclass                                                Name    Sex   Age SibSp Parch
         &lt;int>    &lt;int>  &lt;int>                                               &lt;chr>  &lt;chr> &lt;chr> &lt;int> &lt;int>
1            1        0      3                             Braund, Mr. Owen Harris   male    22     1     0
2            2        1      1 Cumings, Mrs. John Bradley (Florence Briggs Thayer) female    38     1     0
3            3        1      3                              Heikkinen, Miss. Laina female    26     0     0
4            4        1      1        Futrelle, Mrs. Jacques Heath (Lily May Peel) female    35     1     0
5            5        0      3                            Allen, Mr. William Henry   male    35     0     0
6            6        0      3                                    Moran, Mr. James   male    NA     0     0
7            7        0      1                             McCarthy, Mr. Timothy J   male    54     0     0
8            8        0      3                      Palsson, Master. Gosta Leonard   male     2     3     1
9            9        1      3   Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg) female    27     0     2
10          10        1      2                 Nasser, Mrs. Nicholas (Adele Achem) female    14     1     0
# ... with more rows, and 4 more variables: Ticket &lt;chr>, Fare &lt;dbl>, Cabin &lt;chr>, Embarked &lt;chr>
> spark_disconnect(sc) # disconnect from local
> sc &lt;- spark_connect(master=&quot;yarn-client&quot;, config=config, version = '1.6.1') # reconnect to YARN 
> spark_read_csv(sc, name=&quot;titanic_local&quot;, path=&quot;file:///home/ctsats/R/titanic.csv&quot;, header=TRUE) # same local file
Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4, host-hd-03.corp.nodalpoint.com): java.io.FileNotFoundException: File file:/home/ctsats/R/titanic.csv does not exist
[...]
</pre>
<p>By default, the <code>path</code> argument in <code>spark_read_csv</code> refers to the HDFS path; so, after having copied <tt>titanic.csv</tt> to HDFS directory <tt>sparklyr</tt>, we can read it like this:</p>
<pre>
> spark_read_csv(sc, name=&quot;titanic_hdfs&quot;, path=&quot;sparklyr/titanic.csv&quot;, header=TRUE) # HDFS file
Source:   query [?? x 12]
Database: spark connection master=local[12] app=sparklyr local=TRUE

   PassengerId Survived Pclass                                                Name    Sex   Age SibSp Parch
         &lt;int>    &lt;int>  &lt;int>                                               &lt;chr>  &lt;chr> &lt;chr> &lt;int> &lt;int>
1            1        0      3                             Braund, Mr. Owen Harris   male    22     1     0
2            2        1      1 Cumings, Mrs. John Bradley (Florence Briggs Thayer) female    38     1     0
3            3        1      3                              Heikkinen, Miss. Laina female    26     0     0
4            4        1      1        Futrelle, Mrs. Jacques Heath (Lily May Peel) female    35     1     0
5            5        0      3                            Allen, Mr. William Henry   male    35     0     0
6            6        0      3                                    Moran, Mr. James   male    NA     0     0
7            7        0      1                             McCarthy, Mr. Timothy J   male    54     0     0
8            8        0      3                      Palsson, Master. Gosta Leonard   male     2     3     1
9            9        1      3   Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg) female    27     0     2
10          10        1      2                 Nasser, Mrs. Nicholas (Adele Achem) female    14     1     0
# ... with more rows, and 4 more variables: Ticket &lt;chr>, Fare &lt;dbl>, Cabin &lt;chr>, Embarked &lt;chr>
</pre>
<p>Nevertheless, while experimenting, I found one occasion where this simple path reference will not work, demanding a reference to the full HDFS path preceded with <tt>hdfs://</tt>:</p>
<pre>
> spark_read_csv(sc, name=&quot;interactions&quot;, path=&quot;recsys/data/interactions_clean.csv&quot;, header=TRUE) # HDFS file
Error: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://host-hd-01.corp.nodalpoint.com:8020/home/ctsats/recsys/data/interactions_clean.csv
[...]
> spark_read_csv(sc, name=&quot;interactions&quot;, path=&quot;hdfs:///user/ctsats/recsys/data/interactions_clean.csv&quot;, header=TRUE) # full HDFS path provided
Source:   query [?? x 3]
Database: spark connection master=local[12] app=sparklyr local=TRUE

   user_id item_id interaction_type
     &lt;int>   &lt;int>            &lt;int>
1        7 1006839                1
2        9  944146                3
3        9 1053485                1
4       13 2444782                1
5       23  501722                1
6       23 1305844                1
7       28  252140                1
8       28  931378                1
9       28 1038508                1
10      28 1083127                1
# ... with more rows
</pre>
<p>This is rather strange, as the error message suggests that Spark is looking for the file in the path <tt>hdfs:///home/ctsats/recsys/data/interactions_clean.csv</tt>, instead of the correct one <tt>hdfs:///user/ctsats/recsys/data/interactions_clean.csv</tt>. Although I could not reproduce this behavior with other files, it is maybe a good idea to include the full HDFS paths for your file references, just to be on the safe side.</p>
<h4>Summary</h4>
<p>To summarize, here are our first impressions and hints for using <tt>sparklyr</tt> on YARN:</p>
<ul>
<li>The RStudio built-in functionality for managing Spark connections is only available for the server and not for the desktop IDE, although this functionality is not of much use</li>
<li><tt>dplyr</tt> is required along, even for functions that appear to be native to <tt>sparklyr</tt></li>
<li>You cannot access local files</li>
<li>There may be cases where you need to specify the full HDFS path for accessing files stored in Hadoop</li>
</ul>
<p>The post <a target="_blank" rel="nofollow" href="http://www.nodalpoint.com/sparklyr-on-yarn/">sparklyr: a test drive on YARN</a> appeared first on <a target="_blank" rel="nofollow" href="http://www.nodalpoint.com/">Nodalpoint</a>.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/sparklyr-a-test-drive-on-yarn/"></div></div><div><a data-url="https://www.r-bloggers.com/sparklyr-a-test-drive-on-yarn/" data-counturl="https://www.r-bloggers.com/sparklyr-a-test-drive-on-yarn/" data-count="vertical" data-text="sparklyr: a test drive on YARN" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.nodalpoint.com/sparklyr-on-yarn/"> R – Nodalpoint</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/qd5tGsU3J9k"></a></div>