<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/how-to-set-up-a-sparklyr-cluster-in-5-minutes/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2018/02/aztk-sparklyr.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>If you've ever wanted to play around with big data sets in a Spark cluster from R with the <a rel="nofollow" href="http://spark.rstudio.com/" target="_blank">sparklyr package</a>, but haven't gotten started because setting up a Spark cluster is too hard, well … rest easy. You can get up and running in about 5 minutes using the guide <em><a rel="nofollow" href="https://github.com/Azure/aztk/wiki/SparklyR-on-Azure-with-AZTK" target="_blank">SparklyR on Azure with AZTK</a></em>, and you don't even have to install anything yourself. I'll summarize the steps below, but basically you'll run a command-line utility to launch a cluster in Azure with everything you need already installed, and then connect to <a rel="nofollow" href="https://support.rstudio.com/hc/en-us/articles/234653607-Getting-Started-with-RStudio-Server" target="_blank">RStudio Server</a> using your browser to analyze data with sparklyr.<br><a rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d2dcf86b970c-pi" target="_blank"><img border="0" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d2dcf86b970c-800wi" alt="AZTK" title="AZTK"></a></p>
<p><strong>Step 1</strong>: Install the <a rel="nofollow" href="https://github.com/Azure/aztk/blob/master/docs/00-getting-started.md" target="_blank">Azure Distributed Data Engineering Toolkit</a> (aztk). For this, you'll need a Unix command line with Python 3 installed. I'm on Windows, so I used a bash shell from the <a rel="nofollow" href="https://docs.microsoft.com/en-us/windows/wsl/about?WT.mc_id=Revolutions-blog-davidsmi" target="_blank">Windows Subsystem for Linux</a> and it worked great. (I just had to use <code>pip3</code> instead of <code>pip</code> to install, since the default there is Python 2.) The same process should work with other Linux distros or from a Mac terminal.</p>
<p><strong>Step 2</strong>: Log into the <a rel="nofollow" href="https://portal.azure.com/" target="_blank">Azure Portal</a> with your Azure subscription. If you don't have an Azure subscription, you can sign up for free and <a rel="nofollow" href="http://cda.ms/7v%20" target="_blank">get $200 in Azure credits</a>.  </p>
<p><strong>Step 3</strong>: Back at the command line, set up authentication in the <code>secrets.yaml</code> file. You'll be using the Azure portal to retrieve the necessary keys, and you'll need to create an <a rel="nofollow" href="https://docs.microsoft.com/en-us/azure/batch/?WT.mc_id=Revolutions-blog-davidsmi" target="_blank">Azure Batch</a> account if you don't have one already. (Batch is the HPC cluster and job-management service in Azure.) You can find <a rel="nofollow" href="https://github.com/Azure/aztk/blob/master/docs/00-getting-started.md#setting-up-your-accounts" target="_blank">step-by-step details</a> in the aztk documentation.</p>
<p><strong>Step 4: </strong>Configure your cluster defaults in the <code>cluster.yaml</code> file. Here you can define the default <a rel="nofollow" href="https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes?WT.mc_id=Revolutions-blog-davidsmi" target="_blank">VM instance size</a> used for the cluster nodes; for example <code>vm_size: standard_a2</code> gives you basic 2-core nodes. (You can override this in the command line, but it's convenient to set it here.) You'll also need to <a rel="nofollow" href="https://github.com/Azure/aztk/blob/master/docs/12-docker-image.md" target="_blank">specify a dockerfile here</a> that will be used to set up the node images, and for use with sparklyr you'll need to specify one that includes R and the version of Spark you want. I used:</p>
<pre>docker_repo: aztk/r-base:spark2.2.0-r3.4.1-base
</pre>
<p>This <a rel="nofollow" href="https://hub.docker.com/r/aztk/r-base/~/dockerfile/" target="_blank">provides</a> an image with Spark 2.2.0, R 3.4.1, and a suite of R packages pre-installed, including <a rel="nofollow" href="https://mran.microsoft.com/package/sparklyr" target="_blank">sparklyr</a> and <a rel="nofollow" href="https://www.tidyverse.org/" target="_blank">tidyverse</a>. (You could provide your own dockerfile here, if you need other things installed on the nodes.)</p>
<p><strong>Step 5</strong>: Privision a Spark cluster. This is the easy bit: just use the command line tool like this:</p>
<pre>aztk spark cluster create --id mysparklyr4 --size 4
</pre>
<p>In this case, it will launch a cluster of 4 nodes, each with 2 cores (pre the <code>vm_size</code> option configured above.) Each node will be pre-installed with R and (Warning: the default quotas for Azure Batch are laughably low: for me it was 24 cores <em>total</em> at first. You can <a rel="nofollow" href="https://docs.microsoft.com/en-us/azure/batch/batch-quota-limit?WT.mc_id=Revolutions-blog-davidsmi" target="_blank">get your limit raised</a> fairly easily, but it can take a day to get approval.) Provisioning a cluster takes about 5 minutes; while your waiting you can check on the progress by clicking on the cluster name in the &quot;Pools&quot; section of your Azure Batch account within the Azure Portal.</p>
<p><a rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb09f5eae0970d-pi" target="_blank"><img border="0" alt="Pools" title="Pools" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb09f5eae0970d-800wi"></a><br>Once it's ready, you'll also need to provide a password for the head node unless you set up ssh keys in the <code>secrets.yaml</code> file. </p>
<p><strong>Step 6</strong>: Connect to the head node of the Spark cluster. Normally you'd need to find the IP address first, but aztk makes it easy with its ssh command:</p>
<pre>aztk spark cluster ssh --id mysparklyr4
</pre>
<p>(You'll need to provide a password here, if you set one up in Step 5.) This gives you a shell on the head node, but more importantly it maps the ports for Spark and RStudio server, so that you can connect to them using <code>http://localhost</code> URLs in the next step. <strong>Don't exit</strong> from this shell until you're done with the next steps, or the port mappings will be cancelled. </p>
<p><strong>Step 7</strong>: Connect to RStudio Server</p>
<p>Open a browser window on your desktop, and browse to <code>http://localhost:8787</code>. This will open up RStudio Server in your browser. (The default login is rstudio/rstudio.) To be clear, RStudio Server is running on the head node of your cluster in Azure Batch, <em>not</em> on your local machine: the port mapping from the previous step is redirecting your local port 8787 to the remote cluster.</p>
<p>From here, you can use RStudio as you normally would. In particular, the sparklyr package is already installed, so you can connect to the Spark cluster directly and use RStudio Server's built-in features for working with Spark.</p>
<p><a rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d2dcf827970c-pi" target="_blank"><img border="0" alt="RStudio-Spark" title="RStudio-Spark" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d2dcf827970c-800wi"></a></p>
<p>One of the nice things about using RStudio Server is that you can shut down your browser or even your machine, and RStudio Server will preserve its state so that you can pick up exactly where you left off next time you log in. (Just use <code>aztk spark cluster ssh</code> to reapply the port mappings first, if necessary.)</p>
<p><strong>Step 8</strong>: When you're finished, shut down your cluster using the <code>aztk spark cluster delete</code> command. (While you can delete the nodes from the Pools view in the Azure portal, the command does some additional cleanup for you.) You'll be charged for each node in the cluster at the usual VM rates for as long as the cluster is provisioned. (One cost-saving option is to use <a rel="nofollow" href="https://docs.microsoft.com/en-us/azure/batch/batch-low-pri-vms?WT.mc_id=Revolutions-blog-davidsmi" target="_blank">low-priority VMs</a> for the nodes, for savings of up to 90% compared to the usual rates.)</p>
<p>That's it! Once you get used to it, it's all quick and easy — the longest part is waiting for the cluster to spin up in Step 5. This is just a summary, but the full details see the guide <a rel="nofollow" href="https://github.com/Azure/aztk/wiki/SparklyR-on-Azure-with-AZTK" target="_blank">SparklyR on Azure with AZTK</a>.</p>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2018/02/aztk-sparklyr.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/iXETdqPVO04" height="1"></a></div>